feat: transparent tailnet http routing via httputil

CMakeLists.txt

@@ -14,7 +14,7 @@ set(CMAKE_CXX_STANDARD_REQUIRED ON)
 
 include_directories(src/include)
 
-set(EXTENSION_SOURCES src/quackscale_extension.cpp src/attach_ducklake.cpp src/tailscale_bridge.cpp src/tailscale_forwarder.cpp src/tailscale_log_capture.cpp)
+set(EXTENSION_SOURCES src/quackscale_extension.cpp src/attach_ducklake.cpp src/tailscale_bridge.cpp src/tailscale_forwarder.cpp src/tailscale_log_capture.cpp src/tailscale_http.cpp)
 
 if(QUACKSCALE_WITH_TAILSCALE)
     include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/Libtailscale.cmake)

README.md

@@ -243,15 +243,17 @@ flowchart TB
     m -->|encrypted TCP| c2
 ```
 
-**`tailscale_quack_forward`** is required on each client when using embedded tsnet: Quack speaks normal HTTP/TCP, which kernel routing does not send over the tailnet by itself. The forwarder listens on loopback and dials the server via `tailscale_dial`, then clients use **`ATTACH 'quack:…'`** for the remote catalog or **`attach_ducklake`** for server-owned DuckLake tables.
+After `tailscale_up`, QuackScale wraps DuckDB's HTTP layer so requests to tailnet hosts (`100.64.0.0/10`, `*.ts.net`) are dialed over tsnet automatically — clients can **`ATTACH 'quack:100.x.x.x:9494'`** directly, no forwarder. Opt out with `http_route => false`.
+
+**`tailscale_quack_forward`** remains for cases the router does not cover: bare MagicDNS **short** names (e.g. `lake-server` without a `.ts.net` suffix), a pinned `127.0.0.1:<port>` endpoint, or non-HTTP consumers. It listens on loopback and dials the server via `tailscale_dial`. Either path feeds **`ATTACH 'quack:…'`** or **`attach_ducklake`**.
 
 End-to-end recipes and DuckLake patterns: **[docs/GUIDE.md](docs/GUIDE.md)**.
 
 ---
 
 ## SQL API (`LOAD quackscale`)
 
-Use **`CALL`** for table functions (same style as `CALL quack_serve`). Parameters for `tailscale_up` / `tailscale_login`: `hostname`, `authkey` (or `TS_AUTHKEY` env), `control_url`, `state_dir`, `ephemeral`, `loopback_proxy`.
+Use **`CALL`** for table functions (same style as `CALL quack_serve`). Parameters for `tailscale_up` / `tailscale_login`: `hostname`, `authkey` (or `TS_AUTHKEY` env), `control_url`, `state_dir`, `ephemeral`, `loopback_proxy`, `http_route` (transparent HTTP routing to tailnet hosts — default `true`).
 
 ### Tailnet lifecycle
 
@@ -269,7 +271,7 @@ Use **`CALL`** for table functions (same style as `CALL quack_serve`). Parameter
 |---------|---------|
 | [`CALL tailscale_serve_local(port => 9494)`](docs/GUIDE.md#use-case-1--remote-duckdb-hub-pattern-a) | Tailscale Serve: tailnet TCP **→** `127.0.0.1:9494`. Run after local `quack_serve`. |
 | [`CALL tailscale_ping(host => 'peer', port => 9494)`](docs/GUIDE.md#observability) | TCP dial to a peer over tsnet — readiness before Quack `ATTACH`. |
-| [`CALL tailscale_quack_forward(host => 'peer', port => 9494)`](docs/GUIDE.md#standard-client-connection-recipe) | Listen on loopback; dial peer for each Quack HTTP connection. Returns `quack_uri`. **Preferred client path.** |
+| [`CALL tailscale_quack_forward(host => 'peer', port => 9494)`](docs/GUIDE.md#standard-client-connection-recipe) | Listen on loopback; dial peer for each Quack HTTP connection. Returns `quack_uri`. For MagicDNS short names, pinned local ports, or non-HTTP clients — otherwise `ATTACH 'quack:100.x:9494'` works directly after `tailscale_up`. |
 | [`CALL tailscale_quack_proxy()`](docs/DEVELOPMENT.md) | Legacy SOCKS proxy + `ALL_PROXY` — deprecated; use `tailscale_quack_forward`. |
 | [`CALL tailscale_proxy_status()`](docs/DEVELOPMENT.md) | Legacy SOCKS status. |
 

docs/GUIDE.md

@@ -30,7 +30,9 @@ Credentials: [AUTHENTICATION.md](AUTHENTICATION.md). Build and SQL reference: [R
 └─────────────────────────────────────────────────────────────────┘
 ```
 
-**Why `tailscale_quack_forward`?** Quack uses normal HTTP/TCP. Embedded tsnet does not route kernel TCP to tailnet IPs. The forwarder listens on loopback and dials the peer via `tailscale_dial`.
+**Transparent routing (default).** After `tailscale_up`, QuackScale wraps DuckDB's HTTP layer so requests to tailnet hosts (`100.64.0.0/10`, `*.ts.net`) are dialed over tsnet — `ATTACH 'quack:100.x.x.x:9494'` works with no forwarder. Disable with `tailscale_up(..., http_route => false)`.
+
+**Why `tailscale_quack_forward`?** It covers what the router does not: bare MagicDNS **short** names (no `.ts.net` suffix), a pinned `127.0.0.1:<port>` endpoint, or non-HTTP clients. It listens on loopback and dials the peer via `tailscale_dial`. The compose demo uses it (stable hostnames) and also probes the direct-router path.
 
 **Why `tailscale_down`?** `tailscale_up` and the forwarder start background threads. One-shot DuckDB processes **hang after SQL finishes** unless tsnet is shut down.
 
@@ -325,7 +327,7 @@ DuckLake metadata: file (`*.ducklake`), Postgres, or DuckDB — see [DuckLake at
 |-------|------------|
 | `remote.lake.table` does not exist | Use `attach_ducklake`, `quack_query`, or `ducklake:quack:` |
 | Client hangs after SQL completes | Emit done marker, then `CALL tailscale_down()` |
-| Kernel TCP to `100.x:9494` fails from tsnet client | Use `tailscale_quack_forward` |
+| Kernel TCP to `100.x:9494` fails from tsnet client | `ATTACH 'quack:100.x:9494'` after `tailscale_up` (transparent router), or `tailscale_quack_forward` for short names / non-HTTP |
 | `quack_query` + `ATTACH remote` stalls | Run lake queries **before** attach; separate statements |
 | `quack_query(…, quack_discover())` hangs | Discover locally or use known hostname |
 

scripts/e2e/quacktail-compose-bootstrap.sh

@@ -114,6 +114,33 @@ FROM quack_query(
 SQL
 }
 
+# Direct ATTACH over the transparent HTTPUtil router — NO tailscale_quack_forward. tailscale_up
+# auto-installs the router, so the server's 100.x tailnet IP is dialed over tsnet directly.
+# A bare MagicDNS short name is not routable this way (it's neither a *.ts.net FQDN nor a 100.x
+# IP, so IsTailnetHost rejects it), hence we attach the resolved IP.
+compose_sql_router_probe() {
+  local server_ip="${1:?server tailnet ip required}"
+  local router_uri="quack:${server_ip}:${QUACK_PORT}"
+  cat <<SQL
+-- transparent router probe: ATTACH the server tailnet IP directly (no forwarder)
+CREATE SECRET router_secret (
+    TYPE quack,
+    TOKEN '${QUACK_TOKEN}',
+    SCOPE '${router_uri}'
+);
+
+ATTACH '${router_uri}' AS remote_router (
+    TYPE quack,
+    DISABLE_SSL true
+);
+
+SELECT 'ROUTER_PASSED' AS status, '${router_uri}' AS router_uri, COUNT(*)::INTEGER AS total_rows
+FROM remote_router.e2e_payload;
+
+DETACH remote_router;
+SQL
+}
+
 write_server_ducklake_sql() {
   [[ "$ENABLE_DUCKLAKE" == "1" ]] || return 0
   mkdir -p "$(dirname "$LAKE_METADATA")" "$LAKE_DATA_PATH"
@@ -182,6 +209,22 @@ write_client_session_sql() {
   local lake_select=""
   local lake_passed_sql=""
   local lake_discover_sql=""
+  local router_probe_sql=""
+  # Transparent-router probe: only when this build supports it AND the server's tailnet IP is
+  # resolvable (server already registered). Skipped on initial server boot; the client run
+  # regenerates this session after the server is up, so the probe is present by then.
+  if quacktail_quackscale_supports_router; then
+    local server_ip
+    server_ip="$(resolve_server_tailnet_ip)"
+    if [[ -n "$server_ip" ]]; then
+      router_probe_sql="$(compose_sql_router_probe "$server_ip")"
+      echo "✓ router probe enabled — direct ATTACH quack:${server_ip}:${QUACK_PORT} (no forwarder)" >&2
+    else
+      echo "warn: router probe skipped — server tailnet IP not resolvable yet (regenerated at client run)" >&2
+    fi
+  else
+    echo "note: router probe skipped — this quackscale build has no http_route (transparent router)" >&2
+  fi
   if duckdb_has_quackscale_function tailscale_ping; then
     ping_sql="CALL tailscale_ping(host => '${SERVER_HOST}', port => ${QUACK_PORT});"
   fi
@@ -236,6 +279,8 @@ FROM quack_query(
     disable_ssl => true
 );
 
+${router_probe_sql}
+
 ${lake_discover_sql}
 ${lake_attach_sql}
 ${lake_select}

scripts/e2e/quacktail-entrypoint.sh

@@ -197,6 +197,11 @@ quacktail_client_session_succeeded() {
   if [[ "${QUACKTAIL_ENABLE_DUCKLAKE:-0}" == "1" ]]; then
     grep -q "LAKE_PASSED" "$out" 2>/dev/null || return 1
   fi
+  # If the session includes the transparent-router probe (direct ATTACH, no forwarder), it
+  # must pass too. Gated on the generated SQL so older builds without the probe still pass.
+  if grep -q "ROUTER_PASSED" "${WORK}/client_session.sql" 2>/dev/null; then
+    grep -q "ROUTER_PASSED" "$out" 2>/dev/null || return 1
+  fi
   return 0
 }
 
@@ -391,6 +396,14 @@ run_client() {
     exit 1
   fi
 
+  # Make router coverage explicit: green output should never leave it ambiguous whether the
+  # transparent-router path (direct ATTACH, no forwarder) was actually exercised this run.
+  if grep -q "ROUTER_PASSED" "${WORK}/client_session.sql" 2>/dev/null; then
+    echo "✓ transparent router exercised (ROUTER_PASSED)"
+  else
+    echo "note: transparent-router probe not in this session — forwarder path only (see bootstrap log)"
+  fi
+
   if [[ "$QUIET" == "1" ]]; then
     quacktail_show_client_demo_output "$out"
     echo ""

scripts/lib/quacktail_ext.sh

@@ -31,6 +31,32 @@ quacktail_has_quackscale_function() {
   [[ "$count" == "1" ]]
 }
 
+# True if this quackscale build supports transparent HTTP routing (the http_route parameter
+# on tailscale_up). Gates the e2e router probe so older release binaries are not asserted.
+# Fails *loudly* (logs an error) if quackscale can't even load, so a broken build is not
+# silently mistaken for "old build without the router".
+quacktail_quackscale_supports_router() {
+  local duckdb_bin="${DUCKDB_BIN:-/usr/local/bin/duckdb}"
+  local ext_dir="${DUCKDB_EXTENSION_DIRECTORY:-$(quacktail_ext_container_dir)}"
+  local set_ext="SET extension_directory='${ext_dir}';"
+  local out
+  [[ -x "$duckdb_bin" ]] || { echo "warn: router probe: duckdb not executable at $duckdb_bin" >&2; return 1; }
+
+  # Baseline: prove quackscale actually loads and runs. If this fails the build is broken — say so
+  # loudly rather than silently concluding "router unsupported" (which would skip the assertion).
+  if ! "$duckdb_bin" :memory: -batch -c "${set_ext} LOAD quackscale; SELECT 1;" >/dev/null 2>&1; then
+    echo "error: router probe: 'LOAD quackscale; SELECT 1' failed — quackscale not loadable at ${ext_dir}" >&2
+    return 1
+  fi
+
+  # An invalid named parameter makes tailscale_up list its valid ones (a bind-time error, so
+  # tailscale_up never runs). "http_route" in that list ⇒ this build has the router.
+  out="$("$duckdb_bin" :memory: -batch -c \
+    "${set_ext} LOAD quackscale; CALL tailscale_up(quackscale_router_capability_probe => true);" \
+    2>&1)" || true
+  printf '%s\n' "$out" | grep -q 'http_route'
+}
+
 quacktail_list_quackscale_functions() {
   local duckdb_bin="${DUCKDB_BIN:-/usr/local/bin/duckdb}"
   local ext_dir="${DUCKDB_EXTENSION_DIRECTORY:-$(quacktail_ext_container_dir)}"

src/include/tailscale_bridge.hpp

@@ -80,7 +80,15 @@ class TailscaleBridge {
 	//! Dial host:port over the tailnet via tsnet (peer connectivity check).
 	void PingTCP(const string &host, idx_t port, idx_t timeout_ms);
 
+	//! Dial a tailnet peer over tsnet and return the connected socket fd (caller owns/closes
+	//! it). Throws if the node is not up or the dial fails. Used by the HTTPUtil router.
+	int DialTCP(const string &host, idx_t port);
+
 	string PrimaryTailnetIP() const;
+	//! First IPv4 (colon-free) tailnet address — in practice the 100.64.0.0/10 CGNAT IP, since
+	//! `ips` only holds tailnet addresses. This is the one the transparent HTTPUtil router can
+	//! reach. Empty if the node only has an IPv6 tailnet address yet.
+	string RoutableTailnetIP() const;
 	string FormatQuackURI(const string &host, idx_t port) const;
 	string QuackListenURI(idx_t port = QUACKSCALE_DEFAULT_QUACK_PORT) const;
 	vector<QuackDiscoveryEndpoint> QuackDiscoveryEndpoints(idx_t port = QUACKSCALE_DEFAULT_QUACK_PORT) const;

src/include/tailscale_http.hpp

@@ -0,0 +1,110 @@
+#pragma once
+
+#include "duckdb/common/http_util.hpp"
+#include "duckdb/common/unordered_map.hpp"
+#include "duckdb/common/vector.hpp"
+
+#include <mutex>
+
+namespace duckdb {
+
+class DatabaseInstance;
+
+//! True if `proto_host_port` (e.g. "http://100.95.32.19:9494") names a tailnet host:
+//! an IPv4 in the CGNAT range 100.64.0.0/10, or a *.ts.net MagicDNS name. Scheme and
+//! :port are stripped before the test. NOTE: bare MagicDNS short names ("lake-server")
+//! are NOT matched — those still need tailscale_quack_forward.
+bool IsTailnetHost(const string &proto_host_port);
+
+//! Install TailscaleHTTPUtil as the database's global HTTP util, wrapping whatever util is
+//! currently registered (httpfs, after auto-load). Idempotent: a second call is a no-op. Called
+//! from tailscale_up (after the node is up) and from tailscale_login (the wrap is inert until the
+//! node comes up), unless http_route => false.
+void RegisterTailscaleHTTPUtil(DatabaseInstance &db);
+
+//! HTTP/1.1 client that speaks plaintext over a tailscale_dial'd fd to a tailnet peer (the tailnet
+//! is the encryption layer, so only http:// is routed here — https:// is left to httpfs). Holds
+//! one keep-alive connection open across requests and frames responses by Content-Length, chunked
+//! transfer-encoding, or read-to-EOF (which then closes the connection); a stale pooled connection
+//! is transparently redialed once for idempotent requests.
+class TailscaleHTTPClient : public HTTPClient {
+public:
+	explicit TailscaleHTTPClient(const string &proto_host_port);
+	~TailscaleHTTPClient() override;
+
+	void Initialize(HTTPParams &http_params) override;
+
+	unique_ptr<HTTPResponse> Get(GetRequestInfo &info) override;
+	unique_ptr<HTTPResponse> Post(PostRequestInfo &info) override;
+	unique_ptr<HTTPResponse> Put(PutRequestInfo &info) override;
+	unique_ptr<HTTPResponse> Head(HeadRequestInfo &info) override;
+	unique_ptr<HTTPResponse> Delete(DeleteRequestInfo &info) override;
+
+	//! Parsed HTTP response. Public so the file-local ToHTTPResponse() helper can move from it.
+	struct ParsedResponse {
+		int status = 0;
+		string reason;
+		HTTPHeaders headers;
+		string body;
+		//! The peer signalled (or the framing implies) that this connection must not be reused.
+		bool connection_close = false;
+	};
+
+private:
+	//! Send one request and read the full response, redialing once if a *reused* keep-alive
+	//! connection turns out to be dead. `has_body` requests carry Content-Length framing.
+	ParsedResponse RoundTrip(const string &method, const string &path, const HTTPHeaders &headers,
+	                         const_data_ptr_t body, idx_t body_len, bool has_body);
+
+	void CloseConn();
+	bool ReadMore();                            //!< append one chunk of socket data into rx; false on EOF/error
+	bool ReadResponse(const string &method, ParsedResponse &out);
+	bool ReadChunkedBody(string &body);
+
+	string host;            //!< parsed from base_url, scheme + port stripped
+	string port;            //!< defaults to "80" if the URL omits it
+	int fd = -1;            //!< persistent keep-alive connection, -1 == not connected
+	string rx;              //!< bytes read from fd but not consumed; empty on a fresh dial, carried across reused requests
+	bool read_error = false; //!< last ReadMore hit a socket error (vs a clean EOF); reset per response
+	idx_t timeout_seconds = 30;
+};
+
+//! Global HTTP util that intercepts tailnet hosts and delegates everything else to the
+//! previously-registered util (httpfs), preserving real TLS / proxies / secrets and its
+//! keep-alive cache for non-tailnet traffic. Maintains a small idle pool of tailnet
+//! clients so keep-alive survives across DuckDB file handles.
+class TailscaleHTTPUtil : public HTTPUtil {
+public:
+	explicit TailscaleHTTPUtil(HTTPUtil &prev) : prev(prev) {
+	}
+
+	string GetName() const override {
+		return "Tailscale";
+	}
+
+	unique_ptr<HTTPParams> InitializeParameters(DatabaseInstance &db, const string &path) override {
+		return prev.InitializeParameters(db, path);
+	}
+	unique_ptr<HTTPParams> InitializeParameters(ClientContext &context, const string &path) override {
+		return prev.InitializeParameters(context, path);
+	}
+	unique_ptr<HTTPParams> InitializeParameters(optional_ptr<FileOpener> opener,
+	                                            optional_ptr<FileOpenerInfo> info) override {
+		return prev.InitializeParameters(opener, info);
+	}
+
+	unique_ptr<HTTPClient> InitializeClient(HTTPParams &http_params, const string &proto_host_port) override;
+	void CloseClient(unique_ptr<HTTPClient> &&client) override;
+
+private:
+	//! Reference to the previous global util. After SetHTTPUtil() the old util is retained
+	//! in DBConfig (old_http_utils) for the DB lifetime, so this ref stays valid.
+	HTTPUtil &prev;
+
+	//! Idle keep-alive clients keyed by proto_host_port, capped at MAX_IDLE_PER_HOST each.
+	static constexpr idx_t MAX_IDLE_PER_HOST = 8;
+	std::mutex pool_mutex;
+	unordered_map<string, vector<unique_ptr<HTTPClient>>> idle_clients;
+};
+
+} // namespace duckdb