Ducklake support

.dockerignore

@@ -0,0 +1,12 @@
+# Host build outputs — never send macOS/other host artifacts into Linux builds.
+/build/
+build/
+**/build/
+.cache/
+**/.cache/
+.e2e-work/
+examples/.e2e-work/
+
+# Editor / OS noise
+**/.DS_Store
+**/*~

.github/workflows/headscale-e2e.yml

@@ -1,4 +1,4 @@
-# QuackTail e2e over Headscale — one job, Headscale service + concurrent DuckDB workers.
+# QuackTail e2e over Headscale — manual only; uses GitHub release binaries (no source build).
 name: Headscale QuackTail e2e
 
 on:
@@ -22,7 +22,7 @@ env:
 
 jobs:
   quacktail-e2e:
-    name: QuackTail e2e (Headscale + server + client)
+    name: QuackTail e2e (release binary + Headscale)
     runs-on: ubuntu-latest
     timeout-minutes: 30
     permissions:

.github/workflows/headscale-integration.yml

@@ -12,6 +12,8 @@ on:
       - 'src/**'
       - 'cmake/**'
       - 'third_party/libtailscale/**'
+      - 'examples/Dockerfile'
+      - 'scripts/e2e/**'
 
 env:
   HEADSCALE_HOST: headscale
@@ -38,7 +40,7 @@ jobs:
       - name: Install build dependencies
         run: |
           sudo apt-get update
-          sudo apt-get install -y build-essential cmake ninja-build ccache curl
+          sudo apt-get install -y build-essential cmake ninja-build patch ccache curl
 
       - name: Start Headscale
         run: |

CMakeLists.txt

@@ -14,7 +14,7 @@ set(CMAKE_CXX_STANDARD_REQUIRED ON)
 
 include_directories(src/include)
 
-set(EXTENSION_SOURCES src/quackscale_extension.cpp src/tailscale_bridge.cpp src/tailscale_forwarder.cpp src/tailscale_log_capture.cpp)
+set(EXTENSION_SOURCES src/quackscale_extension.cpp src/attach_ducklake.cpp src/tailscale_bridge.cpp src/tailscale_forwarder.cpp src/tailscale_log_capture.cpp)
 
 if(QUACKSCALE_WITH_TAILSCALE)
     include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/Libtailscale.cmake)

docs/AUTHENTICATION.md

@@ -1,163 +1,225 @@
-# Tailscale authentication (QuackScale)
+# Authentication
 
-This document covers **only Tailscale** — getting a DuckDB process onto your tailnet.
+QuackTail uses **two independent credential layers**. Both matter in production unless you deliberately relax Quack auth on a locked-down tailnet.
 
-For **Quack HTTP tokens** (shared secrets between QuackTail servers and clients), see **[QUACK_AUTH.md](QUACK_AUTH.md)**. You need both layers in production.
+| Layer | Question | Configure with |
+|-------|----------|----------------|
+| **Tailnet** | Is this process on our mesh? | `TS_AUTHKEY`, Headscale preauth key, or browser login → `CALL tailscale_up` |
+| **Quack** | May this caller run SQL over HTTP? | `QUACK_TAILNET_TOKEN`, `CREATE SECRET`, or custom auth macro |
 
-| Doc | Topic |
-|-----|--------|
-| [QUACK_AUTH.md](QUACK_AUTH.md) | Shared `QUACK_TAILNET_TOKEN`, `quack_token()`, `CREATE SECRET`, overriding `quack_authentication_function` |
-| [PLAN.md](PLAN.md) | Architecture and roadmap |
-| [../README.md](../README.md) | Quick start and SQL reference |
-
-## How it fits QuackTail
-
-```
-  Client                                    Server
-    │                                         │
-    │  ① Tailscale (wire)                     │  CALL tailscale_up
-    │     TS_AUTHKEY / login                  │  → node on tailnet
-    │                                         │
-    │  ② Quack HTTP :9494                     │  CALL quack_serve(..., token => quack_token())
-    │     QUACK_TAILNET_TOKEN                 │  → SQL API on tailnet IP
-    └─────────────────────────────────────────┘
-```
-
-Tailscale ACLs control **who can open TCP to port 9494**. Quack tokens control **who may run SQL** once connected. See [Quack security](https://duckdb.org/docs/current/quack/security).
+Tailnet ACLs control **who can open TCP to port 9494**. Quack tokens control **who may execute SQL** once connected. See [Quack security](https://duckdb.org/docs/current/quack/security).
 
 ---
 
-QuackScale embeds [libtailscale](https://github.com/tailscale/libtailscale) (Go **tsnet**). Joining a tailnet matches other embedded Tailscale apps: **auth keys**, **environment variables**, **persisted state**, or **interactive browser login**.
+## Tailnet login (Tailscale SaaS)
 
-## How tsnet authenticates
+QuackScale embeds [libtailscale](https://github.com/tailscale/libtailscale) (tsnet). Joining matches other embedded Tailscale apps.
 
 | Mode | How | Best for |
 |------|-----|----------|
 | **Auth key** | `authkey` in `CALL tailscale_up`, or `TS_AUTHKEY` env | Servers, CI, automation |
-| **Persisted state** | `state_dir` — keys on disk after first login | Laptops, repeat use |
-| **Interactive login** | Login URL in logs; open in browser | First-time dev setup |
-| **Headscale** | `control_url` → your [Headscale](https://github.com/juanfont/headscale) URL + Headscale preauth key | Self-hosted tailnet (Tailscale-compatible) |
-| **Test control** | `control_url` → [tstestcontrol](https://github.com/tailscale/libtailscale/tree/main/tstestcontrol) | libtailscale unit tests |
+| **Persisted state** | `state_dir` on disk after first login | Laptops, repeat use |
+| **Browser login** | `CALL tailscale_login` → open `login_url` | First-time dev setup |
+
+### Production server
+
+```sh
+export TS_AUTHKEY='tskey-auth-...'
+```
 
-The libtailscale C API exposes `tailscale_set_authkey`, `tailscale_set_dir`, `tailscale_set_control_url`, `tailscale_set_logfd`, and `tailscale_up`. There is no C API that returns a login URL directly — tsnet prints `https://login.tailscale.com/a/…` on the **log stream** (see [libtailscale Python README](https://github.com/tailscale/libtailscale/blob/main/python/README.md)).
+```sql
+LOAD quackscale;
 
-Reference: [tsnet.Server · Tailscale Docs](https://tailscale.com/kb/1522/tsnet-server).
+CALL tailscale_up(
+    hostname => 'analytics-hub',
+    state_dir => '/var/lib/duckdb/tailscale'
+);
+```
 
-## Loopback forward (Quack HTTP over the tailnet)
+Do not commit auth keys in SQL — use env or your secret store.
 
-Embedded tsnet can dial peers (`tailscale_ping`), but **Quack uses normal HTTP/TCP**. Kernel sockets cannot reach tailnet IPs without help.
+### Developer laptop
 
-The native libtailscale path ([tsnetctest](https://github.com/tailscale/libtailscale/blob/main/tsnetctest/tsnetctest.go)) uses `tailscale_dial`. QuackScale exposes that for Quack via a **localhost TCP forwarder** — no SOCKS, no `ALL_PROXY`:
+`CALL tailscale_up()` **blocks** until login completes. For a non-blocking flow:
 
 ```sql
-CALL tailscale_up(hostname => 'my-client', authkey => '...', state_dir => '/var/lib/duckdb/ts');
-CALL tailscale_quack_forward(host => 'peer-hostname', port => 9494, local_port => 19494);
--- quack_uri => quack:127.0.0.1:19494
-
-CREATE SECRET (TYPE quack, TOKEN '...', SCOPE 'quack:127.0.0.1:19494');
-ATTACH 'quack:127.0.0.1:19494' AS remote (TYPE quack, DISABLE_SSL true);
+CALL tailscale_login(
+    hostname => 'my-laptop',
+    state_dir => '~/.local/share/duckdb/quackscale'
+);
+CALL tailscale_login_status();  -- poll until status = 'up'
 ```
 
-`tailscale_quack_forward` listens on `127.0.0.1:local_port` and dials `host:port` over tsnet for each Quack HTTP connection.
+Open `login_url` in a browser. Reuse `state_dir` on later runs.
+
+### Environment variables (tailnet)
+
+| Variable | Effect |
+|----------|--------|
+| `TS_AUTHKEY` | Auth key if not passed in `CALL tailscale_up` |
+| `TSNET_FORCE_LOGIN` | Force browser login even when an auth key is set (rare) |
 
-Legacy: `CALL tailscale_quack_proxy()` (SOCKS + `ALL_PROXY`) remains but is deprecated.
+---
 
-## Recommended patterns
+## Headscale (self-hosted control plane)
 
-### Production / servers — auth key
+[Headscale](https://github.com/juanfont/headscale) implements the Tailscale control server API. QuackScale uses the same parameters as `tailscale up --login-server`:
 
-Create a [reusable or ephemeral auth key](https://tailscale.com/kb/1085/auth-keys), then:
+| Tailscale CLI | QuackScale |
+|---------------|------------|
+| `--login-server https://hs.example.com` | `control_url => 'https://hs.example.com'` |
+| `--authkey …` | `authkey => '…'` or `TS_AUTHKEY` |
+| `--hostname` | `hostname => '…'` |
+| state directory | `state_dir => '…'` |
+
+Create Headscale preauth keys with `headscale preauthkeys create` (not the Tailscale admin UI).
 
 ```sh
-export TS_AUTHKEY='tskey-auth-...'
+headscale users create quackscale
+headscale preauthkeys create --user 1 --reusable --expiration 168h
 ```
 
 ```sql
-LOAD quackscale;
-
 CALL tailscale_up(
-    hostname => 'analytics-duck-1',
-    state_dir => '/var/lib/duckdb/tailscale'
+    hostname => 'duckdb-node-a',
+    control_url => 'https://headscale.example.com',
+    authkey => '<headscale preauth key>',
+    state_dir => '/var/lib/duckdb/headscale-state'
 );
 ```
 
-Or pass the key in SQL: `CALL tailscale_up(authkey => 'tskey-auth-...', ...)`.
+**Compose demo:** control URL `http://headscale:8080`, preauth key written to `/work/authkey`. See [examples/README.md](../examples/README.md).
 
-Do not commit auth keys in SQL files — use env or your orchestrator’s secret store.
+**Notes:** Production `server_url` should be HTTPS. MagicDNS is optional; `quack_uri()` prefers MagicDNS when available, else tailnet IP.
 
-### Developer laptop — browser login
+---
 
-`CALL tailscale_up()` **blocks** until login completes. For a non-blocking flow:
+## Quack HTTP tokens
+
+After a node is on the tailnet, Quack still requires application-level auth.
+
+### Default Quack behavior (why you override it)
+
+`CALL quack_serve(...)` generates a **random** token unless you pass `token => '...'`. That is fine for local experiments; **fleets need a shared token or allowlist**.
+
+QuackScale provides `quack_token()` to read a shared secret from the environment on the **server**. Clients use the same value via `CREATE SECRET` or `TOKEN`.
+
+### Environment variables (Quack)
+
+Set on **both** servers and clients:
+
+| Variable | Role |
+|----------|------|
+| `QUACK_TAILNET_TOKEN` | **Preferred** — shared token (≥ 4 characters) |
+| `QUACK_TOKEN` | Fallback if `QUACK_TAILNET_TOKEN` is unset |
+
+Keep **`TS_AUTHKEY`** separate from Quack tokens.
+
+---
+
+## Quack auth modes
+
+### Mode 1 — Single shared token (recommended)
+
+**Server:**
 
 ```sql
+LOAD quack;
 LOAD quackscale;
 
-CALL tailscale_login(
-    hostname => 'my-laptop-duckdb',
-    state_dir => '~/.local/share/duckdb/quackscale'
+CALL tailscale_up(hostname => 'warehouse-a', state_dir => '…');
+
+CALL quack_serve(
+    'quack:127.0.0.1:9494',
+    allow_other_hostname => true,
+    token => quack_token()
 );
--- Returns status, login_url, message
+CALL tailscale_serve_local(port => 9494);
+```
 
-CALL tailscale_login_status();  -- poll until status = 'up'
+**Client** (after `tailscale_quack_forward` — see [GUIDE.md](GUIDE.md)):
+
+```sql
+LOAD quack;
+
+CREATE SECRET (
+    TYPE quack,
+    TOKEN 'your-shared-quack-secret',
+    SCOPE 'quack:127.0.0.1:19494'
+);
+
+ATTACH 'quack:127.0.0.1:19494' AS remote (TYPE quack, DISABLE_SSL true);
 ```
 
-Open `login_url` in a browser and approve the device. tsnet may also print the same URL on DuckDB stderr.
+`SCOPE` must match how the client reaches the server. With the forwarder, that is `quack:127.0.0.1:<local_port>`.
+
+**Stateless queries:**
 
-After the first login, reuse `state_dir`; later `CALL tailscale_up()` usually needs no browser.
+```sql
+FROM quack_query(
+    'quack:127.0.0.1:19494',
+    'SELECT 42',
+    token => 'your-shared-quack-secret',
+    disable_ssl => true
+);
+```
 
-### Self-hosted — Headscale
+### Mode 2 — Token allowlist (rotation / teams)
 
-[Headscale](https://github.com/juanfont/headscale) implements the Tailscale control server API. QuackScale uses the same knobs as the Tailscale CLI:
+Use Quack’s [multi-token table](https://duckdb.org/docs/current/quack/security#example-multi-token-table):
 
 ```sql
-CALL tailscale_up(
-    hostname => 'my-node',
-    control_url => 'https://headscale.example.com',
-    authkey => '<headscale preauth key>',
-    state_dir => '/var/lib/duckdb/headscale-state'
+CREATE TABLE quacktail_tokens (auth_token VARCHAR PRIMARY KEY, label VARCHAR);
+INSERT INTO quacktail_tokens VALUES ('primary-2026', 'analytics');
+
+CREATE MACRO quacktail_check_token(sid, client_token, server_token) AS (
+    EXISTS (SELECT 1 FROM quacktail_tokens WHERE auth_token = client_token)
 );
+SET GLOBAL quack_authentication_function = 'quacktail_check_token';
 ```
 
-Create keys with `headscale preauthkeys create`. Full walkthrough: **[HEADSCALE.md](HEADSCALE.md)** and [examples/headscale_quacktail.sql](../examples/headscale_quacktail.sql).
+Validate **`client_token`** (what the caller sent), not `server_token`.
 
-### CI / tests
+### Mode 3 — Developer mode (lab only)
 
-| Workflow | Control plane |
-|----------|----------------|
-| [headscale-integration.yml](../.github/workflows/headscale-integration.yml) | Docker Headscale + `CALL tailscale_up` |
-| [headscale-e2e.yml](../.github/workflows/headscale-e2e.yml) | Two-node QuackTail e2e (linux, manual dispatch) |
-| [libtailscale-integration.yml](../.github/workflows/libtailscale-integration.yml) | libtailscale `tstestcontrol` (`go test`) |
+```sql
+CREATE MACRO quacktail_dev_auth(sid, client_token, server_token) AS true;
+SET GLOBAL quack_authentication_function = 'quacktail_dev_auth';
+```
 
-## SQL surface (Tailscale only)
+**Not for production.** See [Quack developer mode](https://duckdb.org/docs/current/quack/security#example-developer-mode-always-allow).
 
-Invoke with **`CALL`**, like Quack:
+---
 
-| Command | Purpose |
-|---------|---------|
-| `CALL tailscale_up(...)` | Blocking join; `authkey` or `TS_AUTHKEY`; optional `state_dir`, `control_url`, `ephemeral` |
-| `CALL tailscale_login(...)` | Background join; returns `login_url` |
-| `CALL tailscale_login_status()` | Poll `status`, `login_url`, tailnet IPs |
-| `CALL tailscale_status()` | Linked?, running, hostname, IPs |
+## End-to-end checklist
 
-## Environment variables
+**Each long-lived server**
 
-| Variable | Effect |
-|----------|--------|
-| `TS_AUTHKEY` | Tailscale auth key if not passed in `CALL tailscale_up` |
-| `TSNET_FORCE_LOGIN` | Force interactive login even if an auth key is set (rare) |
+1. `export TS_AUTHKEY` (or Headscale preauth key) and `export QUACK_TAILNET_TOKEN`
+2. `LOAD quack; LOAD quackscale;`
+3. `CALL tailscale_up(...)` with persistent `state_dir`
+4. Optional: `SET GLOBAL quack_authentication_function` (Modes 2–3)
+5. `CALL quack_serve(..., token => quack_token()); CALL tailscale_serve_local(port => 9494);`
+6. Do **not** call `tailscale_down()` on steady-state servers
+
+**Each one-shot client**
 
-**Quack tokens are separate:** `QUACK_TAILNET_TOKEN` / `QUACK_TOKEN` — see [QUACK_AUTH.md](QUACK_AUTH.md).
+1. Same `QUACK_TAILNET_TOKEN` available for secrets / `quack_query`
+2. `LOAD quackscale; CALL tailscale_up(...); CALL tailscale_quack_forward(...);`
+3. `LOAD quack; CREATE SECRET ...;` then query / attach
+4. `DETACH remote; SELECT 'done'; CALL tailscale_down();` — required or the process hangs
+
+---
 
-## Security notes
+## Security
 
-- Treat `TS_AUTHKEY` like any infrastructure secret.
-- Tailnet [ACLs](https://tailscale.com/kb/1018/acls) should restrict who can reach peer TCP **9494** (Quack).
-- QuackScale advertises `quack:` URIs; it does not replace Quack’s application-level auth.
+- Rotate `QUACK_TAILNET_TOKEN` like an API key; update servers and clients together
+- Restrict tailnet ACLs to who may reach peer TCP **9494**
+- `allow_other_hostname => true` is for tailnet binds — do not expose raw Quack on the public internet without TLS in front ([Quack exposure model](https://duckdb.org/docs/current/quack/security#exposure-model))
 
-## Related reading
+## References
 
-- [QUACK_AUTH.md](QUACK_AUTH.md) — Quack / QuackTail application tokens
-- [HEADSCALE.md](HEADSCALE.md) — self-hosted Headscale
-- [libtailscale](https://github.com/tailscale/libtailscale)
-- [Headscale](https://github.com/juanfont/headscale)
+- [Quack security](https://duckdb.org/docs/current/quack/security)
+- [Quack overview — Authentication](https://duckdb.org/docs/current/quack/overview#authentication)
 - [Tailscale auth keys](https://tailscale.com/kb/1085/auth-keys)
+- [Headscale docs](https://headscale.net/)