From 492b405846ac122827edc0688189e7a0e088dce2 Mon Sep 17 00:00:00 2001 From: Aaron Brethorst Date: Sun, 10 May 2026 12:06:47 -0700 Subject: [PATCH] docs: reshuffle README around tasks; split quickstart into deployment + ops MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reorient the project's front door around the four tasks a new reader actually has: understand what hooks is, deploy the server, use hooksctl, contribute (including adding new providers). Lead the deploy section with the Docker path, since that's the supported shape. Split the single quickstart into two reference docs: - docs/quickstart.md → docs/deployment.md (env vars, `hooks init` flags, container internals, Render Blueprint specifics, skew-window semantics). One-time setup reference. - docs/operations.md → docs/running-in-production.md (backups, retention, observability, push-subscription health, restarts and signing-secret state, graceful shutdown). Day-2 ops. The split is clean — the two docs cover disjoint topics, and the README cross-references each at the right seam. Specifics worth calling out, accumulated across two review rounds: - "Try it locally first" exports RENDER_WEBHOOK_SECRET (without it `./bin/hooks` fails immediately with "empty secret") and points the reader at `hooksctl tail` and the inspector URL so they can verify the setup is working. Notes the password policy (≥ 12 chars, no email substring) so signup doesn't bounce them to accounts.md. - `make dev` is described accurately: it uses the configured listen addr (default :8080), not a random free port, and does not run init. - Section 2 leads with a pointer to docs/accounts.md as the full developer-onboarding walkthrough, not the admin-ops afterthought. - The "admin token" is described as a system-level credential that predates the user-account system, not a "legacy" credential. - Option B opens with "export RENDER_WEBHOOK_SECRET first" so the `-e` passthrough has a value. TLS-terminator reminders consolidated to the canonical mention at the top of the page. - `hooksctl replay` example takes the positional and --to URL. - rotate-secret callout distinguishes `me sub` (self) from `push` (admin-wide) for restart recovery. - Provider section collapsed to a teaser + pointer to docs/sources.md, with Stripe/GitHub/Vercel as illustrative examples and the --profile flag mentioned inline. - accounts.md cross-references repointed; example hostnames aligned on webhooks.example.com; password-policy wording tightened to match the ≥ 3-char local-part gate in internal/users/policy.go. - running-in-production.md explains the `me sub *` vs `push *` split and shows `hooksctl push test` for smoke-testing a consumer. Clarifies that `/push` is an inspector page. - deployment.md: admin token's *plaintext* is unrecoverable (the Argon2id hash lives in the DB); --token-name added to the init-flags list; "Useful flags" renamed to "Notable flags" so the list isn't misread as exhaustive. - render.yaml header comment repointed at the README's Option A. --- README.md | 236 +++++++++++------- docs/accounts.md | 20 +- docs/deployment.md | 92 +++++++ docs/quickstart.md | 169 ------------- ...operations.md => running-in-production.md} | 24 +- render.yaml | 5 +- 6 files changed, 261 insertions(+), 285 deletions(-) create mode 100644 docs/deployment.md delete mode 100644 docs/quickstart.md rename docs/{operations.md => running-in-production.md} (67%) diff --git a/README.md b/README.md index 15943ec..963a46a 100644 --- a/README.md +++ b/README.md @@ -1,167 +1,209 @@ -## hooks +# hooks -A small, self-hosted relay that durably captures inbound webhooks (Render to start), verifies their signatures, and re-delivers them to one or more developer environments — either pulled over Server-Sent Events or pushed to a registered URL — including replay of anything missed while disconnected, bounded by the source's signature-verification skew window so consumers don't reject stale catch-up traffic. (Older events stay in the store and remain available via the inspector's "Replay to listeners" action and `hooksctl replay`.) +A self-hosted webhook relay. `hooks` receives HMAC-signed inbound webhooks from providers like Render, Stripe, or your own services, persists them to SQLite, and re-delivers them to your developer environments — either pulled over Server-Sent Events with `hooksctl forward`, or pushed to a registered URL. Anything missed while a consumer was disconnected is replayed on reconnect. -To get started: `hooks init`. +The point: never lose a webhook because a laptop was asleep, a deploy was rolling, or a consumer service was down. One small Go binary, one SQLite file, no external dependencies. -## Test it end-to-end with Render +## Status -Walks you from a fresh checkout to a real Render webhook landing on your laptop. About ten minutes. For the production-deployment version of this flow, see [`docs/quickstart.md`](docs/quickstart.md). +- One process, SQLite-backed. Running two `hooks` processes against the same database is unsafe. +- One built-in provider today: **Render** (Standard Webhooks). Adding more is a short patch — see [Contributing](#3-contributing). +- Apache 2.0 licensed. -### Prerequisites +## How it fits together -- Go 1.25+ (for `make build`). -- A Render account with a service or other resource that emits webhooks (a deploy is the easiest trigger). -- A tunnel that gives you a public HTTPS URL pointing at `localhost:8080`. Render refuses plain HTTP and will not POST to a non-public address. Pick one: - - `cloudflared tunnel --url http://localhost:8080` - - `ngrok http 8080` - - any reverse proxy you already have on a real domain +```mermaid +flowchart LR + provider[Provider] -->|"POST /ingest/<source>"| verify[Verify HMAC] + verify --> store[(SQLite)] + store --> sse["SSE: hooksctl forward"] + store --> push["HTTP push to your URL"] + store --> inspector["Inspector UI (browser)"] +``` -### 1. Build the binaries +The relay binds plain HTTP and is meant to sit behind a TLS-terminating proxy (Caddy, nginx, Cloudflare, Render, fly.io — anything). Providers refuse non-HTTPS endpoints, so this is enforced by the outside world either way. -```sh -git clone https://github.com/onebusaway/hooks -cd hooks -make build -``` +--- -You now have `./bin/hooks` (the relay) and `./bin/hooksctl` (the developer CLI). +## Try it locally first -### 2. Scaffold a deployment +To kick the tires without deploying anything: ```sh -./bin/hooks init +make build # ./bin/hooks and ./bin/hooksctl +./bin/hooks init --server-url http://localhost:8080 +export RENDER_WEBHOOK_SECRET=devsecret # any non-empty value; you won't be receiving real Render deliveries +./bin/hooks # leave running; this is the relay ``` -This writes `hooks.yaml`, creates `hooks.db`, and prints an admin token **once**. Copy it now — there is no way to recover it later. On a fresh DB it also prints a one-time **signup URL** (24-hour TTL) so the first human can claim an admin account through `/signup`. +`init` prints a one-time signup URL and a one-time admin token. Open the signup URL in a browser and claim the first admin account (password ≥ 12 characters and not containing your email). Ignore the admin token for the local demo — it's a separate break-glass credential, explained in [Option B below](#option-b--run-the-container-yourself). Then in another terminal: -``` -admin token (shown ONCE): -signup: http://localhost:8080/signup?code= (24h, single-use) +```sh +./bin/hooksctl login --server http://localhost:8080 --scopes render +./bin/hooksctl forward render --to http://localhost:3000/webhooks/render +./bin/hooksctl tail render # watch events arrive in your terminal ``` -Export it for `hooksctl`: +You can also browse the inspector at to see captured events. -```sh -export HOOKS_TOKEN= -``` +To get real provider webhooks landing on your laptop, expose `:8080` to the public internet with [ngrok](https://ngrok.com), [Cloudflare Tunnel](https://www.cloudflare.com/products/tunnel/), or similar, and point the provider's webhook at `https:///ingest/render`. -### 3. Create the webhook in Render +After running `init` once, `make dev` is a faster loop on subsequent runs — it starts the server with debug logging and opens the inspector at . -In the Render dashboard, create a new webhook. For the URL, put a placeholder for now (e.g. `https://example.invalid/ingest/render`) — you will update it in step 5 once the tunnel is running. Render will display a **signing secret**; copy it. +--- -```sh -export RENDER_WEBHOOK_SECRET= -``` +## 1. Deploy the server -The default `hooks.yaml` already references this env var: +The supported deployment path is the included `Dockerfile`. Both binaries (`hooks` and `hooksctl`) ship in the image, so token rotation, push-subscription management, and pruning all work via `docker exec`. -```yaml -sources: - render: - verifier: render - secret: ${RENDER_WEBHOOK_SECRET} - retention: 30d -``` +### Option A — Render Blueprint (one-click) + +The repo includes a `render.yaml` Blueprint. In the Render dashboard: + +1. **New → Blueprint**, point it at this repo (fork first if you want autoDeploy on your own pushes). +2. In the service's **Environment** tab, set: + - `HOOKS_PUBLIC_URL` *(optional, recommended)* — your service's external URL, e.g. `https://hooks-abc1.onrender.com`. Used so the bootstrap signup URL printed at first boot points at your real host. Skip it and the URL prints with a `localhost` placeholder you'll have to swap by hand. + - `RENDER_WEBHOOK_SECRET` — set any placeholder for now; replace it with the real value in step 4 +3. Trigger a deploy. The container's entrypoint detects an empty `/data`, runs `hooks init` automatically, and prints both a **bootstrap signup URL** (24h, single-use) and a one-time **admin token** to the service **Logs**. Copy both. +4. Open the signup URL in a browser and claim the first admin account. Then create a webhook in Render pointing at `https:///ingest/render`, copy the signing secret it gives you, and set `RENDER_WEBHOOK_SECRET` to that value. + +### Option B — Run the container yourself -### 4. Start the relay +Export `RENDER_WEBHOOK_SECRET` (the value Render's dashboard prints when you create the webhook) in your shell first — the `docker run` below passes it through: ```sh -./bin/hooks --dev -``` +make docker-build # builds hooks:dev +mkdir -p ./hooks-data -`--dev` enables verbose logging, opens the inspector in your browser, and prints the URLs you'll need: +docker run --rm -v $(pwd)/hooks-data:/data hooks:dev init \ + --server-url https://webhooks.example.com -``` -inspector: http://localhost:8080/ -ingest: http://localhost:8080/ingest/render -forward: hooksctl forward render --to http://localhost:3000/webhooks/render +docker run -d --name hooks --restart=unless-stopped \ + -p 8080:8080 \ + -v $(pwd)/hooks-data:/data \ + -e RENDER_WEBHOOK_SECRET \ + -e HOOKS_PUBLIC_URL=https://webhooks.example.com \ + hooks:dev ``` -The inspector tab will land on `/login`. To claim your admin account, open the signup URL from step 2 (`http://localhost:8080/signup?code=...`) in the same browser, pick an email + password, and you'll be signed in to the inspector. The `HOOKS_TOKEN` you exported is for `hooksctl`, not the browser. +`HOOKS_PUBLIC_URL` is optional — it only controls the host printed in the bootstrap signup URL. Leave it unset and the URL prints with a `localhost` placeholder you can swap by hand. -Leave the relay running. +The image runs as a non-root user, mounts `/data` as a volume for the SQLite database, listens on `:8080`, and a Dockerfile-level `HEALTHCHECK` polls `/healthz`. Wire your load balancer's health check to `/readyz` (it pings SQLite end-to-end; `/healthz` is liveness-only). TLS termination is on you — see [How it fits together](#how-it-fits-together) above. -### 5. Open a public HTTPS tunnel to it +> **`/data` must be a persistent volume.** The SQLite database lives there, and so does every captured event, every account, and every push subscription. If `/data` is the container's writable layer (no `-v` / no platform-managed disk), every restart nukes it and you start from scratch. The Render Blueprint above provisions a 1 GiB persistent disk; for `docker run`, the `-v $(pwd)/hooks-data:/data` bind mount is what keeps the DB alive. -In a second terminal: +The first `init` invocation prints a **bootstrap signup URL** (24h) and an **admin token** to stdout. Save both — neither is recoverable. Open the signup URL in a browser to claim the first admin account. The admin token is a separate system-level credential that predates the user-account system; keep it for break-glass access, or revoke it later with `hooksctl token revoke ` once your PAT is working. It is not tied to any user account. + +### Option C — Bare binary + +If you'd rather not run a container: ```sh -cloudflared tunnel --url http://localhost:8080 +make build +./bin/hooks init --server-url https://webhooks.example.com +./bin/hooks ``` -Copy the `https://.trycloudflare.com` URL it prints. Back in the Render dashboard, edit your webhook and set its URL to: +Same caveats — set `RENDER_WEBHOOK_SECRET` (and any other provider secrets) in the environment. -``` -https://.trycloudflare.com/ingest/render -``` +For more deployment detail (Render Blueprint internals, env-var precedence, container entrypoint behavior, skew-window semantics), see [`docs/deployment.md`](docs/deployment.md). For day-2 ops (backups, pruning, observability, restarts and signing-secret state, graceful shutdown), see [`docs/running-in-production.md`](docs/running-in-production.md). -### 6. Forward events to a local app +--- -In a third terminal, point `hooksctl forward` at whichever local service you're developing: +## 2. Use `hooksctl` against a deployed relay + +Once the server is up and you have an account, point `hooksctl` at it from your laptop. This section is the quick tour; for the full developer-onboarding walkthrough — invites, ephemeral vs long-lived listener tokens, push-subscription details, deactivation semantics — see [`docs/accounts.md`](docs/accounts.md). ```sh -./bin/hooksctl forward render --to http://localhost:3000/webhooks/render +hooksctl login --server https://webhooks.example.com --scopes render ``` -`forward` first replays any events you missed (none on first run), then tails live. Replay is bounded by the source's signature-verification skew window (5 minutes for Render by default), so events older than that are skipped during the initial catch-up and your local app won't 401 on a stale `webhook-timestamp`. Older events remain in the store and can be redelivered manually via the inspector or `hooksctl replay`. Bytes hitting your local app are byte-for-byte identical to what Render sent — original headers preserved. +`login` runs a device-pairing flow: it prints a short code, opens the relay's `/device` page, and asks you to log in and re-enter your password to approve the pairing. On success it writes a personal access token to `~/.config/hooks/credentials.default` (mode `0600`). Default scope is `account` only; pass `--scopes render,stripe,...` to also subscribe, or `--admin` for admin scope. Pass `--profile ` to keep multiple servers configured side-by-side (e.g. `staging` and `prod`); the default profile is `default`. -### 7. Trigger a webhook from Render - -The fastest trigger is a redeploy of any Render service: `Manual Deploy → Deploy latest commit`. Other event types (suspends, scaling) work too. +```sh +hooksctl whoami # confirm the login worked +``` -You should see, in order: +### Forward live events to a local app -- A `POST /ingest/render` log line in the `hooks --dev` terminal. -- A new row in the inspector at `http://localhost:8080/` (sign in with the admin email/password you set during `hooks init`). -- A `POST /webhooks/render` arriving at your local app via the `hooksctl forward` terminal. +```sh +hooksctl forward render --to http://localhost:3000/webhooks/render +``` -### 8. (Optional) Register a long-lived push subscription +`forward` opens an SSE stream against the relay, replays anything missed since the last cursor, then tails live. Bytes hitting your local app are byte-for-byte identical to what the provider sent — original headers preserved. Initial catch-up is bounded by the source's signature-verification skew window (5 minutes for Render by default) so your verifying consumer doesn't 401 on a stale `webhook-timestamp`. -If you want a permanent consumer instead of an SSE pull session: +### Register a long-lived consumer (HTTP push) ```sh -./bin/hooksctl push add --source render --to https://my-svc.example.com/hooks --name production +hooksctl me sub add \ + --source render \ + --to https://my-svc.example.com/hooks \ + --name production ``` -This prints a per-subscription signing secret **once** — store it on your consumer. The relay will sign every push with `X-Hooks-Signature: t=,v1=.")>`. See [`docs/consumer-verification.md`](docs/consumer-verification.md) for verification snippets in several languages. +`me sub add` prints a per-subscription **signing secret** exactly once. Store it on your consumer. The relay POSTs every event to that URL with `X-Hooks-Signature: t=,v1=.")>`. Your consumer **must** verify the signature and the timestamp window — see [`docs/consumer-verification.md`](docs/consumer-verification.md) for ready-to-paste verifiers in Go and Node. -### Running it under Docker +> The plaintext signing secret only lives in memory. After a server restart, push delivery is paused until each subscription is re-armed — `hooksctl me sub rotate-secret ` for your own, or `hooksctl push rotate-secret ` for relay-wide admin recovery. This is a deliberate trade-off — see [`docs/security.md`](docs/security.md). -A `Dockerfile` and `render.yaml` Blueprint are checked into the repo. The image is a multi-stage build (Go builder → small Alpine runtime), runs as a non-root user, and exposes `/data` as a volume for the SQLite database. +### Common subcommands ```sh -make docker-build # builds hooks:dev locally -mkdir -p ./hooks-data -docker run --rm -v $(pwd)/hooks-data:/data hooks:dev init -docker run --rm -p 8080:8080 \ - -v $(pwd)/hooks-data:/data \ - -e RENDER_WEBHOOK_SECRET \ - hooks:dev +hooksctl tail render # watch events arrive in your terminal +hooksctl replay render --to http://... # POST one historical event to a chosen URL +hooksctl me token list # list your tokens +hooksctl me sub list # list your push subscriptions +hooksctl logout # revoke local PAT and delete credentials file ``` -For a Render Blueprint deploy, push the repo and point Render at `render.yaml` — it provisions a 1 GiB persistent disk at `/data` and wires `/readyz` as the health check. See [`docs/quickstart.md`](docs/quickstart.md) for the full container walkthrough. +Admin operations (invites, user deactivation, audit log) live in the inspector at `/users` and `/audit`. See [`docs/accounts.md`](docs/accounts.md) for the full walkthrough. + +--- -### For developers joining a deployed relay +## 3. Contributing -If your team already runs a `hooks` instance (Render or anywhere else) and you just need a CLI on your laptop, skip the first six steps. Either an admin sends you a signup URL (`https://hooks.example.com/signup?code=...`), or your relay was just deployed and an admin used the bootstrap link to create their account first. Then: +### Dev setup ```sh -hooksctl login --server https://hooks.example.com -hooksctl forward render --to http://localhost:3000/webhooks/render +git clone https://github.com/onebusaway/hooks +cd hooks +make build # builds ./bin/hooks and ./bin/hooksctl +make test # go test ./... +make lint # golangci-lint + sqlc diff + go vet +make dev # runs hooks --dev (verbose, opens inspector) ``` -`login` prints a short user code, opens the relay's `/device` page in your browser, asks you to log in and re-enter your password to approve the pairing, then writes a PAT to `~/.config/hooks/credentials.default`. `forward` uses that PAT — no further token plumbing. +Go toolchain is pinned to the version in `go.mod` (currently 1.26). The SQLite driver is pure-Go (`modernc.org/sqlite`), so cgo is not required. golangci-lint and sqlc are declared as Go `tool` dependencies, so `go tool ` builds them with the project's toolchain — no version drift. + +CI runs `go vet`, `go test -race`, and `make lint`. Match it locally with `make lint && make test` before pushing. + +### Architecture + +The server's wiring root is `internal/server.Build` — reading it end-to-end is the fastest way to understand the system. [`CLAUDE.md`](CLAUDE.md) at the repo root has a layer-by-layer tour. Conventions worth knowing up front: + +- **Body bytes are sacred.** Verifiers and push workers must never re-encode JSON or normalize whitespace; the stored bytes are what was signed. +- **Constant-time compare for any HMAC or token check.** Use `hmac.Equal`, `subtle.ConstantTimeCompare`, or the `internal/secret` helpers. +- **Logs must never contain plaintext secrets, tokens, or full webhook bodies.** On signature mismatch we log only the source name and a 4-byte hex prefix of the body's sha256. +- **HTTP status discipline at `/ingest`:** 200 for duplicate, 202 for newly accepted, 401 for verification failure, 413 for oversize, 404 for unknown source, 503 only for genuine transient store failures. + +### Adding a new webhook provider (Stripe, GitHub, Vercel, …) + +Provider verification lives behind a small `Verifier` interface in `internal/sources/sources.go`. Adding a new source means writing one Go file that implements `Verifier`, registering it in `init()`, and referencing the new name in `hooks.yaml` — zero changes to the ingest layer. [`docs/sources.md`](docs/sources.md) has the full worked Stripe example, the registry contract, and the four invariants every verifier must respect (constant-time compare, skew enforcement, body-bytes-are-sacred, stable delivery id). Start there. + +### Project workflow + +Non-trivial change planning lives in `openspec/` and the `opsx:*` skills (propose / explore / apply / archive). Issues and PRs are welcome at https://github.com/onebusaway/hooks. -See [`docs/accounts.md`](docs/accounts.md) for the full walkthrough (scopes, admin operations, multiple profiles, ephemeral vs long-lived listener tokens, deactivation semantics). +--- -### Troubleshooting +## More docs -- **HTTP 401 in the relay logs** — secret mismatch between `RENDER_WEBHOOK_SECRET` and what Render is signing with. Re-copy the signing secret from the Render dashboard. -- **HTTP 404** — the URL path is wrong; it must end in `/ingest/render`. -- **No request reaches the relay at all** — confirm the tunnel URL works in your browser (`/healthz` should return `ok`), and that you saved the updated URL in the Render dashboard. -- **The `--dev` browser tab won't authenticate** — open the signup URL printed by `hooks init` to claim an admin account first; the inspector signs you in with email/password, not the admin token. +- [`docs/deployment.md`](docs/deployment.md) — deployment reference: env vars, `hooks init` flags, container internals, Render Blueprint specifics, skew-window semantics. +- [`docs/running-in-production.md`](docs/running-in-production.md) — day-2 ops: backups, retention, observability, push-subscription health, restarts, graceful shutdown. +- [`docs/accounts.md`](docs/accounts.md) — invites, scopes, multiple profiles, ephemeral vs long-lived listener tokens, deactivation semantics. +- [`docs/security.md`](docs/security.md) — token kinds, hashing posture, signature verification, secret-handling policy, CSRF, rate limiting, audit log. +- [`docs/sources.md`](docs/sources.md) — adding a new webhook provider (worked example). +- [`docs/consumer-verification.md`](docs/consumer-verification.md) — verify push deliveries on the consumer side (Go, Node, curl). -# LICENSE +## License -(c) Open Transit Software Foundation and made available under the [Apache 2.0 license](./LICENSE). \ No newline at end of file +(c) Open Transit Software Foundation, made available under the [Apache 2.0 license](./LICENSE). diff --git a/docs/accounts.md b/docs/accounts.md index 1ecded3..f9d5b71 100644 --- a/docs/accounts.md +++ b/docs/accounts.md @@ -1,12 +1,12 @@ # Developer accounts -This walkthrough covers a deployed `hooks` relay (Render or any other host you don't shell into). For the local end-to-end demo, see the [README](../README.md). For the security-focused breakdown, see [`docs/security.md`](security.md). +This walkthrough covers a deployed `hooks` relay (Render or any other host you don't shell into). For the deployment recipe and a local-only path, see the [README](../README.md). For the security-focused breakdown, see [`docs/security.md`](security.md). The mental model: one relay deployment per team. Each developer has their own account. Listener tokens, push subscriptions, and PATs (personal access tokens) are owned by users. Deactivating a user revokes their tokens and pauses their push subscriptions in one move. ## 1. Deploy the relay -Build the binaries and bring up the server behind TLS (`docs/quickstart.md` covers the deployment shape). Whatever the host, the URL the rest of this doc uses is `https://hooks.example.com`. +Build the binaries and bring up the server behind TLS (the [README](../README.md) has the recipe; [`deployment.md`](deployment.md) has the reference). Whatever the host, the URL the rest of this doc uses is `https://webhooks.example.com`. ## 2. Bootstrap the first admin @@ -14,11 +14,11 @@ On a freshly initialized database `hooks init` prints a one-time signup URL: ``` admin token (shown ONCE): -signup: https://hooks.example.com/signup?code=ABCDEFGH... +signup: https://webhooks.example.com/signup?code=ABCDEFGH... (single-use; expires in 24h; auto-disables once any account exists) ``` -Open the signup URL in a browser. Pick an email, name, and password (≥ 12 characters; must not contain your email's local-part). The first signup consumes the bootstrap invite; once any user exists, that URL returns 409 even if someone else copies it. +Open the signup URL in a browser. Pick an email, name, and password (≥ 12 characters; must not contain your full email — or its local-part, when the local-part is at least three characters long). The first signup consumes the bootstrap invite; once any user exists, that URL returns 409 even if someone else copies it. The `admin token` printed alongside the signup URL is the legacy system credential. You can keep it for break-glass access or revoke it with `hooksctl token revoke ` once your PAT is working (see step 4). It is **not** tied to any user account. @@ -26,12 +26,12 @@ If the bootstrap link expires before it's used, re-run `hooks init` against the ## 3. Invite teammates -After login, the inspector at `/users` exposes an "Issue invite" form. Pick a role (`user` or `admin`) and a default scope set; the page shows the resulting `https://hooks.example.com/signup?code=...` URL once. Send it to your teammate. Invites are single-use. +After login, the inspector at `/users` exposes an "Issue invite" form. Pick a role (`user` or `admin`) and a default scope set; the page shows the resulting `https://webhooks.example.com/signup?code=...` URL once. Send it to your teammate. Invites are single-use. The same surface is available over JSON at `POST /api/invites` for any tool you'd rather drive programmatically: ```sh -curl -X POST https://hooks.example.com/api/invites \ +curl -X POST https://webhooks.example.com/api/invites \ -H "Authorization: Bearer $ADMIN_PAT" \ -H "Content-Type: application/json" \ -d '{"role": "user", "default_scopes": ["render"]}' @@ -40,7 +40,7 @@ curl -X POST https://hooks.example.com/api/invites \ ## 4. Get a CLI on your laptop ```sh -hooksctl login --server https://hooks.example.com +hooksctl login --server https://webhooks.example.com ``` The CLI prints a short user code (`ABCD-EFGH`) and a verification URL, and tries to open the URL in your browser. The page asks you to log in if you aren't already, then shows you the requesting client's user-agent, IP, and requested scopes. **Approval requires you to re-enter your password**, even if you're already logged in — a live session alone is not sufficient. @@ -48,8 +48,8 @@ The CLI prints a short user code (`ABCD-EFGH`) and a verification URL, and tries Default scope on approval is `account` only — enough to manage your own tokens but not enough to subscribe to webhook events. Pass `--scopes` (comma-separated source names) to request more, and `--admin` to request admin scope: ```sh -hooksctl login --server https://hooks.example.com --scopes render,stripe -hooksctl login --server https://hooks.example.com --admin +hooksctl login --server https://webhooks.example.com --scopes render,stripe +hooksctl login --server https://webhooks.example.com --admin ``` You may also narrow the scopes from the approval page itself — the CLI's request is the upper bound. Approval mints a personal access token (PAT), writes it to `${XDG_CONFIG_HOME:-$HOME/.config}/hooks/credentials.` (mode `0600`), and the next CLI call uses it automatically. @@ -119,7 +119,7 @@ The API requires a `confirm=` body field; the inspector form requires you ```sh # Move a token to a different user. -curl -X PATCH https://hooks.example.com/api/tokens/ \ +curl -X PATCH https://webhooks.example.com/api/tokens/ \ -H "Authorization: Bearer $ADMIN_PAT" \ -d '{"owner_user_id": ""}' ``` diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..afa1628 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,92 @@ +# Deployment reference + +The [README](../README.md) covers the recipe for the three supported deployment paths (Render Blueprint, `docker run`, bare binary). This doc is the reference: env vars, init flags, container internals, what each path does and doesn't set up. + +## TLS termination + +The relay binds plain HTTP. Stand any TLS-terminating reverse proxy in front of it. Providers refuse non-HTTPS endpoints, so this is enforced by the outside world either way. + +- **Caddy** — `webhooks.example.com { reverse_proxy localhost:8080 }`. Caddy obtains a Let's Encrypt cert automatically. +- **nginx** — standard `proxy_pass http://127.0.0.1:8080;` block, plus your existing TLS config. +- **Cloudflare Tunnel / Render / fly.io** — any platform that gives you HTTPS in front of an HTTP origin. + +Wire your load balancer's health check to `/readyz` (it pings SQLite end-to-end). `/healthz` is liveness-only. + +## `hooks init` + +`hooks init` is the bootstrap command. It writes `hooks.yaml`, creates `hooks.db`, mints a one-time **admin token** (the legacy system credential), and — when the users table is empty — prints a one-time **bootstrap signup URL** (24-hour TTL, single-use, auto-disables once any user exists): + +```text +admin token (shown ONCE): +signup: https://webhooks.example.com/signup?code=ABCDEFGH... + (single-use; expires in 24h; auto-disables once any account exists) +``` + +Save both. The admin token has no recovery path; the signup URL is how the first human creates their admin account. If the bootstrap link expires before it's used, re-run `hooks init --force` against the still-userless DB to mint a fresh 24-hour invite. + +Notable flags: + +- `--server-url ` (or `HOOKS_PUBLIC_URL`) — host to use when printing the signup URL. Skip it and the URL prints with a `localhost` placeholder you'll have to swap by hand. +- `--dir ` — directory for `hooks.yaml` and `hooks.db`. Used by the container entrypoint (`--dir /data`). +- `--force` — re-mint the bootstrap signup URL on a still-userless DB. Once any user exists, the bootstrap path is closed and `--force` does nothing useful. +- `--token-name ` — name for the generated admin token (default `operator`). Cosmetic only — surfaced in `hooksctl token list`. + +What `hooks init` does **not** do: + +- Stand up your reverse proxy. That's on you. +- Register the provider-side webhook. That's a step in the provider's dashboard. +- Persist the admin token plaintext or the bootstrap signup URL anywhere except standard out. Save them — neither is recoverable. (The token's Argon2id hash lives in the database, but the plaintext is shown only once.) + +## Env vars and config precedence + +Listen-address precedence: `HOOKS_LISTEN_ADDR` > yaml `listen_addr` > `:$PORT` (when `$PORT` is a valid port — for Render/Heroku/Fly/Cloud Run) > `:8080`. + +Other env vars: + +- `HOOKS_DATABASE_URL` — SQLite path. Defaults to `./hooks.db`; the Docker image overrides to `/data/hooks.db`. +- `HOOKS_LOG_LEVEL` — `debug`, `info`, `warn`, `error`. Default `info`. +- `HOOKS_PUBLIC_URL` *(optional)* — host used when printing the bootstrap signup URL. +- Provider secrets (e.g. `RENDER_WEBHOOK_SECRET`) — referenced from `hooks.yaml` via `${VAR}` interpolation. + +`hooks.yaml` supports `${VAR}` and `${VAR:-default}` interpolation. A `tokens:` field is rejected at load time — listener tokens live in the database, not in YAML. Every source must declare a `verifier:`; unsigned sources are not supported. + +Defaults: body size limit 1 MiB, dedupe window 24h, skew window 5m, retention 30d per source. Retention `0` / `forever` / `never` disables auto-prune for that source. + +## The container image + +The `Dockerfile` is a multi-stage build (Go builder → small Alpine runtime). It runs as UID 65532, mounts `/data` as a volume for the SQLite database, and ships both `hooks` and `hooksctl` so you can `docker exec` to manage tokens, push subscriptions, and pruning. + +Image defaults: + +- `HOOKS_DATABASE_URL=/data/hooks.db` +- Listens on `:8080` (or `$PORT` if set) +- A Dockerfile-level `HEALTHCHECK` polls `/healthz`. Behind a load balancer, prefer `/readyz`. + +The image's entrypoint (`docker-entrypoint.sh`) detects an empty `/data` (no `hooks.yaml` and no `hooks.db`) on first boot and runs `hooks init --dir /data` automatically. Without this, Render Blueprint deploys would crash-loop on first boot — the volume is empty, the server can't read `hooks.yaml`, and Render's Shell tab is gated on a running instance, so the documented recovery path would be unreachable. The auto-init prints the one-time admin token and bootstrap signup URL to stdout (which lands in the platform's log stream — treat both as secrets). + +Subcommands (`init`, `invite`, `prune`, `verify`, `help`) bypass the bootstrap. + +## Render Blueprint specifics + +The repo includes a `render.yaml` Blueprint: + +- Single instance only. The SQLite store is a single-writer design; two pods against the same disk corrupt it. `numInstances: 1` is intentional. +- 1 GiB persistent disk mounted at `/data`. +- `HOOKS_DATABASE_URL=/data/hooks.db` is set in the Blueprint. +- `HOOKS_PUBLIC_URL` and `RENDER_WEBHOOK_SECRET` are declared `sync: false` — set them in the service's **Environment** tab before the first deploy. +- No `HOOKS_LISTEN_ADDR`. The server honors `$PORT` (which Render injects) automatically. +- `/readyz` is the health check. + +Both `hooks` and `hooksctl` are on `$PATH` in the Render Shell, so token rotation, push-subscription management, and pruning all work without leaving the platform. + +## Single-process / SQLite limitation + +The default deployment is one process with SQLite. There is **no** built-in coordination for multi-process; SSE delivery and push dispatch assume a single in-process notifier. The storage interface is shaped to accept a Postgres backend later (and a Redis/NATS pub/sub for cross-process notifications), but that code isn't written yet — running two `hooks` processes against the same SQLite file is unsafe. + +## Skew-window semantics on initial backfill + +`hooksctl forward` replays from the cursor on connect, then tails live. **Initial backfill is bounded by the source's signature-verification skew window** (`skew_window` per source in `hooks.yaml`, or 5 minutes when unset). Events older than that window are skipped on the initial drain so a verifying consumer doesn't 401 on a stale `webhook-timestamp`. The cursor still advances past skipped events, so reconnects don't reconsider them. + +Skipped events remain in the store. Redeliver them via the inspector ("Replay to listeners") or `hooksctl replay`. + +This filter is **only on the initial backfill**. Live tail (notifier-triggered or keepalive-triggered drains) is unfiltered, so manual replays via the inspector still reach currently-connected subscribers. diff --git a/docs/quickstart.md b/docs/quickstart.md deleted file mode 100644 index cf6a049..0000000 --- a/docs/quickstart.md +++ /dev/null @@ -1,169 +0,0 @@ -# Quickstart - -Get from "fresh checkout" to "real Render webhook landing in a developer environment" in about ten minutes. This is the production-deployment shape (deploy once, log in from laptops). For a fully-local end-to-end demo, see the [README](../README.md). - -## 1. Install - -```sh -go install github.com/onebusaway/hooks/cmd/hooks@latest -go install github.com/onebusaway/hooks/cmd/hooksctl@latest -``` - -Or build from source: - -```sh -git clone https://github.com/onebusaway/hooks -cd hooks -make build # produces ./bin/hooks and ./bin/hooksctl -``` - -## 2. Scaffold a deployment - -On the server (or wherever you'll run `hooks`): - -```sh -hooks init --server-url https://webhooks.example.com -``` - -This writes `hooks.yaml`, creates `hooks.db`, mints a one-time **admin token** (the legacy system credential), and — because the users table is empty — prints a one-time **bootstrap signup URL** (24-hour TTL): - -```text -admin token (shown ONCE): -signup: https://webhooks.example.com/signup?code=ABCDEFGH... - (single-use; expires in 24h; auto-disables once any account exists) -``` - -Save both. The admin token has no recovery path; the signup URL is how the first human creates their admin account. If you skip `--server-url` (or `HOOKS_PUBLIC_URL`), the URL prints with a `localhost` placeholder you'll have to swap by hand. - -Edit `hooks.yaml` to point at your real provider secret(s): - -```yaml -sources: - render: - verifier: render - secret: ${RENDER_WEBHOOK_SECRET} - retention: 30d -``` - -Then export `RENDER_WEBHOOK_SECRET` (the per-webhook signing secret Render gave you when you created the webhook) in the environment that will run `hooks`. - -## 3. Stand up TLS termination - -The relay speaks plain HTTP. Stand any TLS-terminating reverse proxy in front of it: - -- **Caddy:** add a site like `webhooks.example.com { reverse_proxy localhost:8080 }`. Caddy obtains a Let's Encrypt cert automatically. -- **nginx:** standard `proxy_pass http://127.0.0.1:8080;` block, plus your existing TLS config. -- **Cloudflare Tunnel / Render itself / fly.io:** any platform that gives you HTTPS in front of an HTTP origin works. - -Start the relay: - -```sh -hooks # production -hooks --dev # verbose logs + opens the inspector locally -``` - -Wire your load balancer's health check to `/readyz` (which pings SQLite); `/healthz` is liveness-only. - -### 3a. Or run it as a container - -If you'd rather ship a container than a binary, the repo has a multi-stage `Dockerfile` (Go builder → small Alpine runtime, non-root). The image runs as UID 65532, mounts `/data` as a volume for the SQLite database, and ships both `hooks` and `hooksctl` so you can `docker exec` to manage tokens. - -```sh -make docker-build # builds hooks:dev -mkdir -p ./hooks-data -docker run --rm -v $(pwd)/hooks-data:/data hooks:dev init \ - --server-url https://webhooks.example.com -docker run -d --name hooks --restart=unless-stopped \ - -p 8080:8080 \ - -v $(pwd)/hooks-data:/data \ - -e RENDER_WEBHOOK_SECRET \ - -e HOOKS_PUBLIC_URL=https://webhooks.example.com \ - hooks:dev -``` - -Defaults set by the image: `HOOKS_DATABASE_URL=/data/hooks.db`, `HOOKS_LISTEN_ADDR=:8080`. Point your TLS-terminating proxy at `localhost:8080` exactly as in the binary path above. - -A Dockerfile-level `HEALTHCHECK` polls `/healthz`; in front of a load balancer, prefer `/readyz` (which also pings SQLite). - -### 3b. Or deploy to Render with the Blueprint - -The repo also includes a `render.yaml` Blueprint. To deploy: - -1. In Render: **New → Blueprint** and select this repo (fork first if you want autoDeploy on your own pushes). Render reads `render.yaml` and provisions a Docker web service plus a 1 GiB persistent disk mounted at `/data`. Before the first deploy, set the two `sync: false` env vars in the service's **Environment** tab: - - `RENDER_WEBHOOK_SECRET` — the per-webhook signing secret Render gives you when you create the webhook in step 5 below. (Use a placeholder for now and rotate it once the webhook exists.) - - `HOOKS_PUBLIC_URL` — your service's external URL, e.g. `https://hooks-abc1.onrender.com`. Used to build the bootstrap signup link printed during first-boot init. -2. Trigger a deploy. The container's entrypoint detects an empty `/data`, runs `hooks init --dir /data` automatically, and prints both a **bootstrap signup URL** and a one-time **admin token** (legacy fallback credential) to the service **Logs**. Copy both — they're secrets, and the token is shown only once. The server then starts normally; you don't need to start it yourself. -3. The same log block prints a Render-aware "Next steps" checklist that walks through the rest of this guide with `HOOKS_PUBLIC_URL` already filled in. The signup URL from step 2 is the path you actually want to use — open it in a browser and continue at [§4](#4-claim-the-first-admin-account). The admin token is only needed if you want to authenticate `hooksctl` before claiming the human account, or if the signup URL expires before you use it. - -The server honors `$PORT` (which Render injects) automatically, so the Blueprint only wires `/readyz` as the health check — no listen-address knob to keep in sync. Both `hooks` and `hooksctl` are on `$PATH` in the shell, so token rotation, push subscription management, and pruning all work without leaving Render. - -## 4. Claim the first admin account - -Open the bootstrap signup URL from step 2 in a browser. Pick an email, name, and password (≥ 12 characters; must not contain your email or its local-part). Submitting the form consumes the bootstrap invite, signs you into the inspector at `/`, and the URL returns 409 from then on. - -If the link expires before you use it, open the service's **Shell** (now available since the deploy is healthy) and re-run `hooks init --force --server-url "$HOOKS_PUBLIC_URL"` to mint a fresh 24-hour invite. Once any user exists, the bootstrap path is closed — invite teammates from `/users` (or `POST /api/invites`) instead. - -## 5. Register the webhook with Render - -In Render, create (or edit) the webhook so its URL points at: - -```text -https://webhooks.example.com/ingest/render -``` - -with the same secret you set in `RENDER_WEBHOOK_SECRET`. - -## 6. Connect a laptop with `hooksctl login` - -On your dev laptop: - -```sh -hooksctl login --server https://webhooks.example.com --scopes render -``` - -The CLI prints a `Visit:` URL and a `Code:` to type into the relay's `/device` page. Open the URL in a browser where you're logged in (or sign up via an invite from another admin first), enter the code, and re-enter your password to approve the pairing. The CLI then writes a personal access token to `~/.config/hooks/credentials.default` (mode `0600`). Default scope on approval is `account` only, so pass `--scopes` (comma-separated source names) to also subscribe, or `--admin` for admin scope. - -Verify: - -```sh -hooksctl whoami -``` - -## 7. Forward to a local app (SSE pull) - -```sh -hooksctl forward render --to http://localhost:3000/webhooks/render -``` - -Against a logged-in profile, `forward` auto-mints an ephemeral `kind='listener'` token, replays anything missed since the last cursor, then tails live. Initial catch-up is bounded by the source's signature-verification skew window (5 minutes for Render by default): events older than that are skipped on the initial drain so your verifying consumer doesn't 401 on a stale `webhook-timestamp`. The cursor still advances past skipped events, so reconnects don't reconsider them; the events remain in the store and can be redelivered via the inspector or `hooksctl replay`. The token is revoked on clean exit; the server's prune loop reaps any ephemeral token whose `last_used_at` falls 24h behind. Bytes hitting your local app are byte-for-byte identical to what Render sent. Original headers (other than hop-by-hop) are preserved. - -For a long-lived listener (skip the mint/revoke dance every run), see [`docs/accounts.md`](accounts.md#power-user-long-lived-listener-token). - -## 8. Or, register a long-lived consumer (HTTP push) - -For a production service that's always up: - -```sh -hooksctl me sub add --source render --to https://my-svc.example.com/hooks --name production -``` - -`me sub add` prints a per-subscription **signing secret** exactly once. Store it on the consumer. The relay will POST every event to that URL with `X-Hooks-Signature: t=,v1=.")>`. See [`docs/consumer-verification.md`](consumer-verification.md) for verification snippets. - -The plaintext signing secret only lives in memory, so push delivery for each subscription is paused after a server restart until you re-arm it with `hooksctl me sub rotate-secret ` (or `hooksctl push rotate-secret ` for admin-owned subscriptions). - -## 9. Browse - -Open `https://webhooks.example.com/` and sign in with the email/password from step 4. You can browse every captured event, replay any of them to live listeners, manage tokens and push subscriptions, invite teammates, and review the audit log at `/audit`. - -## What `hooks init` does NOT do - -- Set up your reverse proxy. That's step 3. -- Register the Render-side webhook. That's step 5. -- Persist the admin token or bootstrap signup URL anywhere except standard out. Save them. - -## Where to next - -- [`docs/accounts.md`](accounts.md) — invites, scopes, multiple profiles, ephemeral vs long-lived listener tokens, deactivation semantics. -- [`docs/security.md`](security.md) — token kinds, hashing posture, signature verification, secret-handling policy. -- [`docs/sources.md`](sources.md) — how to add a new webhook provider. -- [`docs/operations.md`](operations.md) — pruning, retention, body-integrity verification. diff --git a/docs/operations.md b/docs/running-in-production.md similarity index 67% rename from docs/operations.md rename to docs/running-in-production.md index e371de5..bb3873f 100644 --- a/docs/operations.md +++ b/docs/running-in-production.md @@ -1,4 +1,6 @@ -# Operations +# Running hooks in production + +Day-2 ops: backups, retention, observability, push-subscription health, restarts, and graceful shutdown. For one-time deployment setup (env vars, `hooks init`, container internals, Render Blueprint), see [`deployment.md`](deployment.md). ## Backup @@ -12,7 +14,7 @@ This is the SQLite-blessed online-backup form: it cooperates with WAL and is saf ## Retention and pruning -- Default retention is **30 days per source**, configurable via `retention:` per source. +- Default retention is **30 days per source**, configurable via `retention:` per source in `hooks.yaml`. - Special values: `0` and `forever` disable auto-prune for that source. - The pruner wakes once an hour and logs the row count it deleted per source per pass. Log lines look like: @@ -26,6 +28,8 @@ This is the SQLite-blessed online-backup form: it cooperates with WAL and is saf hooks prune --older-than 7d ``` +The same loop also reaps `ephemeral=true` listener tokens whose `last_used_at` is more than 24h in the past (forward crash-safety net) and `device_pairings` rows 24h after terminal state. The audit log is never pruned — growth is bounded by operator actions, not webhook traffic. + ## Observability In v1 the only observability primitive is structured JSON logs to stderr. Notable events: @@ -37,13 +41,11 @@ In v1 the only observability primitive is structured JSON logs to stderr. Notabl `/healthz` returns 200 once the listener is open. `/readyz` returns 200 only when the SQLite store can complete a round-trip ping; use this for load-balancer health checks. -## Multi-process limitations - -The default deployment is one process with SQLite. There is **no** built-in coordination for multi-process; SSE delivery and push dispatch assume a single in-process Notifier. The storage interface is shaped to accept a Postgres backend later (and a Redis/NATS pub/sub for cross-process notifications), but that code is not written yet — running two `hooks` processes against the same SQLite file is unsafe. - ## Push-subscription health -Use `/push` (or `hooksctl push list`) to monitor: +`hooksctl` exposes two parallel command trees for push subscriptions: `hooksctl me sub *` operates only on subscriptions the calling user owns (self-service), and `hooksctl push *` is the admin/operator form that operates on every subscription on the relay (admin scope required). The commands below use `push *` because day-2 ops typically means triaging across the whole relay. + +Open the inspector's `/push` page (or run `hooksctl push list`) to monitor: - **Queue depth**: `latest_sequence_for_source - cursor`. Grows during outages; should return to 0 within seconds after recovery. - **`consecutive_failures`**: resets to 0 on the next 2xx. @@ -51,6 +53,14 @@ Use `/push` (or `hooksctl push list`) to monitor: A subscription that stays in a failing state with `consecutive_failures > 100` produces a single WARN log line on the streak's first crossing of 100. There is no built-in alerting; operators are expected to wire whatever they have (logs to Loki/Datadog, etc.). +To smoke-test a consumer end-to-end without waiting for a real provider event, send a synthetic delivery: + +```sh +hooksctl push test +``` + +The relay POSTs a small probe payload to the subscription's URL, signed with the live secret, and reports the consumer's status code. A healthy consumer should: return 2xx within a few seconds, log the `X-Hooks-Delivery-Id`, and validate `X-Hooks-Signature.t` against its current clock. `consecutive_failures` should sit at 0 in steady state — any non-zero baseline means the consumer is dropping deliveries and silently retrying isn't fixing it. + If a target is permanently broken, the safe pause-or-delete commands are: ```sh diff --git a/render.yaml b/render.yaml index 83070b5..f0ee361 100644 --- a/render.yaml +++ b/render.yaml @@ -1,5 +1,6 @@ -# Render Blueprint for the hooks relay. After deploy, follow -# docs/quickstart.md (section 3b) to bootstrap the admin account. +# Render Blueprint for the hooks relay. After deploy, follow the README's +# "Option A — Render Blueprint (one-click)" section to bootstrap the admin +# account. services: - type: web name: hooks