Skip to content

refactor(workers): make the queue scale to zero when idle #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jaredLunde merged 1 commit into from
Jun 27, 2026
Merged

refactor(workers): make the queue scale to zero when idle #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions .sqlx/query-26e3cc708d766ff096f19c7af4537d82b9d5664f6f43478ec6ab10b591a591b0.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

20 changes: 20 additions & 0 deletions .sqlx/query-2e9b41722e7cc1dcdf7068a7b3900412e40fc9aee82e1913a4d881c0ccdb629a.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

20 changes: 20 additions & 0 deletions .sqlx/query-a4981444643a40ea7b2c8359335a041ebbd1650888b17af984dddb5ccc0fa95c.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

48 changes: 24 additions & 24 deletions ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ The signature follows the SNS v2 spec: alphabetically sorted field name/value pa

Both steps happen inside the same HTTP handler (`src/ops/event.rs`). The SQS fan-out is synchronous; HTTP deliveries are asynchronous (picked up by the delivery worker).

The delivery worker (`src/ops/delivery.rs`) polls `http_deliveries` in a background task. Each poll opens a transaction, fetches pending rows with `FOR UPDATE SKIP LOCKED`, POSTs to each endpoint, and deletes on success or updates `attempt` / `next_attempt_at` on failure. Backoff schedule: 10s → 30s → 60s → 300s. Exhausted rows (`attempt >= max_attempts`) stay for inspection; no automatic reaping.
The delivery worker (`src/ops/delivery.rs`) is event-driven. Each cycle it opens a transaction, fetches pending rows with `FOR UPDATE SKIP LOCKED`, POSTs to each endpoint, and deletes on success or updates `attempt` / `next_attempt_at` on failure. When a batch comes back empty it sleeps until the earliest pending row's `next_attempt_at` (or parks if the table is empty), woken early in-process when a publish inserts new deliveries — so an idle queue issues no queries and can scale to zero. Backoff schedule: 10s → 30s → 60s → 300s. Rows that exhaust all attempts (`attempt >= max_attempts`) are swept at the end of a delivery batch — deleted, logged, and counted via `delivery_exhausted_total` — so dead letters surface in logs/metrics instead of silently accumulating.

**REST API subscriptions** default to `raw_delivery=true` (raw payload). Opt in to the SNS envelope with `"envelope": true` in the subscribe body. **SNS wire protocol subscriptions** default to `raw_delivery=false` (envelope); set `RawMessageDelivery=true` via `SetSubscriptionAttributes` to switch.

Expand All @@ -205,7 +205,7 @@ Bindings are stored in `queue.topic_subscriptions` with columns `protocol`, `end

### Schedules (time-based triggers)

`queue.schedule` is a single table indexed by `(next_fire_at) WHERE status = 'active'`. The scheduler worker (`src/ops/schedule_worker.rs`) polls it every `QUEUE_SCHEDULE_POLL_MS` (default 1000ms):
`queue.schedule` is a single table indexed by `(next_fire_at) WHERE status = 'active'`. The scheduler worker (`src/ops/schedule_worker.rs`) is event-driven: each cycle it fires everything due, then sleeps until the earliest active schedule's `next_fire_at` (capped at `KEEPALIVE_CAP`, under the instance light-sleep window so the VM stays awake while schedules are active and the fire lands on time), woken early in-process by any `/schedules` mutation. With no active schedules it parks and issues no queries, so the queue and Postgres scale to zero. Each firing cycle:

1. Open transaction, claim up to `QUEUE_SCHEDULE_BATCH_SIZE` due rows with `FOR UPDATE SKIP LOCKED ORDER BY next_fire_at`.
2. For each row, open a `SAVEPOINT`, then dispatch by `target_kind`:
Expand All @@ -220,7 +220,7 @@ Every fired message carries `headers._schedule = { name, scheduled_for, out_of_b

Schedule expressions accept four shapes (`cron`, `every`, `when`, `fire_at`); the parser layer (`src/schedule/`) canonicalizes each to either a cron string + timezone or a single UTC instant before storage. Dry-runs via `POST /v1/previews` round-trip the canonical form, a `human_readable` description, and the next N fire times — the agent-affordance that catches the "I wrote `0 9 *` and it doesn't do what I think" class of mistake before commit.

The worker is always on and trivially cheap when idle: an empty partial index probe per poll cycle, sub-millisecond.
The worker runs for the process lifetime but does no work when idle: with no active schedules it parks on an in-process notification and issues zero queries, letting the VM sleep. `QUEUE_SCHEDULE_POLL_MS` is now only the retry backoff after a transient error, not a steady-state poll interval.

### Write coalescer

Expand Down Expand Up @@ -289,27 +289,27 @@ Within the selected group, messages are delivered in `msg_id ASC` order (FIFO).

## Configuration

| Variable | Default | What It Controls |
| ---------------------------------- | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------- |
| `DATABASE_URL` | (required) | PostgreSQL connection string passed to sqlx `PgPoolOptions`. |
| `QUEUE_ADDRESS` | `0.0.0.0:9324` | TCP bind address for the HTTP server. |
| `QUEUE_DEFAULT_VISIBILITY_TIMEOUT` | `30` | Seconds applied when a `ReceiveMessage` request omits `VisibilityTimeout`. |
| `QUEUE_MAX_CONNECTIONS` | `10` | Hard cap on the sqlx connection pool. Excess operations wait for a free slot. |
| `LOG_LEVEL` | `info` | `EnvFilter` directive (e.g. `beyond_queue=debug,info`). JSON-structured output. |
| `OTLP_ENABLED` | `false` | Enable OpenTelemetry OTLP trace export over gRPC. |
| `OTLP_ENDPOINT` | `http://localhost:4317` | gRPC OTLP collector. Used when `OTLP_ENABLED=true`. |
| `OTLP_SAMPLE_RATE` | `0.1` | Fraction of traces sampled (0.0 = never, 1.0 = always). Only effective when `OTLP_ENABLED=true`. |
| `QUEUE_LINGER_MS` | `0` | Write coalescer window (ms). Non-FIFO sends are held up to this duration and flushed as a single batch. `0` disables coalescing. |
| `QUEUE_BASE_URL` | `http://{QUEUE_ADDRESS}` | Base URL for SQS queue URLs returned to clients (`{QUEUE_BASE_URL}/000000000000/{name}`). Override when behind a proxy. |
| `QUEUE_HTTP_DELIVERY_ENABLED` | `true` | Enable the background HTTP/HTTPS delivery worker. |
| `QUEUE_HTTP_DELIVERY_POLL_MS` | `1000` | Delivery worker poll interval (ms). Lower values increase responsiveness at the cost of idle DB load. |
| `QUEUE_HTTP_DELIVERY_TIMEOUT_SECS` | `5` | Per-request timeout for outbound webhook POSTs. |
| `QUEUE_HTTP_DELIVERY_BATCH_SIZE` | `50` | Maximum rows the delivery worker claims per poll cycle. Tune up under high SNS fanout load. |
| `QUEUE_SCHEDULE_ENABLED` | `true` | Enable the background schedule worker (cron / every / when triggers). |
| `QUEUE_SCHEDULE_POLL_MS` | `1000` | Schedule worker poll interval (ms). Floor on fire latency; idle cost is one sub-millisecond partial-index probe per cycle. |
| `QUEUE_SCHEDULE_BATCH_SIZE` | `32` | Maximum rows the schedule worker claims per poll cycle. |
| `QUEUE_SCHEDULE_PREVIEW_COUNT` | `5` | Number of upcoming fire timestamps projected in `next_fires` on schedule and preview responses. |
| `QUEUE_SCHEDULE_LIST_MAX` | `1000` | Hard cap on `GET /v1/schedules` response size. |
| Variable | Default | What It Controls |
| ---------------------------------- | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DATABASE_URL` | (required) | PostgreSQL connection string passed to sqlx `PgPoolOptions`. |
| `QUEUE_ADDRESS` | `0.0.0.0:9324` | TCP bind address for the HTTP server. |
| `QUEUE_DEFAULT_VISIBILITY_TIMEOUT` | `30` | Seconds applied when a `ReceiveMessage` request omits `VisibilityTimeout`. |
| `QUEUE_MAX_CONNECTIONS` | `10` | Hard cap on the sqlx connection pool. Excess operations wait for a free slot. |
| `LOG_LEVEL` | `info` | `EnvFilter` directive (e.g. `beyond_queue=debug,info`). JSON-structured output. |
| `OTLP_ENABLED` | `false` | Enable OpenTelemetry OTLP trace export over gRPC. |
| `OTLP_ENDPOINT` | `http://localhost:4317` | gRPC OTLP collector. Used when `OTLP_ENABLED=true`. |
| `OTLP_SAMPLE_RATE` | `0.1` | Fraction of traces sampled (0.0 = never, 1.0 = always). Only effective when `OTLP_ENABLED=true`. |
| `QUEUE_LINGER_MS` | `0` | Write coalescer window (ms). Non-FIFO sends are held up to this duration and flushed as a single batch. `0` disables coalescing. |
| `QUEUE_BASE_URL` | `http://{QUEUE_ADDRESS}` | Base URL for SQS queue URLs returned to clients (`{QUEUE_BASE_URL}/000000000000/{name}`). Override when behind a proxy. |
| `QUEUE_HTTP_DELIVERY_ENABLED` | `true` | Enable the background HTTP/HTTPS delivery worker. |
| `QUEUE_HTTP_DELIVERY_POLL_MS` | `1000` | Retry backoff (ms) after a transient delivery-batch error. Not a steady-state poll: the worker is event-driven (sleeps until the next `next_attempt_at`, woken in-process on publish). |
| `QUEUE_HTTP_DELIVERY_TIMEOUT_SECS` | `5` | Per-request timeout for outbound webhook POSTs. |
| `QUEUE_HTTP_DELIVERY_BATCH_SIZE` | `50` | Maximum rows the delivery worker claims per poll cycle. Tune up under high SNS fanout load. |
| `QUEUE_SCHEDULE_ENABLED` | `true` | Enable the background schedule worker (cron / every / when triggers). |
| `QUEUE_SCHEDULE_POLL_MS` | `1000` | Retry backoff (ms) after a transient schedule-poll error. Not a steady-state poll: the worker is event-driven (sleeps until the next `next_fire_at`, capped under the light-sleep window, woken in-process on `/schedules` writes). |
| `QUEUE_SCHEDULE_BATCH_SIZE` | `32` | Maximum rows the schedule worker claims per poll cycle. |
| `QUEUE_SCHEDULE_PREVIEW_COUNT` | `5` | Number of upcoming fire timestamps projected in `next_fires` on schedule and preview responses. |
| `QUEUE_SCHEDULE_LIST_MAX` | `1000` | Hard cap on `GET /v1/schedules` response size. |

---

Expand Down
50 changes: 31 additions & 19 deletions SCHEDULES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,10 @@ already uses. One new table, zero new pgrx functions, one new SDK verb.
built.
- **Forks with the rest of the platform.** Schedules live in user
Postgres, on the user's GlideFS volume.
- **Trivial idle cost.** The scheduler worker is always on but polls a
partial index `WHERE status = 'active'`; an empty schedule table costs
one sub-millisecond index probe per poll cycle.
- **Zero idle cost.** The scheduler worker is event-driven: with no active
schedules it parks and issues no queries at all, so the queue (and Postgres)
scale to zero. With schedules active it sleeps until the next `next_fire_at`
(a partial-index probe on `WHERE status = 'active'`), not on a fixed poll.
- The minimum effective surface. One table, one worker, three expression
forms, three target kinds.

Expand Down Expand Up @@ -57,33 +58,44 @@ into a workflow is the same fanout a topic does.

## The runtime model

The queue server is platform infrastructure. It runs alongside the user's
Postgres on their GlideFS volume and does not scale to zero — same as the
database. Users don't pay for it any more than they pay for Postgres being up.
The queue server scales to zero with the app. The platform sleeps an idle VM
(no network traffic for ~5 min → light sleep → deep sleep), and the queue
server is built so an idle app generates no traffic: the schedule worker only
talks to Postgres when there is work to do (see _Worker lifecycle_). When no
schedules are active and no deliveries are pending, the queue VM goes idle and
sleeps — and because the queue is the only thing querying it, Postgres sleeps
too.

While any schedule **is** active, the worker re-checks Postgres at least every
~`KEEPALIVE_CAP` (kept under the light-sleep window). Those checks are network
traffic to Postgres, so both VMs stay awake and the fire lands on time — the
keepalive is just the re-check the worker has to do anyway, not a separate
mechanism. So: idle app → both sleep; app with schedules → both awake until the
schedules are paused/deleted.

Function VMs — user code — sleep when idle. When a schedule fires, the
scheduler worker (running in the queue server) calls `queue.send` or
`queue.publish_event`. The fan-out sends HTTP to the function VM's endpoint.
The gateway sees a sleeping VM, wakes it, delivers the request. The function
processes it and goes back to sleep.

No special wakeup mechanism. No external scheduler. HTTP fan-out handles
sleeping function VMs as a natural consequence of the existing wake-on-traffic
machinery — the same path any other inbound request takes.
processes it and goes back to sleep — the same wake-on-traffic path any other
inbound request takes.

### Worker lifecycle

The scheduler worker starts at queue server boot and runs for the lifetime
of the process. Its idle cost is one partial-index probe per poll cycle —
sub-millisecond on an empty `schedule_due_idx` — so chasing "zero overhead"
by stopping the worker when no schedules exist would add intra-process
signaling complexity without measurable benefit. Same pattern as the HTTP
delivery worker.
The scheduler worker starts at queue server boot and runs for the lifetime of
the process, but it is **event-driven, not a fixed poll**. Each cycle it fires
everything currently due, then sleeps until the earliest active schedule's
`next_fire_at` (capped at `KEEPALIVE_CAP`), woken early in-process when a
`/schedules` mutation changes the table. When there are no active schedules it
parks entirely — zero queries, zero traffic — so the VM can sleep. The HTTP
delivery worker follows the same shape, keyed on `event_deliveries`.

On boot, the worker runs one catchup pass (firing any missed occurrences
for active schedules subject to each schedule's `catchup` and
`catchup_limit` settings) before entering the normal poll loop. This
covers fires missed while the queue server was down.
`catchup_limit` settings) before entering the normal loop. This covers fires
missed while the queue server was down or asleep. (Light sleep is a
pause/resume, not a restart, so it does not re-run catchup; while schedules are
active the VM stays awake anyway, so no fire is missed.)

---

Expand Down
Loading