diff --git a/AGENTS.md b/AGENTS.md index f3fa047..bea4b46 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,15 +5,17 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors # AGENTS.md - `benchmarks-website/` -Read [`README.md`](README.md) first for the architecture and the v3/v4 -side-by-side situation. Then this file. The root [`CLAUDE.md`](../CLAUDE.md) -covers Rust style, test layout, commit conventions. +Read [`README.md`](README.md) first for the architecture. Then this file. The +root [`CLAUDE.md`](../CLAUDE.md) covers Rust style, test layout, commit +conventions. Everything about the legacy generations (v2, v3) is in +[`docs/legacy.md`](docs/legacy.md). ## The legacy v2 site -The v2 Vite/React site is still live at `bench.vortex.dev` but is deployed from -elsewhere — its source no longer lives in this repo. The migrator still reads -the v2 S3 dump as its historical source (see [`migrate/README.md`](migrate/README.md)). +The v2 Vite/React site is retired from serving (`bench.vortex.dev` now points at +v4) and was deployed from elsewhere — its source never lived in this repo. The +migrator still reads the v2 S3 dump as its historical source (see +[`migrate/README.md`](migrate/README.md)). The v3 deploy lives entirely under `server/`, `migrate/`, and `ops/`. The operator runbook is [`ops/README.md`](ops/README.md). diff --git a/README.md b/README.md index 19cd906..43cc93a 100644 --- a/README.md +++ b/README.md @@ -5,103 +5,110 @@ SPDX-FileCopyrightText: Copyright the Vortex contributors # benchmarks-website -The public home for **Vortex benchmark results** — and the data pipeline behind -it. Benchmark jobs in the [`vortex-data/vortex`](https://github.com/vortex-data/vortex) -monorepo emit one measurement per commit; this repo stores those measurements -and renders them as time-series charts — one per `(benchmark, dataset, …)` -dimension tuple, plotted across the Vortex commit history. +The public home for **Vortex benchmark results** — and the data pipeline behind it. Benchmark jobs +in the [`vortex-data/vortex`](https://github.com/vortex-data/vortex) monorepo emit one measurement +per commit; this repo stores those measurements in Postgres and renders them as time-series +charts — one per `(benchmark, dataset, …)` dimension tuple, plotted across the Vortex commit +history. **Live at [bench.vortex.dev](https://bench.vortex.dev).** -> This repo owns the **storage, the read services, and the ingest contract** — -> not the benchmark runs. The emitters live in the monorepo and stay there. +> This repo owns the **storage, the read service, and the ingest contract** — not the benchmark +> runs. The emitters live in the monorepo and stay there. -## How it works, in one glance +## Architecture + +The system is a small pipeline with three stages: benchmark CI runs **emit** measurements, a +hosted Postgres database **stores** them, and a serverless web app **reads** them back out as +charts. There is no ingest server and no box to operate — CI writes straight to the database, and +the site reads straight from it. ``` -monorepo CI ──emit──▶ database ──read──▶ charts - (the runs) (this repo) one per (benchmark, dataset, …), - x-axis = the Vortex commit history + vortex-data/vortex monorepo (CI) +┌──────────────────────────────────┐ +│ benchmark jobs │ +│ vortex-bench ─▶ JSONL records │ +│ scripts/post-ingest.py │ +└───────┬──────────────────┬───────┘ + │ upsert rows │ POST /api/revalidate + │ (direct SQL) │ (flush the read cache) + ▼ ▼ +┌────────────────┐ SQL ┌─────────────────────────────┐ ┌────────────────────┐ +│ AWS RDS │◀──────│ web/ — Next.js read service │─────▶│ charts, one per │ +│ Postgres │ │ on Vercel (server-rendered, │ │ dimension tuple │ +│ (vortex_bench) │ │ Data Cache + CDN in front) │ │ @ bench.vortex.dev │ +└────────────────┘ └─────────────────────────────┘ └────────────────────┘ ``` -The site has been rebuilt twice, and all three generations run side-by-side in -production while the final cutover finishes — they share one emitter output, so -the same measurement feeds every generation. Only the v3 and v4 source lives in -this repo; v2 is deployed elsewhere: - -| Gen | Stack | Storage | Status | -|---|---|---|---| -| **v2** | Vite/React SPA on Cloudflare | static S3 dump | **live** at `bench.vortex.dev` (deployed elsewhere); retired after cutover | -| **v3** | Rust `axum` + `maud` on EC2 | DuckDB on local disk | experimental; an emit target only | -| **v4** | Next.js App Router on Vercel | AWS RDS Postgres | the **forward stack**; `develop` = production at `benchmarks-website.vercel.app` | - -Each generation traded the previous one's main weakness: v2→v3 moved grouping -from read-time-in-the-browser to ingest-time records in a real analytical store; -v3→v4 moved off a self-managed box onto managed serverless. The full story — data -flow, why there are three generations, and the design decisions and tradeoffs -behind the current stack — is in +**Emit.** Each benchmark CI run in the monorepo writes JSONL measurement records +(`vortex-bench --gh-json-v3`), and `scripts/post-ingest.py` delivers them. Every measurement is +one of five kinds — query time, compression time, compression size, random-access time, and +vector search — each carrying a commit SHA plus the dimension tuple that identifies its chart. +The versioned wire format is this repo's [`CONTRACT.md`](CONTRACT.md). + +**Store.** One AWS RDS Postgres database holds one fact table per measurement kind plus a +`commits` dimension table. Each fact row is keyed by `measurement_id`, a deterministic hash of +the commit and dimension tuple, so re-running a benchmark for the same commit updates the row +instead of duplicating it. The schema lives in [`migrations/`](migrations/) and deploys through a +GitHub-OIDC workflow, and a single `SCHEMA_VERSION` integer keeps emitters and readers in +lockstep across the two repos. + +**Read.** [`web/`](web/README.md) is a Next.js App Router app on Vercel. Every page is rendered +on the server from live Postgres queries — there is no build-time data step — and two cache +layers make that cheap: the Next.js Data Cache keeps the default views warm (flushed by the +ingest hook `POST /api/revalidate` after each run lands), and the Vercel CDN serves repeat +traffic with a five-minute freshness window. Charts and groups are addressed by opaque slugs that +encode their dimension tuple. + +The full story — the data pipeline in detail, the read path and its caches, how a low-traffic +serverless site stays fast, deploy and infra, and the design decisions behind the stack — is in [**`docs/architecture/`**](docs/architecture/README.md). **Start there.** +This site is the third generation of the benchmarks site; the previous two (v2 and v3) are +retired or being decommissioned, and everything about them lives in +[`docs/legacy.md`](docs/legacy.md). + ## Find your way around | If you want to… | Go to | |---|---| | understand the whole system | [`docs/architecture/`](docs/architecture/README.md) | -| know the emitter ↔ ingester wire format | [`CONTRACT.md`](CONTRACT.md) | +| know the emitter ↔ ingest wire format | [`CONTRACT.md`](CONTRACT.md) | | work in this tree (env vars, conventions, footguns) | [`AGENTS.md`](AGENTS.md) | | deploy, set up secrets, or run a data refresh | [`docs/runbooks/`](docs/runbooks/) | +| read about the previous generations (v2, v3) | [`docs/legacy.md`](docs/legacy.md) | | dig into one component | the per-directory READMEs in the layout below | ## Layout | Path | What it is | |------|------------| -| `web/` | **v4** — the Next.js read service on Vercel ([README](web/README.md)). | -| `server/` | **v3** — the Rust `axum` ingest/read server ([ARCHITECTURE](server/ARCHITECTURE.md)). | -| `migrate/` | `vortex-bench-migrate`, the v2→v3→v4 migration tool ([README](migrate/README.md)). | -| `migrations/` | the Postgres schema — SQL migrations + the `_applied_migrations` ledger. | +| `web/` | The Next.js read service on Vercel ([README](web/README.md)). | +| `migrations/` | The Postgres schema — SQL migrations + the `_applied_migrations` ledger. | | `infra/` | AWS provisioning for the hosted Postgres + IAM ([README](infra/README.md)). | -| `ops/` | the legacy v3 host deploy runbook + scripts ([README](ops/README.md)). | -| `scripts/` | the schema-deploy runner and golden fixtures. | +| `scripts/` | The schema-deploy runner and golden fixtures. | +| `docs/` | Architecture docs, operational runbooks, and the legacy-generations doc. | +| `server/`, `migrate/`, `ops/` | **Legacy** v3 pieces — see [`docs/legacy.md`](docs/legacy.md). | ## Quick start -The forward stack is the v4 web app: - ```bash -# v4 web app (Next.js). Needs BENCH_DB_* env for a real database — see web/README.md. +# The web app needs BENCH_DB_* env vars for a real database — see web/README.md. cd web && pnpm install && pnpm dev ``` -Working on the legacy Rust stack (the v3 server and the migrator): - -```bash -# v3 server (DuckDB) + workspace tests: -INGEST_BEARER_TOKEN=dev cargo run -p vortex-bench-server -cargo nextest run -p vortex-bench-server -p vortex-bench-migrate - -# Build a fresh DuckDB from the v2 S3 dump: -cargo run -p vortex-bench-migrate -- run --output ./bench.duckdb -``` - -See [`AGENTS.md`](AGENTS.md) for the full local-dev and env-var contract. +`pnpm build` deliberately works without a database, so CI can build with no secrets. See +[`AGENTS.md`](AGENTS.md) for the full local-dev and env-var contract, and +[`docs/legacy.md`](docs/legacy.md) if you need to run the legacy Rust stack. ## Status -The split from the monorepo is **complete** — standalone build, CI, Vercel -deploy, OIDC schema deploy, and secrets all live here, and v4 serves the full -benchmark history. What remains is deliberately deferred (make v4 good before -tearing anything down): - -- **Emitter / ingest cutover** — point the monorepo emitters at the v4 ingest - path (direct RDS write + cache revalidate) instead of the v2/v3 paths. - Cross-repo plan: [`docs/runbooks/emitter-ingest-cutover.md`](docs/runbooks/emitter-ingest-cutover.md). -- **DNS cutover** — repoint `bench.vortex.dev` at v4 and make the Vercel - deployment public. -- **Decommission v2 and v3** once nothing depends on them. +The site is **live production**: `bench.vortex.dev` is served by the Vercel deployment, and the +monorepo emitters write each run directly to Postgres. What remains is legacy teardown: -Until the emitter cutover lands, v4 data is refreshed by re-running -`vortex-bench-migrate` (see [`migrate/README.md`](migrate/README.md)). +- Promote the monorepo's v4 ingest steps from best-effort (`continue-on-error`) to required. +- Decommission the v2 and v3 deployments and delete their code from both repos. The inventory is + in [`docs/legacy.md`](docs/legacy.md). ## License diff --git a/docs/architecture/README.md b/docs/architecture/README.md index eac45f5..cfab0bd 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -35,8 +35,9 @@ Benchmark jobs in the [`vortex-data/vortex`](https://github.com/vortex-data/vort monorepo emit per-commit measurements. Those measurements land in a database, and a web app renders them as time-series charts — one chart per `(benchmark, dataset, …)` dimension tuple, with the x-axis being the Vortex commit history. The site has -been rebuilt twice; **all three generations live side-by-side in this repo** while -the cutover finishes. +been rebuilt twice; **v4 is live production** at `bench.vortex.dev`, and the two +previous generations survive only until their teardown finishes (see +[`../legacy.md`](../legacy.md)). ## The three generations @@ -56,15 +57,15 @@ the cutover finishes. on Cloudflare on an EC2 host (server components) │ │ │ ▼ ▼ ▼ - bench.vortex.dev (experimental) benchmarks-website.vercel.app - ★ LIVE today emit target only ★ the forward stack + (retired from (experimental) bench.vortex.dev + serving) emit target only ★ LIVE production ``` -| Gen | Stack | Storage | Status (2026-06-18) | +| Gen | Stack | Storage | Status (2026-07-08) | |---|---|---|---| -| **v2** | Node `server.js` + Vite/React SPA, on Cloudflare | A static `data.json.gz` dump in public S3, aggregated in memory at read time | **Live production** at `bench.vortex.dev`. To be retired after the v4 cutover. | +| **v2** | Node `server.js` + Vite/React SPA, on Cloudflare | A static `data.json.gz` dump in public S3, aggregated in memory at read time | Retired from serving; its S3 dump still receives CI uploads. To be torn down. | | **v3** | `vortex-bench-server` (Rust, `axum` + `maud`), on an EC2 host | DuckDB file on local disk | Experimental. Still receives emit data; not user-facing. To be decommissioned. | -| **v4** | `web/` (Next.js App Router) on Vercel | AWS RDS Postgres | **Live** at `benchmarks-website.vercel.app`; `develop` = production. The target the others cut over to. | +| **v4** | `web/` (Next.js App Router) on Vercel | AWS RDS Postgres | **Live production** at `bench.vortex.dev`; `develop` = production. | Why three? Each generation traded the previous one's main weakness: @@ -84,18 +85,20 @@ that carried the full benchmark history v2 → v3 → v4 without losing a row The decoupling from the monorepo is **complete** (PR #1 merged to `develop`): standalone Cargo workspace, standalone CI, Vercel deploy, schema deploy via OIDC, -and secrets are all in this repo. v4 is live and serving the full history. - -Remaining cutover steps (deliberately deferred — make v4 good first): - -1. **Emitter/ingest cutover** — point the monorepo emitters at the v4 ingest path - (direct RDS write + `POST /api/revalidate`) instead of the v2 S3 dump / v3 server. - Full plan, spanning both repos, in - [`../runbooks/emitter-ingest-cutover.md`](../runbooks/emitter-ingest-cutover.md). -2. **DNS cutover** — repoint `bench.vortex.dev` at v4 and make the Vercel - deployment protection public. -3. **Decommission v2 and v3** once nothing depends on them. - -Until then, v4 data is refreshed by re-running `vortex-bench-migrate` against the -v2 dump (see the [data-pipeline](data-pipeline.md) doc and +and secrets are all in this repo. The cutover itself is also complete: monorepo CI +writes each run directly to RDS (the v4 ingest path — direct Postgres write plus +`POST /api/revalidate`; see +[`../runbooks/emitter-ingest-cutover.md`](../runbooks/emitter-ingest-cutover.md)), +and `bench.vortex.dev` is served by the v4 Vercel deployment. + +Remaining teardown steps: + +1. **Promote the v4 ingest steps to required** — they are still + `continue-on-error: true` in monorepo CI while v3 remains the hard-required + emit target. +2. **Decommission v2 and v3** once nothing depends on them. The inventory is in + [`../legacy.md`](../legacy.md). + +Independent of live ingest, `vortex-bench-migrate` remains the backfill and +atomic full-refresh path (see the [data-pipeline](data-pipeline.md) doc and [`../../migrate/README.md`](../../migrate/README.md)). diff --git a/docs/architecture/data-pipeline.md b/docs/architecture/data-pipeline.md index 14de7e6..856381b 100644 --- a/docs/architecture/data-pipeline.md +++ b/docs/architecture/data-pipeline.md @@ -114,10 +114,10 @@ mirroring the server-internal hash — still never a wire field. Every v4 step i `continue-on-error` and gated on a configured role ARN, so it is additive and never blocks Path A. -> **Status:** Path B is not yet wired up for the live v4 deployment — the emitter -> still targets v2/v3. Until the emitter cutover, v4's data is refreshed by the -> migrator (below). This is why `BENCH_REVALIDATE_TOKEN` is intentionally not yet -> set on the v4 Vercel project. +> **Status:** Path B is live — it feeds the production v4 deployment at +> `bench.vortex.dev`. Its CI steps remain `continue-on-error` while Path A is +> still the hard-required target; they should be promoted to required when v3 is +> retired (see [`../legacy.md`](../legacy.md)). ## Storage by generation diff --git a/docs/architecture/read-path.md b/docs/architecture/read-path.md index c755871..cd201e0 100644 --- a/docs/architecture/read-path.md +++ b/docs/architecture/read-path.md @@ -76,10 +76,9 @@ fires. The endpoint **fails closed** — a missing `BENCH_REVALIDATE_TOKEN` retu `503`, never silently accepting an unauthenticated flush; the token compare is constant-time. -> Because the live deployment is currently fed by the *migrator* (not the ingest -> hook), a data refresh does not call `/api/revalidate`. In practice the cold/ -> expired Data Cache entries simply refill from the fresh RDS on the next read. -> When the emitter cutover lands, the hook becomes the freshness driver. +> One caveat: a *migrator* backfill or full refresh (as opposed to a CI ingest) +> does not call `/api/revalidate`, so after one the cold/expired Data Cache +> entries simply refill from the fresh RDS on the next read. ### Read API and windows diff --git a/docs/legacy.md b/docs/legacy.md new file mode 100644 index 0000000..1189f9d --- /dev/null +++ b/docs/legacy.md @@ -0,0 +1,82 @@ + + +# Previous generations (v2 and v3) + +The benchmarks site has been rebuilt twice; the current site is the third generation (**v4**). +This document is the home for everything about the previous two generations while they are +decommissioned: what they were, what state they are in, how to run their code, and what remains +to tear down. The full history and the design rationale behind each rebuild live in +[`architecture/`](architecture/README.md). + +## The generations + +| Gen | Stack | Storage | Status | +|---|---|---|---| +| **v2** | Node `server.js` + Vite/React SPA on Cloudflare | Static `data.json.gz` dump in public S3 | Retired from serving; its S3 dump still receives CI uploads | +| **v3** | Rust `axum` + `maud` server on an EC2 host | DuckDB file on local disk | Never user-facing; still a hard-required emit target | +| **v4** | `web/` (Next.js App Router) on Vercel | AWS RDS Postgres | **Live production** at `bench.vortex.dev` | + +The v2 source was deployed from elsewhere and never lived in this repo, though its S3 dump is +still the migrator's historical source. + +All three generations were fed by the same emitter output, so the record shapes are identical +across them — that shared shape is what made the v2→v3→v4 migration a faithful copy rather than a +re-derivation. + +Why three? Each generation traded the previous one's main weakness: + +- **v2 → v3** moved from *read-time* classification of loose name strings (all the grouping logic + ran in the browser/Node server on every load) to *ingest-time* structured records in a real + analytical store (DuckDB), with a precomputed read model so the landing page costs zero SQL. +- **v3 → v4** moved from a single self-managed EC2 host (DuckDB on local disk, a systemd polling + deploy) to a managed serverless stack (Next.js on Vercel reading hosted RDS Postgres), so there + is no box to operate. + +## What lives where + +| Path | What it is | +|---|---| +| [`server/`](../server/ARCHITECTURE.md) | The v3 Rust `axum` ingest/read server. | +| [`migrate/`](../migrate/README.md) | `vortex-bench-migrate`, the v2→v3→v4 migration tool. | +| [`ops/`](../ops/README.md) | The v3 host deploy runbook and scripts. | +| [`runbooks/v3-host-repoint.md`](runbooks/v3-host-repoint.md) | Repointing the v3 EC2 host. | + +`migrate/` is not purely legacy: its `load` mode is still the v4 backfill and atomic full-refresh +path (`load --replace` against RDS), so it outlives the v2/v3 deployments. See +[`architecture/data-pipeline.md`](architecture/data-pipeline.md). + +## Running the legacy Rust stack + +```bash +# v3 server (DuckDB) + workspace tests: +INGEST_BEARER_TOKEN=dev cargo run -p vortex-bench-server +cargo nextest run -p vortex-bench-server -p vortex-bench-migrate + +# Build a fresh DuckDB from the v2 S3 dump: +cargo run -p vortex-bench-migrate -- run --output ./bench.duckdb +``` + +## Teardown inventory + +v4 serves the full history and is a superset of the v2 dump (after the 2026-07-07 merge backfill +restored the appian and fineweb `[s3]` history), so the teardown is unblocked. The accepted loss +is ~910 v3 chart points that never reached the v2 dump. + +In the monorepo: + +- Promote the v4 ingest steps (`post-ingest.py --postgres`, gated on the ingest role ARN) from + `continue-on-error: true` to required. +- Delete the three `V3_INGEST_URL`-gated v3 ingest steps (`bench.yml`, `sql-benchmarks.yml`, + `commit-metadata.yml`). +- Repoint the PR-benchmark compare off the v2 S3 bucket, then delete the v2 uploads and + `publish-benchmarks-website.yml`. +- Keep `vortex-bench/src/v3.rs` and `post-ingest.py --postgres` — they are v4's wire format, not + v3 leftovers. + +In this repo (after the v3 EC2 host and its S3 backups are gone): + +- Delete `server/`, `ops/`, the v3 runbooks, and the Rust CI workflow. +- Keep `migrate/` for as long as the full-refresh path is wanted. diff --git a/web/README.md b/web/README.md index afe0367..cde9b00 100644 --- a/web/README.md +++ b/web/README.md @@ -1,9 +1,9 @@ # Benchmarks web (v4 read service) -Next.js 15 (App Router) read service for `benchmarks.vortex.dev`, serving the benchmark charts -from the benchmarks Postgres database. This is the v4 frontend that replaces both the v2 -Vite/React SPA (`../../src/`) and the v3 Axum server (`../server/`) at the Phase-5 cutover; until -then it runs behind a dev-only Vercel domain. +Next.js 15 (App Router) read service serving the benchmark charts at +[bench.vortex.dev](https://bench.vortex.dev) from the benchmarks Postgres database. This is the +v4 frontend that replaced both the v2 Vite/React SPA and the v3 Axum server (`../server/`) — see +[`../docs/legacy.md`](../docs/legacy.md). ## Local development