Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .oxfmtrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
"ignorePatterns": [
"node_modules",
"apps",
"packages",
"dist",
"coverage",
"*.tsbuildinfo",
Expand Down
2 changes: 1 addition & 1 deletion .oxlintrc.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,5 @@
"typescript/await-thenable": "off",
"import/no-cycle": "error"
},
"ignorePatterns": ["node_modules", "apps", "dist"]
"ignorePatterns": ["node_modules", "apps", "packages", "dist"]
}
136 changes: 120 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,56 @@ vx watch lint # re-run on file changes
vx run build --dry # show the plan, don't execute
```

## Beyond a task runner — what shipped in the 2026-06 platform arc

vx now ships an **open platform**, not just a CLI. Every surface
below is built into the binary; no external services required.

```sh
vx mcp # Model Context Protocol server (stdio)
# — Claude Code / Cursor / Continue.dev talk to vx as a typed tool

vx coordinator build test --workers 4 # start a distributed-CI coordinator
vx run --worker ws://coord:5180 # pull tasks from a coordinator and execute them

vx insights serve # localhost Solid+DuckDB-WASM SPA over cache.db
# — historical run flamegraphs, no backend

vx serve # WebSocket + SSE + NDJSON event stream
# GET /version /events /stream — `curl` works
```

### Open platform highlights

- **Wire protocol = JSON-RPC 2.0 + OTel LogRecord payload.** Any
JSON-RPC client works against `vx serve`. Three transports — WS,
SSE, NDJSON — off the same bus.
- **MCP server with live tools** (`getCacheStats`, `getRunHistory`,
`explainCacheKey`, `whyDidThisRerun`) that read your real
`cache.db`. Drop into any agent's stdio MCP slot.
- **Plugin API.** Declare `plugins: [...]` in `vx.workspace.ts`;
each plugin subscribes to bus lifecycle events with crash
isolation per hook.
- **Predictive scheduling.** Opt in with `predictive: true` in
`vx.workspace.ts` — the scheduler reads run history and picks
the task on the longest expected remaining critical path.
- **Distributed CI.** `vx coordinator` + `vx run --worker` over the
same protocol. Content-addressed: any worker producing
`<hash>` satisfies every consumer of `<hash>`.
- **OTel CI/CD spans.** Set `OTEL_EXPORTER_OTLP_ENDPOINT` and
install `@vzn/vx-otel-bridge` — every event flows to Grafana /
Honeycomb / Datadog / Tempo with zero config.
- **vx Cloud (Cloudflare-native).** `apps/cloud/` is a Wrangler
project: `bun wrangler deploy` from your fork gives you a private
hosted vx in your CF account in ~5 minutes — Workers + R2 + D1 +
Durable Objects + Queues + KV, with HMAC artifact signing.
- **`vx insights serve`** — Solid + UnoCSS + DuckDB-WASM SPA that
reads the workspace's `cache.db` directly. No backend, no daemon.

Each lives behind a one-paragraph design doc under
`docs/design/*-2026-06.md`. Phase-by-phase implementation log:
`docs/progress/implementation-log-2026-06.md`.

## A cache that actually understands your build

Every task runner caches. vx caches _correctly_ — and stops work
Expand Down Expand Up @@ -123,16 +173,23 @@ traces · `vx cache prune` with TTL and size caps.

## How it compares

| | vx | Turborepo | Nx |
| ------------------------- | ------------------------------------------ | ------------------------------ | ---------------- |
| Fully cached, 100 pkgs¹ | **144 ms** | 279 ms | 583+ ms |
| Config | TypeScript, evaluated into the cache key | JSON (static) | JSON (static) |
| Output ownership | **Strict** — wiped before exec AND restore | Additive (stale files survive) | Additive |
| Clean-tree hashing | **Zero reads** (git index OIDs) | git OIDs | re-hash / daemon |
| Daemon required for speed | **No** | Optional | Yes |
| Artifact signing | **Hard-fail** on unsigned | Soft | No |
| Per-task sandbox | **Yes** — kernel-level, opt-in | No | No |
| Install | **Single binary** — 1 curl line | npm + Node | npm + Node |
| | vx | Turborepo | Nx |
| ------------------------- | ------------------------------------------------------------------- | ------------------------------ | ------------------- |
| Fully cached, 100 pkgs¹ | **144 ms** | 279 ms | 583+ ms |
| Config | TypeScript, evaluated into the cache key | JSON (static) | JSON (static) |
| Output ownership | **Strict** — wiped before exec AND restore | Additive (stale files survive) | Additive |
| Clean-tree hashing | **Zero reads** (git index OIDs) | git OIDs | re-hash / daemon |
| Daemon required for speed | **No** | Optional | Yes |
| Artifact signing | **Hard-fail** on unsigned | Soft | No |
| Per-task sandbox | **Yes** — kernel-level, opt-in | No | No |
| MCP server for AI agents | **Yes** (`vx mcp`, stdio) | No | No |
| Distributed CI execution | **Yes** — OSS, self-hostable (`vx coordinator` + `vx run --worker`) | No | Paid (Nx Cloud DTE) |
| Local-only dashboard SPA | **Yes** (`vx insights serve`, DuckDB-WASM) | No | Paid |
| Cloudflare-template cloud | **Yes** (`apps/cloud/`, `wrangler deploy`) | Vercel-only | No (proprietary) |
| Plugin API | **Yes** — Vite-style lifecycle hooks | No | Yes (TS-tied) |
| Predictive scheduling | **Yes** (opt-in: `predictive: true`) | No | No |
| OTel CI/CD spans | **Yes** (`OTEL_EXPORTER_OTLP_ENDPOINT`) | No | Paid |
| Install | **Single binary** — 1 curl line | npm + Node | npm + Node |

¹ Wall-clock, direct binaries, same machine and workspace — full
methodology and more scenarios in
Expand Down Expand Up @@ -188,14 +245,32 @@ Side-by-side feature matrix + every known gap: [`docs/comparison.md`](./docs/com

## Architecture (one paragraph)

`bin.ts → cli.ts` dispatches subcommands. `orchestrator.ts:run()` calls `prepareRun()` which discovers the workspace, loads configs, builds the package + task graph, and opens the cache (local SQLite + optional remote layer). The scheduler runs the graph in topological order with bounded concurrency; each task hits the cache (hash → get → restore on hit; spawn → save on miss) or short-circuits as a group / persistent. Outcomes go to the run-history table for direct SQL analytics. Every module has a docs page; every interface is a swappable seam.

Read [`docs/architecture.md`](./docs/architecture.md) for the module map and design principles.
`bin.ts → cli/index.ts` dispatches subcommands.
`orchestrator/run.ts:run()` calls `prepareRun()` which discovers the
workspace, loads configs, builds the package + task graph, opens the
cache (local SQLite + optional remote layer), loads
`HistoryProvider` (if `predictive: true`), and installs plugins from
`vx.workspace.ts`. The scheduler runs the graph in topological order
with bounded concurrency; each task hits the cache (hash → get →
restore on hit; spawn → save on miss) or short-circuits as a group /
persistent. Every observation flows through one event bus —
terminal renderer, MCP server, OTel bridge, user plugins, and cloud
uploader all subscribe to it. The on-wire form (JSON-RPC 2.0 +
OTel-LogRecord-shaped payload) is the same across WS / SSE / NDJSON
on `vx serve` and across the distributed-CI coordinator. Every
module has a docs page; every interface is a swappable seam.

Read [`docs/architecture.md`](./docs/architecture.md) for the module
map. The 2026-06 platform arc is documented under
[`docs/design/`](./docs/design/) and
[`docs/progress/implementation-log-2026-06.md`](./docs/progress/implementation-log-2026-06.md).

## Documentation

Full technical docs live under [`docs/`](./docs/):

**Core**

- [`docs/architecture.md`](./docs/architecture.md) — module map + data flow
- [`docs/schema.md`](./docs/schema.md) — every config field
- [`docs/caching.md`](./docs/caching.md) — cache-key derivation + invalidation table
Expand All @@ -204,11 +279,40 @@ Full technical docs live under [`docs/`](./docs/):
- [`docs/comparison.md`](./docs/comparison.md) — Turbo / Nx / vite-task feature matrix
- [`docs/modules/`](./docs/modules/) — one reference page per source module

## Status
**Design + 2026-06 platform arc** (`docs/design/`)

- [`architecture-north-star-2026-06.md`](./docs/design/architecture-north-star-2026-06.md) — the unified vision
- [`architecture-review-2026-06.md`](./docs/design/architecture-review-2026-06.md) — review + applied checklist
- [`wire-protocol-2026-06.md`](./docs/design/wire-protocol-2026-06.md) — JSON-RPC 2.0 + OTel envelope (shipped)
- [`distributed-ci-2026-06.md`](./docs/design/distributed-ci-2026-06.md) — coordinator + worker (Phase A-B shipped)
- [`vx-cloud-2026-06.md`](./docs/design/vx-cloud-2026-06.md) — Cloudflare cloud (Phases A-C shipped)
- [`extension-protocol-2026-06.md`](./docs/design/extension-protocol-2026-06.md) — subscriber/inspector/driver/plugin (Phase 1 shipped)
- [`predictive-execution-2026-06.md`](./docs/design/predictive-execution-2026-06.md) — history-aware scheduling (Phase A-B shipped)
- [`docs/progress/implementation-log-2026-06.md`](./docs/progress/implementation-log-2026-06.md) — phase-by-phase narrative

**Pre-alpha.** The schema is settling; we bump `CACHE_VERSION` rather than maintain back-compat. 500+ tests; CI green on every commit; the project dogfoods itself (`bun run ci` → `vx run ci`).
## Status

Production readiness: not yet. The semantics are solid; the rough edges are operational (Windows unsupported, no published versions on npm, no managed remote-cache offering).
**Pre-alpha.** The schema is settling; we bump `CACHE_VERSION` rather
than maintain back-compat. **958+ tests across 70 files; CI green on
every commit**; the project dogfoods itself (`bun run ci` → `vx run ci`).

Production readiness for the **core task runner**: the semantics are
solid. The rough edges are operational (Windows unsupported, no
published versions on npm).

Production readiness for the **2026-06 platform layer**:

| Surface | Maturity | Notes |
| -------------------------------------------- | -------------------------------- | ------------------------------------------------------------------------ |
| Core task runner + caching | **production-ready** | dogfooded continuously; 836 tests pre-existing, all green |
| `vx mcp` | **shippable** | live cache.db tools, stdio; agents work today |
| `vx serve` (WS + SSE + NDJSON, JSON-RPC 2.0) | **shippable** | accepts both legacy + new envelope; `curl` works |
| `vx coordinator` + `vx run --worker` | **shippable for self-hosted CI** | content-addressed assignment, disconnect recovery |
| Plugin API | **shippable** | crash-isolated, lifecycle hooks fire end-to-end |
| Predictive scheduling | **shippable as opt-in** | gated on `predictive: true` + observed data |
| `apps/cloud/` (Cloudflare deployment) | **shippable scaffold** | HMAC verify + queue→D1 + DO submit live; OAuth deferred |
| `apps/insights/` (Solid SPA) | **scaffold** | DuckDB-WASM cache.db read works; the SPA pages need real-world iteration |
| `packages/otel-bridge/` | **shippable** | env-var auto-attach in `run()`; ships event stream to any OTLP backend |

## Development

Expand Down
103 changes: 103 additions & 0 deletions apps/cloud/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
# @vzn/vx-cloud

vx Cloud is a Cloudflare Workers reference deployment for hosted
observability, cache, and execution for `@vzn/vx`. Same code runs the
hosted SaaS at `cloud.vx.dev`; clone it, deploy into your own CF
account, and you have a private hosted vx for your team.

See [`docs/design/vx-cloud-2026-06.md`](../../docs/design/vx-cloud-2026-06.md)
for the architecture rationale and
[`docs/design/wire-protocol-2026-06.md`](../../docs/design/wire-protocol-2026-06.md)
for the JSON-RPC 2.0 envelope every endpoint speaks.

## Bindings at a glance

| Binding | Type | Purpose |
| ----------------- | ----------------------- | ------------------------------------------------------------- |
| `DB` | D1 | orgs, members, api_tokens, runs, run_tasks, run_events |
| `ARTIFACTS` | R2 | content-addressed cache artifacts, per-org key prefix |
| `TOKEN_CACHE` | KV | bearer-token → (org, role) lookup cache (TTL 60s) |
| `EVENT_INGEST` | Queue | buffer for `/v1/events/ingest`, consumer batches into D1 |
| `RUN_COORDINATOR` | Durable Object | one DO per live run; holds graph + WS subscribers |
| `INFLIGHT_DEDUP` | Durable Object | one DO per task hash; first-claim wins, others wait |

## Routes

| Method | Path | Purpose |
| ------ | ----------------------------- | ------------------------------------------------------ |
| GET | `/` | minimal status landing page |
| GET | `/health` | `{ok: true}` |
| GET | `/version` | protocol version + supported channels + RPC methods |
| PUT | `/v8/artifacts/:hash` | Turbo-wire cache PUT (HMAC-validated when key set) |
| GET | `/v8/artifacts/:hash` | Turbo-wire cache GET (HMAC-verified when key set) |
| HEAD | `/v8/artifacts/:hash` | R2 head check |
| POST | `/v1/events/ingest` | batched WireEvent uploader → queue → D1 |
| GET | `/v1/runs` | list org runs (most-recent first) |
| GET | `/v1/runs/:runId` | single run + tasks |
| GET | `/v1/runs/:runId/events` | SSE stream of WireEvents for a run |
| GET | `/v1/ws` | WS upgrade, delegates to `RunCoordinatorDO` |

All `/v8/*` and `/v1/*` routes require `Authorization: Bearer <token>`.
Loopback (`localhost`, `127.0.0.1`) bypasses auth for local development.

## Deploy

```sh
# From this directory.
bun install

# One-time auth.
bun wrangler login

# Provision the bindings (each command prints an id — paste into wrangler.toml).
bun wrangler d1 create vx_cloud
# → copy `database_id` into [[d1_databases]] block

bun wrangler r2 bucket create vx-cloud-artifacts
# → no id needed; bucket_name is enough

bun wrangler kv namespace create TOKEN_CACHE
# → copy `id` into [[kv_namespaces]] block

bun wrangler queues create vx-event-ingest
bun wrangler queues create vx-event-ingest-dlq

# Apply the D1 schema.
bun wrangler d1 migrations apply vx_cloud

# (Optional) HMAC signing key for cache artifacts — Turbo-wire-compatible.
bun wrangler secret put VX_REMOTE_CACHE_SIGNATURE_KEY

# Deploy.
bun wrangler deploy
```

The deploy URL prints at the end: `https://vx-cloud-<subdomain>.workers.dev`.
Point a `vx run` at it via:

```sh
export VX_REMOTE_CACHE_URL=https://vx-cloud-<subdomain>.workers.dev/v8/artifacts
export VX_REMOTE_CACHE_TOKEN=<a token you issued>
vx run build test
```

## Hyperdrive escape hatch

D1 caps a single database at 10 GB. When your team outgrows it, swap
the `[[d1_databases]]` block for a [Hyperdrive](https://developers.cloudflare.com/hyperdrive/)
binding pointing at an external Postgres (RDS, Neon, Supabase). The
schema in `migrations/0001_init.sql` is compatible Postgres syntax
modulo `STRICT` keywords — port directly.

## Status

This is a **scaffold** (2026-06-21). Handlers are wired and persist to
the right backends, but several pieces are TODO-marked:

- HMAC signing/verification on cache PUT/GET
- per-run monotonic `seq` allocation in the queue consumer
- `submit.run` graph fan-out in `RunCoordinatorDO`
- waiter broadcast from `InflightDedupDO`
- GitHub OAuth login flow

Track progress in [`docs/progress/`](../../docs/progress/).
Loading
Loading