diff --git a/docs/session-startup-benchmarks.md b/docs/session-startup-benchmarks.md new file mode 100644 index 00000000..f11ab3f5 --- /dev/null +++ b/docs/session-startup-benchmarks.md @@ -0,0 +1,163 @@ +# Session-start latency baselines + +Workcell starts an isolated session by resolving the runtime image, booting the +container runtime (Colima today, Apple `container` under evaluation in C1), and +completing the supervisor handshake. **C2** measures that start latency and drives +it down with cached images and an optional kept-warm lane — the sibling of +[syscall-shim-benchmarks.md](syscall-shim-benchmarks.md) (C5). Numbers are captured +on a host with a live runtime, not in PR CI (a real start needs a booted VM); the +results tables below are **placeholders pending a live capture** +([Filling in the numbers](#filling-in-the-numbers)) — not measured values. + +## What is measured + +A **sample** is one full session start: the wall-clock time from invoking the +session-start command to the point the session is ready. The harness times three +modes that span the latency shapes C2 targets: + +| Mode | Runtime state before the sample | What it isolates | +|---|---|---| +| `cold` | image not cached, no kept-warm session | worst-case first start (image resolve + full boot) | +| `warm` | image cached and a kept-warm session available | best-case start off the kept-warm lane | +| `cache-hit` | image cached, no kept-warm session | the image-cache win alone, without the warm lane | + +The `cold` vs `warm` delta is the headline C2 number; `cache-hit` separates the +image-cache win from the kept-warm-lane win so each is credited independently. + +## Methodology + +The harness (`scripts/bench/startup-bench.sh`) times one mode: `WORKCELL_STARTUP_ +WARMUP` discarded launches settle first-touch page-cache/loader costs, then +`WORKCELL_STARTUP_ITERATIONS` measured launches run inside one long-lived timer +process (like C5's in-process loop) on a **monotonic** clock (`CLOCK_MONOTONIC`), +so neither an NTP step/sleep-wake nor a per-sample interpreter launch corrupts or +inflates a sample. Stats conventions match the C5 exec-guard harness, so the pages +compare directly: + +- **median** (`sorted[floor(n/2)]`, the outlier-robust headline), **p90** + (`sorted[floor(n*9/10)]` clamped, the tail a slow start shows), **mean/stddev** + (population, to expose a skewed distribution), and **min/max** (observed range). + +The driver (`scripts/bench/run-startup-bench.sh`) establishes each mode's runtime +state through a per-mode prep hook (`WORKCELL_STARTUP_COLD_PREP` / +`WORKCELL_STARTUP_CACHE_HIT_PREP` / `WORKCELL_STARTUP_WARM_PREP` — e.g. evicting +the cached image and stopping the kept-warm session for `cold`, pre-pulling the +image but leaving the warm lane down for `cache-hit`, or pre-pulling and priming +the warm lane for `warm`), then times the configured `WORKCELL_STARTUP_CMD` for +that mode. The whole measurement is repeated for `WORKCELL_STARTUP_RUNS` passes. + +Live runs are guarded so a misconfigured capture cannot look publishable: + +- **Every driven mode's prep hook is required** (`*_COLD_PREP` / + `*_CACHE_HIT_PREP` / `*_WARM_PREP`); an unset hook fails fast rather than + measuring whatever state happened to be present. +- **The runtime must be usable, not just installed** — a cheap read-only probe + (`docker info` / `colima status` / `container system status`) sends a + client-only host to the clean CI-safe skip. `WORKCELL_STARTUP_RUNTIME` overrides + and skips the probe. +- **`WORKCELL_STARTUP_RUNS >= 2`** — the stability gate needs cross-run evidence, + so a single-run capture is rejected. + +For `cold` **and `cache-hit`** the driver re-runs the mode's prep hook before +**every** measured sample (warmup `0`) and aggregates the per-sample timings — a +start warms the cache/lane the next start would spend, so prepping once per pass +would leave only the first sample genuine; those hooks must be **repeatable**. +Only `warm` legitimately shares one prep per pass and keeps `WORKCELL_STARTUP_WARMUP`. + +`WORKCELL_STARTUP_CMD` is parsed with shell quoting, so a spaced argument keeps its +boundary (`--workspace '/path/with space'` stays one argv element, not word-split +tokens). The canned dry run needs no prep hooks or runtime and never executes +hooks — these guards are live-only. + +### The cross-run stability gate + +Reproducibility is the C2 acceptance bar, so the driver enforces it: after all +runs it computes, per mode, the run-to-run **median** spread as a percentage of +the smallest run's median, and **fails** (non-zero exit) if any mode exceeds +`WORKCELL_STARTUP_STABILITY_PCT` (default 15%) — the evidence a published number +repeats. A zero median fails the gate outright: a 0 ns start is impossible, so it +signals a broken clock rather than a 0% spread that would read as `STABLE`. + +### Runner caveats + +Numbers are **relative** to the host's hardware and runtime backend, so treat the +cold-vs-warm delta (not absolute medians) as the portable signal; session start +includes VM boot, so expect a wider stddev than the C5 exec-guard numbers. + +## Results + +**Status: numbers pending live capture.** Replace the `TODO` cells once a live run +is captured (see [Filling in the numbers](#filling-in-the-numbers)); do not +fabricate values. + +### Start latency by mode (median of N samples, R runs) + +| Mode | Median (ns) | p90 (ns) | Mean (ns) | Stddev (ns) | vs cold | +|---|---|---|---|---|---| +| `cold` | TODO | TODO | TODO | TODO | — | +| `cache-hit` | TODO | TODO | TODO | TODO | TODO | +| `warm` | TODO | TODO | TODO | TODO | TODO | + +### Cross-run stability (median) + +| Mode | Min median (ns) | Max median (ns) | Spread (ns) | Spread (%) | Verdict | +|---|---|---|---|---|---| +| `cold` | TODO | TODO | TODO | TODO | TODO | +| `cache-hit` | TODO | TODO | TODO | TODO | TODO | +| `warm` | TODO | TODO | TODO | TODO | TODO | + +## Filling in the numbers + +On a host with a live runtime, run the driver (see [Rerunning](#rerunning)) with +`WORKCELL_STARTUP_OUTPUT` set. A `0` exit means the stability gate passed and the +numbers are reproducible (non-zero means the spread was too wide — fix first). +Transcribe the report's medians, p90s and stability table into the tables above, +fill `vs cold`, and record the host, runtime backend, and `N`/`R`. + +## Rerunning + +From the repository root on a host with a container runtime: + +```sh +# A live run needs all three prep hooks and RUNS >= 2. WORKCELL_STARTUP_CMD is +# shell-quoted; COLD_PREP/CACHE_HIT_PREP re-run per sample, so make them idempotent. +export WORKCELL_STARTUP_CMD='./scripts/workcell ' +export WORKCELL_STARTUP_COLD_PREP='' +export WORKCELL_STARTUP_CACHE_HIT_PREP='' +export WORKCELL_STARTUP_WARM_PREP='' +export WORKCELL_STARTUP_OUTPUT=session-startup-results.md + +# Defaults: 5 iterations, 1 warmup, 2 runs, 15% stability threshold. +./scripts/bench/run-startup-bench.sh +``` + +Tunable via environment: `WORKCELL_STARTUP_ITERATIONS`, `WORKCELL_STARTUP_WARMUP` +(forced to `0` for `cold`/`cache-hit`), `WORKCELL_STARTUP_RUNS`, +`WORKCELL_STARTUP_STABILITY_PCT`, `WORKCELL_STARTUP_CMD`, the three `*_PREP` hooks, +and `WORKCELL_STARTUP_OUTPUT`. Numeric controls are validated up front +(`ITERATIONS`/`RUNS` `>= 1`, `WARMUP`/`STABILITY_PCT` `>= 0`); anything else fails +fast rather than silently misreporting. + +### Dry run without a runtime + +The driver is CI-safe: with no runtime it exits `0` with a clear skip message. To +rehearse the full report and stability gate on any host, feed canned samples — the +dry run runs no prep hooks (any exported `*_PREP` is ignored) and times nothing: + +```sh +# One stable run set (gate passes, exit 0): +WORKCELL_STARTUP_SAMPLES_NS='10 20 30 40 50' ./scripts/bench/run-startup-bench.sh + +# Two ';'-separated per-run groups with divergent medians (gate fails, exit 2): +WORKCELL_STARTUP_SAMPLES_NS='10 20 30;100 200 300' ./scripts/bench/run-startup-bench.sh +``` + +The unit tests in `internal/startupbench` use this canned path (plus benign live +targets) to pin the stats math, the gate, and the guards without a container. + +## Where this fits + +The harness and driver live in `scripts/bench/` alongside the C5 exec-guard +benchmark ([syscall-shim-benchmarks.md](syscall-shim-benchmarks.md)). A scheduled, +non-PR-blocking lane that captures the live numbers on a runtime-capable runner is +**deferred** until the image build is unblocked. diff --git a/internal/startupbench/bench_test.go b/internal/startupbench/bench_test.go new file mode 100644 index 00000000..30cdb083 --- /dev/null +++ b/internal/startupbench/bench_test.go @@ -0,0 +1,573 @@ +// SPDX-License-Identifier: Apache-2.0 +// Copyright 2026 Omkhar Arasaratnam + +// Package startupbench pins the pure logic of the C2 session-start latency bench +// scripts (median/p90/stddev math, stability gate, driver skip/validation guards) +// so it runs under `go test ./...` with no runtime. +package startupbench + +import ( + "bytes" + "os" + "os/exec" + "path/filepath" + "runtime" + "strconv" + "strings" + "testing" +) + +func repoRoot(tb testing.TB) string { + tb.Helper() + _, file, _, ok := runtime.Caller(0) + if !ok { + tb.Fatal("unable to determine repo root") + } + return filepath.Clean(filepath.Join(filepath.Dir(file), "..", "..")) +} + +// hermeticEnv builds an explicit child env (not inherited) so a stray exported +// WORKCELL_STARTUP_* can't leak in: only PATH/HOME/TMPDIR + extra (which overrides). +func hermeticEnv(extra map[string]string) []string { + base := map[string]string{"PATH": os.Getenv("PATH")} + for _, k := range []string{"HOME", "TMPDIR"} { + if v, ok := os.LookupEnv(k); ok { + base[k] = v + } + } + for k, v := range extra { + base[k] = v + } + env := make([]string, 0, len(base)) + for k, v := range base { + env = append(env, k+"="+v) + } + return env +} + +// runScriptSplit runs a bench script with args + extra env, returning exit code, +// stdout and stderr separately, with a hermetic child environment. +func runScriptSplit(tb testing.TB, relScript string, env map[string]string, args ...string) (int, string, string) { + tb.Helper() + root := repoRoot(tb) + cmd := exec.Command(filepath.Join(root, filepath.FromSlash(relScript)), args...) + cmd.Dir = root + cmd.Env = hermeticEnv(env) + var stdout, stderr bytes.Buffer + cmd.Stdout = &stdout + cmd.Stderr = &stderr + err := cmd.Run() + if err == nil { + return 0, stdout.String(), stderr.String() + } + if exitErr, ok := err.(*exec.ExitError); ok { + return exitErr.ExitCode(), stdout.String(), stderr.String() + } + tb.Fatalf("run %s failed: %v\n%s%s", relScript, err, stdout.String(), stderr.String()) + return -1, "", "" +} + +// runScript is runScriptSplit with stdout+stderr combined (order not preserved). +func runScript(tb testing.TB, relScript string, env map[string]string, args ...string) (int, string) { + tb.Helper() + code, stdout, stderr := runScriptSplit(tb, relScript, env, args...) + return code, stdout + stderr +} + +// writeExec writes an executable helper script for a test. +func writeExec(tb testing.TB, path, script string) { + tb.Helper() + if err := os.WriteFile(path, []byte(script), 0o755); err != nil { + tb.Fatalf("write %s: %v", path, err) + } +} + +// liveEnv returns a live-run env (auto-detect bypassed, all prep hooks no-op, +// RUNS>=2, gate widened so real timing can't flake) merged with extra, which +// overrides. Tests set only what they exercise. +func liveEnv(extra map[string]string) map[string]string { + env := map[string]string{ + "WORKCELL_STARTUP_RUNTIME": "colima", + "WORKCELL_STARTUP_CMD": "true", + "WORKCELL_STARTUP_RUNS": "2", + "WORKCELL_STARTUP_STABILITY_PCT": "100000000", + "WORKCELL_STARTUP_COLD_PREP": ":", + "WORKCELL_STARTUP_CACHE_HIT_PREP": ":", + "WORKCELL_STARTUP_WARM_PREP": ":", + } + for k, v := range extra { + env[k] = v + } + return env +} + +const harness = "scripts/bench/startup-bench.sh" +const driver = "scripts/bench/run-startup-bench.sh" + +// parseFields turns a "k=v k=v" harness line into a map. +func parseFields(line string) map[string]string { + fields := map[string]string{} + for _, tok := range strings.Fields(strings.TrimSpace(line)) { + if k, v, ok := strings.Cut(tok, "="); ok { + fields[k] = v + } + } + return fields +} + +func TestHarnessStatsOddSampleSet(t *testing.T) { + // Unsorted input; harness sorts. n=5 -> median=3rd value, p90 (idx 4)=max. + code, out := runScript(t, harness, + map[string]string{"WORKCELL_STARTUP_SAMPLES_NS": "50 10 40 20 30"}, + "cold", "0", "0") + if code != 0 { + t.Fatalf("exit %d, out=%q", code, out) + } + f := parseFields(out) + want := map[string]string{ + "mode": "cold", "n": "5", "mean_ns": "30", "median_ns": "30", + "p90_ns": "50", "stddev_ns": "14", "min_ns": "10", "max_ns": "50", + } + for k, v := range want { + if f[k] != v { + t.Errorf("field %s = %q, want %q (line: %s)", k, f[k], v, strings.TrimSpace(out)) + } + } +} + +func TestHarnessMedianEvenSampleSet(t *testing.T) { + // n=6: matches the C5 harness convention median=sorted[n/2] (upper-middle). + code, out := runScript(t, harness, + map[string]string{"WORKCELL_STARTUP_SAMPLES_NS": "10 20 30 40 50 60"}, + "warm", "0", "0") + if code != 0 { + t.Fatalf("exit %d, out=%q", code, out) + } + f := parseFields(out) + if f["median_ns"] != "40" { + t.Errorf("median_ns = %q, want 40", f["median_ns"]) + } + if f["p90_ns"] != "60" { + t.Errorf("p90_ns = %q, want 60", f["p90_ns"]) + } + if f["n"] != "6" { + t.Errorf("n = %q, want 6", f["n"]) + } +} + +func TestHarnessRejectsUnknownMode(t *testing.T) { + code, out := runScript(t, harness, + map[string]string{"WORKCELL_STARTUP_SAMPLES_NS": "10 20 30"}, + "bogus", "0", "0") + if code == 0 { + t.Fatalf("expected non-zero exit for unknown mode, got 0: %s", out) + } + if !strings.Contains(out, "unknown mode") { + t.Errorf("missing 'unknown mode' diagnostic: %s", out) + } +} + +func TestHarnessRejectsNonIntegerSample(t *testing.T) { + code, out := runScript(t, harness, + map[string]string{"WORKCELL_STARTUP_SAMPLES_NS": "10 x 30"}, + "cold", "0", "0") + if code == 0 { + t.Fatalf("expected non-zero exit for non-integer sample, got 0: %s", out) + } + if !strings.Contains(out, "non-integer sample") { + t.Errorf("missing 'non-integer sample' diagnostic: %s", out) + } +} + +func TestHarnessLivePathTimesTarget(t *testing.T) { + // No canned samples: times a benign target on the real clock; assert structure. + code, out := runScript(t, harness, nil, "warm", "3", "1", "--", "true") + if code != 0 { + t.Fatalf("exit %d, out=%q", code, out) + } + f := parseFields(out) + if f["n"] != "3" { + t.Errorf("n = %q, want 3 (line: %s)", f["n"], strings.TrimSpace(out)) + } + for _, k := range []string{"median_ns", "p90_ns", "min_ns", "max_ns"} { + if f[k] == "" { + t.Errorf("missing field %s in live output: %s", k, strings.TrimSpace(out)) + } + } +} + +func TestHarnessTimesLaunchesInOneProcess(t *testing.T) { + // The measured loop must run in ONE long-lived timer process. Each launch + // records its parent PID: all share one PPID (the timer) != the harness's. + dir := t.TempDir() + ppidF := filepath.Join(dir, "ppids") + rec := filepath.Join(dir, "ppid.sh") + writeExec(t, rec, "#!/usr/bin/env bash\necho \"$PPID\" >> \""+ppidF+"\"\n") + root := repoRoot(t) + cmd := exec.Command(filepath.Join(root, filepath.FromSlash(harness)), "warm", "3", "0", "--", rec) + cmd.Dir = root + cmd.Env = hermeticEnv(nil) + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("harness failed: %v\n%s", err, out) + } + data, err := os.ReadFile(ppidF) + if err != nil { + t.Fatalf("read ppids: %v", err) + } + ppids := strings.Fields(string(data)) + if len(ppids) != 3 { + t.Fatalf("want 3 launches, got %d: %q", len(ppids), data) + } + if harnessPid := strconv.Itoa(cmd.Process.Pid); ppids[0] == harnessPid { + t.Errorf("target parent == harness pid %s: launches ran in the harness shell, not a dedicated in-process timer", harnessPid) + } + for _, p := range ppids[1:] { + if p != ppids[0] { + t.Errorf("launches had different parents %v; want a single timer process", ppids) + } + } +} + +func TestDriverSkipsWithoutRuntime(t *testing.T) { + code, out := runScript(t, driver, map[string]string{"WORKCELL_STARTUP_RUNTIME": "none"}) + if code != 0 { + t.Fatalf("skip should exit 0, got %d: %s", code, out) + } + if !strings.Contains(out, "skipping") || !strings.Contains(out, "no container runtime") { + t.Errorf("missing clean-skip message: %s", out) + } +} + +func TestDriverDryRunStablePasses(t *testing.T) { + code, out := runScript(t, driver, + map[string]string{"WORKCELL_STARTUP_SAMPLES_NS": "10 20 30 40 50"}) + if code != 0 { + t.Fatalf("stable dry run should exit 0, got %d: %s", code, out) + } + for _, want := range []string{ + "# session-start latency benchmark results", + "| cold |", "| warm |", + "Cross-run stability (median)", + "Stability gate: STABLE", + } { + if !strings.Contains(out, want) { + t.Errorf("report missing %q\n---\n%s", want, out) + } + } +} + +func TestDriverDryRunUnstableFailsGate(t *testing.T) { + // Two groups with very different medians (20 vs 200) exceed the 15% threshold. + code, out := runScript(t, driver, + map[string]string{"WORKCELL_STARTUP_SAMPLES_NS": "10 20 30;100 200 300"}) + if code != 2 { + t.Fatalf("unstable dry run should exit 2, got %d: %s", code, out) + } + if !strings.Contains(out, "Stability gate: UNSTABLE") { + t.Errorf("missing UNSTABLE gate verdict: %s", out) + } + if !strings.Contains(out, "stability gate FAILED") { + t.Errorf("missing gate-failure diagnostic: %s", out) + } +} + +func TestDriverColdSkipsWarmupAndDrivesCacheHit(t *testing.T) { + // cold/cache-hit force warmup=0 (re-prep per sample), only warm keeps it; a stub harness logs each mode's warmup. + dir := t.TempDir() + logPath := filepath.Join(dir, "harness.log") + stub := filepath.Join(dir, "stub-harness.sh") + writeExec(t, stub, "#!/usr/bin/env bash\n"+ + "set -euo pipefail\n"+ + "mode=\"$1\"; iters=\"$2\"; warmup=\"$3\"\n"+ + "printf '%s %s\\n' \"$mode\" \"$warmup\" >> \"${HARNESS_LOG}\"\n"+ + "printf 'mode=%s n=%s mean_ns=1 median_ns=1 p90_ns=1 stddev_ns=0 min_ns=1 max_ns=1\\n' \"$mode\" \"$iters\"\n") + env := liveEnv(map[string]string{ + "HARNESS_LOG": logPath, + "WORKCELL_STARTUP_HARNESS": stub, + "WORKCELL_STARTUP_ITERATIONS": "2", + "WORKCELL_STARTUP_WARMUP": "1", + }) + code, out := runScript(t, driver, env) + if code != 0 { + t.Fatalf("driver exit %d: %s", code, out) + } + if !strings.Contains(out, "| cache-hit |") { + t.Errorf("report missing cache-hit row (P2): %s", out) + } + data, err := os.ReadFile(logPath) + if err != nil { + t.Fatalf("read harness log: %v", err) + } + warmupByMode := map[string]string{} + for _, ln := range strings.Split(strings.TrimSpace(string(data)), "\n") { + parts := strings.Fields(ln) + if len(parts) == 2 { + warmupByMode[parts[0]] = parts[1] + } + } + for _, mode := range []string{"cold", "cache-hit", "warm"} { + if _, ok := warmupByMode[mode]; !ok { + t.Errorf("driver never invoked harness for mode %q; saw %v", mode, warmupByMode) + } + } + if got := warmupByMode["cold"]; got != "0" { + t.Errorf("cold warmup = %q, want 0 (must not warm before measuring)", got) + } + if got := warmupByMode["cache-hit"]; got != "0" { + t.Errorf("cache-hit warmup = %q, want 0 (re-preps per sample like cold)", got) + } + if got := warmupByMode["warm"]; got != "1" { + t.Errorf("warm warmup = %q, want 1 (configured warmup preserved)", got) + } +} + +func TestRunScriptEnvIsHermetic(t *testing.T) { + // A stray exported WORKCELL_STARTUP_* must not leak in: a no-runtime run must still SKIP (not a dry run) nor run the hook. + t.Setenv("WORKCELL_STARTUP_SAMPLES_NS", "999") + t.Setenv("WORKCELL_STARTUP_COLD_PREP", "echo LEAKED_PREP_RAN") + code, out := runScript(t, driver, map[string]string{"WORKCELL_STARTUP_RUNTIME": "none"}) + if code != 0 { + t.Fatalf("hermetic skip run should exit 0, got %d: %s", code, out) + } + if !strings.Contains(out, "skipping") || !strings.Contains(out, "no container runtime") { + t.Errorf("stray WORKCELL_STARTUP_SAMPLES_NS leaked (expected a clean skip): %s", out) + } + if strings.Contains(out, "LEAKED_PREP_RAN") { + t.Errorf("stray prep hook leaked into the run: %s", out) + } +} + +func TestDriverColdAndCacheHitRepPerSample(t *testing.T) { + // cold/cache-hit re-run their prep hook per sample, warm once per pass; hooks append a byte per call. + dir := t.TempDir() + coldF := filepath.Join(dir, "cold") + warmF := filepath.Join(dir, "warm") + chF := filepath.Join(dir, "cachehit") + env := liveEnv(map[string]string{ + "WORKCELL_STARTUP_ITERATIONS": "3", + "WORKCELL_STARTUP_WARMUP": "0", + "WORKCELL_STARTUP_COLD_PREP": "printf c >> " + coldF, + "WORKCELL_STARTUP_WARM_PREP": "printf w >> " + warmF, + "WORKCELL_STARTUP_CACHE_HIT_PREP": "printf h >> " + chF, + }) + code, out := runScript(t, driver, env) + if code != 0 { + t.Fatalf("driver exit %d: %s", code, out) + } + countPreps := func(path string) int { + data, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read prep counter %s: %v", path, err) + } + return len(data) + } + // 3 samples x 2 runs -> 6 preps each for cold/cache-hit; warm -> 2 (one/pass). + if got := countPreps(coldF); got != 6 { + t.Errorf("cold prep ran %d time(s), want 6 (per measured sample x 2 runs)", got) + } + if got := countPreps(chF); got != 6 { + t.Errorf("cache-hit prep ran %d time(s), want 6 (per measured sample x 2 runs)", got) + } + if got := countPreps(warmF); got != 2 { + t.Errorf("warm prep ran %d time(s), want 2 (one prep per pass x 2 runs)", got) + } + if !strings.Contains(out, "| cache-hit |") || !strings.Contains(out, " 3 |") { + t.Errorf("aggregated cache-hit row missing/incorrect: %s", out) + } +} + +func TestDriverDryRunSkipsPrep(t *testing.T) { + // A canned dry run must NEVER execute prep hooks; the marker must not exist. + dir := t.TempDir() + marker := filepath.Join(dir, "prep-ran") + env := map[string]string{ + "WORKCELL_STARTUP_SAMPLES_NS": "10 20 30 40 50", + "WORKCELL_STARTUP_COLD_PREP": "printf c >> " + marker, + "WORKCELL_STARTUP_WARM_PREP": "printf w >> " + marker, + "WORKCELL_STARTUP_CACHE_HIT_PREP": "printf h >> " + marker, + } + code, out := runScript(t, driver, env) + if code != 0 { + t.Fatalf("dry run should exit 0, got %d: %s", code, out) + } + if _, err := os.Stat(marker); !os.IsNotExist(err) { + data, _ := os.ReadFile(marker) + t.Errorf("dry run executed prep hook(s) (marker = %q); dry runs must skip prep", data) + } + if !strings.Contains(out, "Stability gate: STABLE") { + t.Errorf("dry run should still produce a stable report: %s", out) + } +} + +func TestDriverPrepOutputStaysOffReport(t *testing.T) { + // A prep hook's stdout (e.g. `docker pull`) must go to stderr, not the stdout report (else `run.sh > report.md` breaks). + env := liveEnv(map[string]string{ + "WORKCELL_STARTUP_COLD_PREP": "echo PREP_STDOUT_MARKER", + "WORKCELL_STARTUP_CACHE_HIT_PREP": "echo PREP_STDOUT_MARKER", + "WORKCELL_STARTUP_WARM_PREP": "echo PREP_STDOUT_MARKER", + }) + code, stdout, stderr := runScriptSplit(t, driver, env) + if code != 0 { + t.Fatalf("driver exit %d:\n%s%s", code, stdout, stderr) + } + if strings.Contains(stdout, "PREP_STDOUT_MARKER") { + t.Errorf("prep-hook stdout leaked into the report stream:\n%s", stdout) + } + if !strings.Contains(stdout, "# session-start latency benchmark results") { + t.Errorf("report missing from stdout:\n%s", stdout) + } + if !strings.Contains(stderr, "PREP_STDOUT_MARKER") { + t.Errorf("prep-hook output should appear on stderr:\n%s", stderr) + } +} + +func TestDriverRejectsInvalidNumericControls(t *testing.T) { + // Invalid numeric controls (RUNS=0, non-integer) must fail fast, not exit 0. + cases := []struct{ name, key, val string }{ + {"RUNS_zero", "WORKCELL_STARTUP_RUNS", "0"}, + {"RUNS_nonnumeric", "WORKCELL_STARTUP_RUNS", "abc"}, + {"ITERATIONS_zero", "WORKCELL_STARTUP_ITERATIONS", "0"}, + {"WARMUP_negative", "WORKCELL_STARTUP_WARMUP", "-1"}, + {"STABILITY_PCT_nonnumeric", "WORKCELL_STARTUP_STABILITY_PCT", "5x"}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + env := map[string]string{ + "WORKCELL_STARTUP_SAMPLES_NS": "10 20 30", + c.key: c.val, + } + code, out := runScript(t, driver, env) + if code == 0 { + t.Fatalf("expected non-zero exit for %s=%s, got 0: %s", c.key, c.val, out) + } + if !strings.Contains(out, c.key) { + t.Errorf("error should name the offending control %s: %s", c.key, out) + } + }) + } +} + +func TestDriverPreservesCommandArgv(t *testing.T) { + // Shell-quoted CMD: a spaced arg (--workspace '/a b') must reach as one element. + dir := t.TempDir() + argvF := filepath.Join(dir, "argv") + rec := filepath.Join(dir, "record.sh") + writeExec(t, rec, "#!/usr/bin/env bash\n"+ + "set -euo pipefail\n"+ + ": > \"${ARGV_FILE}\"\n"+ + "for a in \"$@\"; do printf '%s\\n' \"$a\" >> \"${ARGV_FILE}\"; done\n") + env := liveEnv(map[string]string{ + "ARGV_FILE": argvF, + "WORKCELL_STARTUP_CMD": rec + " alpha 'beta gamma'", + "WORKCELL_STARTUP_ITERATIONS": "1", + "WORKCELL_STARTUP_WARMUP": "0", + }) + code, out := runScript(t, driver, env) + if code != 0 { + t.Fatalf("driver exit %d: %s", code, out) + } + data, err := os.ReadFile(argvF) + if err != nil { + t.Fatalf("read argv file: %v", err) + } + got := strings.Split(strings.TrimRight(string(data), "\n"), "\n") + want := []string{"alpha", "beta gamma"} + if len(got) != len(want) || got[0] != want[0] || got[1] != want[1] { + t.Errorf("target argv = %q, want %q (word-splitting would break the spaced arg)\n%s", got, want, out) + } +} + +func TestDriverLiveRequiresPrepHooks(t *testing.T) { + // On a LIVE run a missing mode prep hook must fail fast (naming mode + env var); dry-run needs none. + live := map[string]string{ + "WORKCELL_STARTUP_RUNTIME": "colima", // bypass the no-runtime skip + "WORKCELL_STARTUP_CMD": "true", + // WORKCELL_STARTUP_COLD_PREP deliberately omitted. + "WORKCELL_STARTUP_CACHE_HIT_PREP": ":", + "WORKCELL_STARTUP_WARM_PREP": ":", + } + code, out := runScript(t, driver, live) + if code == 0 { + t.Fatalf("live run with a missing cold prep hook should fail, got exit 0: %s", out) + } + if !strings.Contains(out, "WORKCELL_STARTUP_COLD_PREP") || !strings.Contains(out, "cold") { + t.Errorf("missing-prep error must name the mode and the env var: %s", out) + } + code, out = runScript(t, driver, + map[string]string{"WORKCELL_STARTUP_SAMPLES_NS": "10 20 30 40 50"}) + if code != 0 { + t.Fatalf("dry-run with no prep hooks should pass, got %d: %s", code, out) + } + if !strings.Contains(out, "Stability gate: STABLE") { + t.Errorf("dry-run should still produce a stable report: %s", out) + } +} + +func TestDriverSkipsWhenRuntimeClientButNoDaemon(t *testing.T) { + // Runtime client present but daemon unusable must cleanly skip. Fake clients whose health probe fails, first on PATH. + dir := t.TempDir() + for _, name := range []string{"colima", "container", "docker"} { + writeExec(t, filepath.Join(dir, name), "#!/usr/bin/env bash\nexit 1\n") + } + env := map[string]string{ + // Fakes first, then the real system bins the driver needs. + "PATH": dir + string(os.PathListSeparator) + os.Getenv("PATH"), + } + code, out := runScript(t, driver, env) + if code != 0 { + t.Fatalf("client-only host should cleanly skip (exit 0), got %d: %s", code, out) + } + if !strings.Contains(out, "skipping") || !strings.Contains(out, "no container runtime") { + t.Errorf("expected clean skip when only the client (no daemon) is present: %s", out) + } +} + +func TestDriverLiveRequiresTwoRuns(t *testing.T) { + // A single-run live capture has no repeatability evidence, so RUNS >= 2 is required. + live := liveEnv(map[string]string{"WORKCELL_STARTUP_RUNS": "1"}) + code, out := runScript(t, driver, live) + if code == 0 { + t.Fatalf("live run with RUNS=1 should fail fast, got exit 0: %s", out) + } + if !strings.Contains(out, "WORKCELL_STARTUP_RUNS") || !strings.Contains(out, ">= 2") { + t.Errorf("error should require RUNS >= 2 for a live run: %s", out) + } + // Dry-run with RUNS=1 is a rehearsal, not gated, and must keep working. + code, out = runScript(t, driver, map[string]string{ + "WORKCELL_STARTUP_SAMPLES_NS": "10 20 30 40 50", + "WORKCELL_STARTUP_RUNS": "1", + }) + if code != 0 { + t.Fatalf("dry-run with RUNS=1 should still pass, got %d: %s", code, out) + } +} + +func TestDriverZeroMedianIsUnstable(t *testing.T) { + // A 0 median in one run vs nonzero in another is degenerate (impossible), not a 0% spread; the gate must fail. + code, out := runScript(t, driver, + map[string]string{"WORKCELL_STARTUP_SAMPLES_NS": "0 0 0;10 20 30"}) + if code != 2 { + t.Fatalf("zero-vs-nonzero medians should fail the gate (exit 2), got %d: %s", code, out) + } + if !strings.Contains(out, "Stability gate: UNSTABLE") { + t.Errorf("expected UNSTABLE verdict for a zero median: %s", out) + } + if strings.Contains(out, "Stability gate: STABLE") { + t.Errorf("a zero median must not read as STABLE: %s", out) + } +} + +func TestDriverStabilityThresholdIsConfigurable(t *testing.T) { + // The same 5% spread passes the default threshold but a 1% threshold rejects it. + env := map[string]string{ + "WORKCELL_STARTUP_SAMPLES_NS": "10 20 30;11 21 31", + "WORKCELL_STARTUP_STABILITY_PCT": "1", + } + code, out := runScript(t, driver, env) + if code != 2 { + t.Fatalf("5%% spread under a 1%% threshold should fail (exit 2), got %d: %s", code, out) + } + if !strings.Contains(out, "UNSTABLE") { + t.Errorf("missing UNSTABLE verdict at tight threshold: %s", out) + } +} diff --git a/scripts/bench/run-startup-bench.sh b/scripts/bench/run-startup-bench.sh new file mode 100755 index 00000000..1cdc3908 --- /dev/null +++ b/scripts/bench/run-startup-bench.sh @@ -0,0 +1,313 @@ +#!/usr/bin/env -S BASH_ENV= ENV= bash +# +# run-startup-bench.sh -- drive the session-start latency benchmark (C2). +# +# For each mode (cold, cache-hit, warm) it runs a per-mode prep hook, then times +# WORKCELL_STARTUP_ITERATIONS session starts via scripts/bench/startup-bench.sh. +# `cold` and `cache-hit` re-prep before every measured sample (warmup 0) so a +# discarded warmup can't spend that state; only `warm` shares one prep per pass. +# Repeated for WORKCELL_STARTUP_RUNS passes; the driver FAILS if any mode's +# run-to-run median spread exceeds the stability threshold (C5's sibling). +# +# With no live runtime the driver exits 0 with a clear skip message; +# WORKCELL_STARTUP_SAMPLES_NS switches to a canned dry-run (no runtime) used by the +# unit tests. All configuration env vars and the full methodology are documented +# in docs/session-startup-benchmarks.md. +set -euo pipefail + +ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/.." +ROOT_DIR="$(cd "${ROOT_DIR}" && pwd)" +# WORKCELL_STARTUP_HARNESS overrides the harness path (test seam only). +HARNESS="${WORKCELL_STARTUP_HARNESS:-${ROOT_DIR}/scripts/bench/startup-bench.sh}" + +MODES="cold cache-hit warm" +ITERATIONS="${WORKCELL_STARTUP_ITERATIONS:-5}" +WARMUP="${WORKCELL_STARTUP_WARMUP:-1}" +RUNS="${WORKCELL_STARTUP_RUNS:-2}" +STABILITY_PCT="${WORKCELL_STARTUP_STABILITY_PCT:-15}" +OUTPUT_PATH="${WORKCELL_STARTUP_OUTPUT:-}" +SAMPLES="${WORKCELL_STARTUP_SAMPLES_NS:-}" + +# Validate a numeric driver control is an integer at/above a floor; fail fast. +validate_int() { + # $1 env var name, $2 value, $3 floor + case "$2" in + '' | *[!0-9]*) + echo "run-startup-bench: $1 must be an integer, got '$2'." >&2 + exit 1 + ;; + esac + if [ "$2" -lt "$3" ]; then + echo "run-startup-bench: $1 must be >= $3, got '$2'." >&2 + exit 1 + fi +} + +# Probe that an auto-detected runtime's daemon is usable (cheap read-only status), +# not just that the client binary exists, so detection can fall through to skip. +runtime_usable() { + case "$1" in + docker) docker info ;; + colima) colima status ;; + container) container system status ;; + *) return 1 ;; + esac >/dev/null 2>&1 +} + +[ -x "${HARNESS}" ] || { + echo "run-startup-bench: harness not found or not executable: ${HARNESS}" >&2 + exit 1 +} + +# Detect an available container runtime. A canned-sample dry run needs none. +DRY_RUN=0 +RUNTIME="" +SAMPLE_GROUPS=() +if [ -n "${SAMPLES}" ]; then + DRY_RUN=1 + RUNTIME="dry-run (canned samples)" + # A ';' separates per-run sample groups; multiple groups define RUNS. + local_ifs="${IFS}" + IFS=';' read -ra SAMPLE_GROUPS <<<"${SAMPLES}" + IFS="${local_ifs}" + if [ "${#SAMPLE_GROUPS[@]}" -gt 1 ]; then + RUNS="${#SAMPLE_GROUPS[@]}" + fi +else + detected="${WORKCELL_STARTUP_RUNTIME:-}" + if [ -z "${detected}" ]; then + # Only select a runtime whose daemon is usable (else an installed-but-dead + # client picks live mode then hard-fails). Explicit override skips the probe. + for candidate in colima container docker; do + command -v "${candidate}" >/dev/null 2>&1 || continue + if runtime_usable "${candidate}"; then + detected="${candidate}" + break + fi + done + fi + if [ -z "${detected}" ] || [ "${detected}" = "none" ]; then + echo "run-startup-bench: no container runtime (Colima / Apple container) is" \ + "available on this host; session-start latency needs a live runtime." >&2 + echo "run-startup-bench: skipping (clean exit). Set WORKCELL_STARTUP_SAMPLES_NS" \ + "for a canned dry run, or run on a host with a runtime. See" \ + "docs/session-startup-benchmarks.md." >&2 + exit 0 + fi + RUNTIME="${detected}" + if [ -z "${WORKCELL_STARTUP_CMD:-}" ]; then + echo "run-startup-bench: WORKCELL_STARTUP_CMD (the session-start command to" \ + "time) is required for a live run." >&2 + exit 1 + fi + # Live runs must establish each mode's runtime state; a missing prep hook would + # leave prep_mode a no-op and measure arbitrary state as publishable. Fail fast. + for mode in ${MODES}; do + case "${mode}" in + cold) prep_var="WORKCELL_STARTUP_COLD_PREP" ;; + cache-hit) prep_var="WORKCELL_STARTUP_CACHE_HIT_PREP" ;; + warm) prep_var="WORKCELL_STARTUP_WARM_PREP" ;; + esac + if [ -z "${!prep_var:-}" ]; then + echo "run-startup-bench: live run requires a prep hook for mode '${mode}':" \ + "set ${prep_var} to establish the ${mode} runtime state. Without it the" \ + "harness measures arbitrary state and the numbers are not publishable." >&2 + exit 1 + fi + done + # Parse CMD into argv honoring quoting (--workspace '/a b' stays one word). + eval "CMD_ARGV=( ${WORKCELL_STARTUP_CMD} )" +fi + +# Validate numeric controls (RUNS=0/non-integer would else misreport); RUNS last. +validate_int "WORKCELL_STARTUP_ITERATIONS" "${ITERATIONS}" 1 +validate_int "WORKCELL_STARTUP_WARMUP" "${WARMUP}" 0 +validate_int "WORKCELL_STARTUP_RUNS" "${RUNS}" 1 +validate_int "WORKCELL_STARTUP_STABILITY_PCT" "${STABILITY_PCT}" 0 + +# A publishable live capture needs >=2 runs: one run skips the gate yet exits 0. +if [ "${DRY_RUN}" -eq 0 ] && [ "${RUNS}" -lt 2 ]; then + echo "run-startup-bench: a live run requires WORKCELL_STARTUP_RUNS >= 2 for" \ + "cross-run stability evidence (got ${RUNS}); a single run is not publishable." >&2 + exit 1 +fi + +WORKDIR="$(mktemp -d "${TMPDIR:-/tmp}/startup-bench.XXXXXX")" +trap 'rm -rf "${WORKDIR}"' EXIT +REPORT="${WORKDIR}/report.md" + +# Extract an integer key=value field from a harness output line. +field() { + printf '%s\n' "$1" | sed -n "s/.*[[:space:]]$2=\([0-9]*\).*/\1/p" +} + +# Run the per-mode prep hook; its stdout goes to stderr so it can't pollute the +# report (exit status preserved). +prep_mode() { + case "$1" in + cold) eval "${WORKCELL_STARTUP_COLD_PREP:-:}" ;; + cache-hit) eval "${WORKCELL_STARTUP_CACHE_HIT_PREP:-:}" ;; + warm) eval "${WORKCELL_STARTUP_WARM_PREP:-:}" ;; + esac >&2 +} + +# Invoke the harness for warm (one prep/pass): dry-run passes canned samples, +# live times the parsed session command. +run_harness() { + # $1 mode, $2 run index (1-based) + if [ "${DRY_RUN}" -eq 1 ]; then + local gi=$(($2 - 1)) + [ "${gi}" -lt "${#SAMPLE_GROUPS[@]}" ] || gi=$((${#SAMPLE_GROUPS[@]} - 1)) + WORKCELL_STARTUP_SAMPLES_NS="${SAMPLE_GROUPS[gi]}" "${HARNESS}" "$1" "${ITERATIONS}" "${WARMUP}" + else + "${HARNESS}" "$1" "${ITERATIONS}" "${WARMUP}" -- "${CMD_ARGV[@]}" + fi +} + +# Measure cold/cache-hit with a genuine first start per sample: re-run the mode's +# prep hook before EACH sample (warmup 0), then aggregate via the harness stats. +measure_reprep() { + # $1 mode, $2 run index (1-based) + local mode="$1" + if [ "${DRY_RUN}" -eq 1 ]; then + # Dry-run: canned samples, no prep, no launch -- emit the group's stats. + local gi=$(($2 - 1)) + [ "${gi}" -lt "${#SAMPLE_GROUPS[@]}" ] || gi=$((${#SAMPLE_GROUPS[@]} - 1)) + WORKCELL_STARTUP_SAMPLES_NS="${SAMPLE_GROUPS[gi]}" "${HARNESS}" "${mode}" "${ITERATIONS}" 0 + return + fi + local samples=() one i=0 + while [ "${i}" -lt "${ITERATIONS}" ]; do + prep_mode "${mode}" + one="$("${HARNESS}" "${mode}" 1 0 -- "${CMD_ARGV[@]}")" + samples+=("$(field "${one}" min_ns)") + i=$((i + 1)) + done + WORKCELL_STARTUP_SAMPLES_NS="${samples[*]}" "${HARNESS}" "${mode}" "${#samples[@]}" 0 +} + +{ + echo "# session-start latency benchmark results" + echo + echo "- date (UTC): $(date -u '+%Y-%m-%dT%H:%M:%SZ')" + echo "- host: $(uname -srm)" + echo "- online CPUs: $(getconf _NPROCESSORS_ONLN 2>/dev/null || echo unknown)" + echo "- runtime: ${RUNTIME}" + echo "- iterations: ${ITERATIONS} (warmup ${WARMUP}; cold/cache-hit re-prep + warmup 0 per sample) x ${RUNS} run(s)" + echo "- stability threshold: ${STABILITY_PCT}% cross-run median spread" + echo +} >"${REPORT}" + +run_index=1 +while [ "${run_index}" -le "${RUNS}" ]; do + run_file="${WORKDIR}/run-${run_index}" + : >"${run_file}" + + { + echo "## Run ${run_index}" + echo + echo "| Mode | Median (ns) | p90 (ns) | Mean (ns) | Stddev (ns) | Min (ns) | Max (ns) | n |" + echo "|---|---|---|---|---|---|---|---|" + } >>"${REPORT}" + + for mode in ${MODES}; do + # cold and cache-hit re-prep per measured sample; only warm shares one prep + # for the whole pass (and dry-run runs no prep -- canned samples never launch). + if [ "${mode}" = "warm" ]; then + [ "${DRY_RUN}" -eq 1 ] || prep_mode "${mode}" + line="$(run_harness "${mode}" "${run_index}")" + else + line="$(measure_reprep "${mode}" "${run_index}")" + fi + med="$(field "${line}" median_ns)" + p90="$(field "${line}" p90_ns)" + mean="$(field "${line}" mean_ns)" + std="$(field "${line}" stddev_ns)" + lo="$(field "${line}" min_ns)" + hi="$(field "${line}" max_ns)" + n="$(field "${line}" n)" + + printf '| %s | %s | %s | %s | %s | %s | %s | %s |\n' \ + "${mode}" "${med}" "${p90}" "${mean}" "${std}" "${lo}" "${hi}" "${n}" >>"${REPORT}" + printf '%s %s\n' "${mode}" "${med}" >>"${run_file}" + done + + echo >>"${REPORT}" + run_index=$((run_index + 1)) +done + +# Cross-run stability gate: per mode, the median spread across runs as a percent +# of the smallest run's median; if any mode exceeds the threshold the driver fails. +GATE_STATUS="STABLE" +if [ "${RUNS}" -ge 2 ]; then + { + echo "## Cross-run stability (median)" + echo + echo "| Mode | Min median (ns) | Max median (ns) | Spread (ns) | Spread (%) | Verdict |" + echo "|---|---|---|---|---|---|" + } >>"${REPORT}" + + all_runs="${WORKDIR}/all-runs" + cat "${WORKDIR}"/run-* >"${all_runs}" + # Emit " " on stdout; the per-mode rows go to + # the report via stderr. A zero median in any run is a degenerate/broken + # measurement (a 0 ns session start is impossible), not a 0% spread -- flag it + # so the gate fails instead of reading STABLE. + gate_line="$(awk -v thr="${STABILITY_PCT}" ' + { m = $1; v = $2 + if (!(m in seen)) { order[++k] = m; seen[m] = 1; min[m] = v; max[m] = v } + if (v < min[m]) min[m] = v + if (v > max[m]) max[m] = v } + END { + worst = 0; degenerate = 0 + for (i = 1; i <= k; i++) { + m = order[i]; s = max[m] - min[m] + if (min[m] <= 0) { + degenerate = 1 + printf "| %s | %d | %d | %d | n/a | UNSTABLE |\n", m, min[m], max[m], s > "/dev/stderr" + } else { + p = s * 100.0 / min[m] + verdict = (p > thr) ? "UNSTABLE" : "STABLE" + if (p > worst) worst = p + printf "| %s | %d | %d | %d | %.1f | %s |\n", m, min[m], max[m], s, p, verdict > "/dev/stderr" + } + } + printf "%.1f %d", worst, degenerate + } + ' "${all_runs}" 2>>"${REPORT}")" + echo >>"${REPORT}" + worst="${gate_line%% *}" + degenerate="${gate_line##* }" + + if [ "${degenerate}" -eq 1 ]; then + GATE_STATUS="UNSTABLE" + elif awk -v w="${worst}" -v thr="${STABILITY_PCT}" 'BEGIN { exit (w > thr) ? 0 : 1 }'; then + GATE_STATUS="UNSTABLE" + fi + { + if [ "${GATE_STATUS}" = "STABLE" ]; then + echo "Stability gate: STABLE (max cross-run median spread ${worst}% <= ${STABILITY_PCT}%)." + elif [ "${degenerate}" -eq 1 ]; then + echo "Stability gate: UNSTABLE (a mode reported a zero median across runs" \ + "-- degenerate measurement, not a fast start)." + else + echo "Stability gate: UNSTABLE (max cross-run median spread ${worst}% > ${STABILITY_PCT}%)." + fi + echo + } >>"${REPORT}" +fi + +cat "${REPORT}" +if [ -n "${OUTPUT_PATH}" ]; then + cp "${REPORT}" "${OUTPUT_PATH}" + echo "run-startup-bench: report written to ${OUTPUT_PATH}" >&2 +fi + +if [ "${GATE_STATUS}" = "UNSTABLE" ]; then + if [ "${degenerate:-0}" -eq 1 ]; then + echo "run-startup-bench: cross-run stability gate FAILED (degenerate zero median)" >&2 + else + echo "run-startup-bench: cross-run stability gate FAILED (spread ${worst}% > ${STABILITY_PCT}%)" >&2 + fi + exit 2 +fi diff --git a/scripts/bench/startup-bench.sh b/scripts/bench/startup-bench.sh new file mode 100755 index 00000000..32dc791c --- /dev/null +++ b/scripts/bench/startup-bench.sh @@ -0,0 +1,140 @@ +#!/usr/bin/env -S BASH_ENV= ENV= bash +# +# startup-bench.sh -- session-start latency microbenchmark harness (C2). +# +# Times session-start latency for one mode (cold / warm / cache-hit) over an +# N-sample median + p90. A small, dependency-free core; run-startup-bench.sh +# orchestrates the passes and stability gate. See docs/session-startup-benchmarks.md. +# Stats conventions match the C5 exec-guard harness exactly: +# median = sorted[floor(n/2)] (0-based) +# p90 = sorted[floor(n*9/10)] (0-based, clamped to n-1) +# mean = sum/n ; stddev = sqrt(sumsq/n - mean^2) (population) +# +# Usage: startup-bench.sh [--] [target-cmd ...] +# mode cold|warm|cache-hit; iterations/warmup = measured/discarded launches. +# Dry-run: if WORKCELL_STARTUP_SAMPLES_NS is a whitespace-separated list of +# non-negative integers, those are the samples verbatim and NO command launches +# (how the driver/unit tests exercise the stats core with no runtime). +# +# Output (one line, ns): mode=<> n=<> mean_ns=<> median_ns=<> p90_ns=<> \ +# stddev_ns=<> min_ns=<> max_ns=<>. A failed launch aborts non-zero with no +# numbers, so a broken session start can never masquerade as a measurement. +set -euo pipefail + +die() { + echo "startup-bench: $*" >&2 + exit 1 +} + +[ "$#" -ge 3 ] || die "usage: startup-bench.sh [--] [target-cmd ...]" + +MODE="$1" +ITERATIONS="$2" +WARMUP="$3" +shift 3 +if [ "${1:-}" = "--" ]; then + shift +fi + +case "${MODE}" in + cold | warm | cache-hit) ;; + *) die "unknown mode '${MODE}' (want: cold warm cache-hit)" ;; +esac + +case "${ITERATIONS}" in + '' | *[!0-9]*) die "iterations must be a non-negative integer, got '${ITERATIONS}'" ;; +esac +case "${WARMUP}" in + '' | *[!0-9]*) die "warmup must be a non-negative integer, got '${WARMUP}'" ;; +esac + +# Reduce sorted newline-separated integer samples on stdin to the key=value +# stats line (n .. max), using the exact C5 harness conventions. Kept pure so a +# unit test can pin the median/p90/stddev math against a known sample set. +stats() { + sort -n | awk ' + { s[NR] = $1; sum += $1; sumsq += $1 * $1 } + END { + n = NR + if (n == 0) { print "startup-bench: no samples" > "/dev/stderr"; exit 1 } + mean = sum / n + var = sumsq / n - mean * mean + if (var < 0) { var = 0 } + stddev = sqrt(var) + median = s[int(n / 2) + 1] + p90i = int(n * 9 / 10) + if (p90i >= n) { p90i = n - 1 } + p90 = s[p90i + 1] + printf "n=%d mean_ns=%.0f median_ns=%d p90_ns=%d stddev_ns=%.0f min_ns=%d max_ns=%d\n", + n, mean, median, p90, stddev, s[1], s[n] + }' +} + +emit() { + # $@ = integer samples (one per argument) + local line + line="$(printf '%s\n' "$@" | stats)" + printf 'mode=%s %s\n' "${MODE}" "${line}" +} + +# --- Deterministic path: canned samples, no launch ------------------------- +if [ -n "${WORKCELL_STARTUP_SAMPLES_NS:-}" ]; then + read -ra samples <<<"${WORKCELL_STARTUP_SAMPLES_NS}" + [ "${#samples[@]}" -ge 1 ] || die "WORKCELL_STARTUP_SAMPLES_NS is empty" + for sample in "${samples[@]}"; do + case "${sample}" in + '' | *[!0-9]*) die "WORKCELL_STARTUP_SAMPLES_NS holds a non-integer sample '${sample}'" ;; + esac + done + emit "${samples[@]}" + exit 0 +fi + +# --- Live path: launch the target and time each start ---------------------- +[ "$#" -ge 1 ] || die "no target command given (and WORKCELL_STARTUP_SAMPLES_NS unset)" +[ "${ITERATIONS}" -ge 1 ] || die "iterations must be >= 1 for a live measurement" + +# Pick a MONOTONIC nanosecond clock (mirrors the C5 exec-guard CLOCK_MONOTONIC). +# Wall-clock (`date +%s%N`) is unusable: an NTP step or sleep/wake mid-launch +# could make end-start negative or absorb the step. perl/python3 expose +# CLOCK_MONOTONIC; plain `date` does not, so it is not a fallback. +CLOCK="" +if command -v perl >/dev/null 2>&1 && + perl -MTime::HiRes=clock_gettime,CLOCK_MONOTONIC -e1 >/dev/null 2>&1; then + CLOCK="perl" +elif command -v python3 >/dev/null 2>&1; then + CLOCK="python3" +else + die "no monotonic nanosecond clock (need perl Time::HiRes or python3)" +fi + +# Time the WHOLE loop inside ONE long-lived process (like C5's in-process loop) so +# no per-sample interpreter launch sits in a measured interval -- only the target's +# fork/exec. Args (warmup iterations -- target...); target is exec'd (no shell) so +# quoting survives, output discarded, non-zero launch aborts. +# shellcheck disable=SC2016 # perl/python program text: $vars are NOT shell expansions +timing_perl='use strict; use warnings; use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC); +my $warm = shift @ARGV; my $iter = shift @ARGV; +sub launch { my $p = fork; die "fork failed\n" unless defined $p; + if (!$p) { open STDOUT, ">", "/dev/null"; open STDERR, ">", "/dev/null"; exec { $ARGV[0] } @ARGV; exit 127 } + waitpid $p, 0; die "target launch failed\n" if $? } +launch() for (1 .. $warm); +for (1 .. $iter) { my $t = clock_gettime(CLOCK_MONOTONIC); launch(); printf "%.0f\n", (clock_gettime(CLOCK_MONOTONIC) - $t) * 1e9 }' +# shellcheck disable=SC2016 # python program text: $-free, quotes intentional +timing_python='import sys, time, subprocess +warm = int(sys.argv[1]); iters = int(sys.argv[2]); argv = sys.argv[3:] +def launch(): + if subprocess.call(argv, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) != 0: + sys.exit("target launch failed") +for _ in range(warm): + launch() +for _ in range(iters): + t = time.monotonic_ns(); launch(); print(time.monotonic_ns() - t)' + +case "${CLOCK}" in + perl) raw="$(perl -e "${timing_perl}" -- "${WARMUP}" "${ITERATIONS}" "$@")" || die "target launch failed during timing" ;; + python3) raw="$(python3 -c "${timing_python}" "${WARMUP}" "${ITERATIONS}" "$@")" || die "target launch failed during timing" ;; +esac + +[ -n "${raw}" ] || die "no samples produced" +printf 'mode=%s %s\n' "${MODE}" "$(printf '%s\n' "${raw}" | stats)" diff --git a/scripts/dev-quick-check.sh b/scripts/dev-quick-check.sh index 0c51a1df..8243da42 100755 --- a/scripts/dev-quick-check.sh +++ b/scripts/dev-quick-check.sh @@ -26,6 +26,8 @@ require_tool rustfmt require_cargo_subcommand clippy shell_files=( + "${ROOT_DIR}/scripts/bench/run-startup-bench.sh" + "${ROOT_DIR}/scripts/bench/startup-bench.sh" "${ROOT_DIR}/scripts/bootstrap-dev.sh" "${ROOT_DIR}/scripts/check-dead-code.sh" "${ROOT_DIR}/scripts/check-public-repo-hygiene.sh" diff --git a/scripts/validate-repo.sh b/scripts/validate-repo.sh index b3f091eb..db4f4fbf 100755 --- a/scripts/validate-repo.sh +++ b/scripts/validate-repo.sh @@ -132,6 +132,8 @@ shell_files=( "${ROOT_DIR}/scripts/build-and-test.sh" "${ROOT_DIR}/scripts/ci-plan.sh" "${ROOT_DIR}/scripts/bench/run-exec-guard-bench.sh" + "${ROOT_DIR}/scripts/bench/run-startup-bench.sh" + "${ROOT_DIR}/scripts/bench/startup-bench.sh" "${ROOT_DIR}/scripts/workcell" "${ROOT_DIR}/scripts/check-workflows.sh" "${ROOT_DIR}/scripts/ci/build-validator-image.sh"