Follow-up to the benchmark suite. The per-PR suite tracks per-phase time + logical memory (CodSpeed) and the end-to-end pipeline (build→export logical peak). What it deliberately does not track is physical RSS — the "will this OOM / fit in N GB" number. This issue captures the design for adding it as a separate nightly tier, decided after prototyping it inside the suite and (correctly) ripping it back out.
Why a separate tier, not the per-PR suite
- Coupling: it needs an isolated-RSS tool (pytest-benchmem / getrusage); the per-PR suite must stay coupled to nothing but pytest-benchmark + CodSpeed.
- Coarse + slow: RSS is ~2% CV and needs a fresh process per measurement — wrong for the fast path.
- Floor-dominated at PR sizes: RSS only carries signal when the workload dwarfs the ~150 MB interpreter/import floor. At the per-PR sweep sizes it's mostly floor.
- It can't be derived from CodSpeed's logical memory — no fixed ratio (measured:
pypsa_scigrid rss≫peak, build-resident; kvl_cycles peak≫rss, transient). So it must be measured directly.
Design principles
- Nightly, separate job (not per-PR), outside
benchmarks/.
- Large sizes only — pushed well above the per-PR sweep, capped under the runner's RAM.
- Whole-job, isolated — build→export in one fresh process; read
getrusage ru_maxrss (or cgroup memory.peak for solver-inclusive). A phase's RSS can't be isolated — capacity is whole-job.
- Generous gate on a stable stat — min-of-N, thresholds ≥5–10%.
- Picklable/rebuild-in-child recipes (
partial(build_and_export, spec, n)), so the child rebuilds cold rather than receiving a pre-built model.
Two flavors (pick by goal)
- A — OOM safety net (simplest): nightly CI runs a few large cases under a hard cap and asserts they survive (
systemd-run --scope -p MemoryMax=… / /usr/bin/time -v + threshold). Binary, deployment-truthful, zero tracking infra.
- B — RSS trend KPI (fits a sparsity fork): standalone
getrusage worker over large whole-job cases → emit Bencher Metric Format → bencher run. Hosted chart of RSS trending down + threshold alerts. (Bencher is metric-agnostic; the pytest adapter doesn't capture RSS, so use the json/BMF adapter.)
Prototype reference
The capacity.py + capacity_rss.py pattern was built and removed during the suite work — the right idea in the wrong place (kept trying to fold it into the per-PR suite). Resurrect it as a standalone nightly job: top-level picklable recipes, a few large cases (expression_arithmetic@1500 ≈ 2.3 GB, sparse_network@1000 ≈ 1.9 GB, pypsa_scigrid@50 ≈ 1.3 GB — the heaviest, real model), min-of-3, ~50 s total.
Motivating findings (from the suite work)
pypsa_scigrid is the memory ceiling (1.3 GB rss, build-dominated) — a single-spec set would miss it; the capacity cases must be model-diverse.- Logical and physical diverge with no fixed ratio — RSS is its own metric.
- The investigation also surfaced a real
slice_size 3.4× write-memory lever in the LP writer (separate optimization).
Open decisions
- A (cap-assert) vs B (Bencher trend) — A first, graduate to B if the trend gets watched.
- Which large cases + sizes (under the target runner's RAM).
- Solver-inclusive (cgroup
memory.peak, Linux-only) — only if gating solver memory becomes a need.
(AI-assisted; the numbers are real measurements from the suite work.)
Follow-up to the benchmark suite. The per-PR suite tracks per-phase time + logical memory (CodSpeed) and the end-to-end pipeline (build→export logical peak). What it deliberately does not track is physical RSS — the "will this OOM / fit in N GB" number. This issue captures the design for adding it as a separate nightly tier, decided after prototyping it inside the suite and (correctly) ripping it back out.
Why a separate tier, not the per-PR suite
pypsa_scigridrss≫peak, build-resident;kvl_cyclespeak≫rss, transient). So it must be measured directly.Design principles
benchmarks/.getrusageru_maxrss(or cgroupmemory.peakfor solver-inclusive). A phase's RSS can't be isolated — capacity is whole-job.partial(build_and_export, spec, n)), so the child rebuilds cold rather than receiving a pre-built model.Two flavors (pick by goal)
systemd-run --scope -p MemoryMax=…//usr/bin/time -v+ threshold). Binary, deployment-truthful, zero tracking infra.getrusageworker over large whole-job cases → emit Bencher Metric Format →bencher run. Hosted chart of RSS trending down + threshold alerts. (Bencher is metric-agnostic; the pytest adapter doesn't capture RSS, so use the json/BMF adapter.)Prototype reference
The
capacity.py+capacity_rss.pypattern was built and removed during the suite work — the right idea in the wrong place (kept trying to fold it into the per-PR suite). Resurrect it as a standalone nightly job: top-level picklable recipes, a few large cases (expression_arithmetic@1500≈ 2.3 GB,sparse_network@1000≈ 1.9 GB,pypsa_scigrid@50≈ 1.3 GB — the heaviest, real model),min-of-3, ~50 s total.Motivating findings (from the suite work)
pypsa_scigridis the memory ceiling (1.3 GB rss, build-dominated) — a single-spec set would miss it; the capacity cases must be model-diverse.slice_size3.4× write-memory lever in the LP writer (separate optimization).Open decisions
memory.peak, Linux-only) — only if gating solver memory becomes a need.(AI-assisted; the numbers are real measurements from the suite work.)