Skip to content

benchmarking.md

Latest commit

 

History

History
76 lines (58 loc) · 3.19 KB

benchmarking.md

File metadata and controls

76 lines (58 loc) · 3.19 KB

Benchmarking

bench-incremental measures rebuild time for go2nix's default-mode builder after touching a single file at different depths in the dependency graph. Use it to see how contentAddressed and the iface early-cutoff behave on a representative project before adopting them.

Running it

nix run github:numtide/go2nix#bench-incremental -- -fixture light

(or nix run .#bench-incremental -- ... from a checkout).

The harness spins up a rooted local store (NIX_REMOTE=local?root=$TMPDIR/...), loads the Nix plugin via --option plugin-files, does a full warm build, then repeatedly edits one file and times the rebuild. It needs network access (the warm build fetches modules from substituters), so it cannot run inside a nix build sandbox — nix flake check only verifies that the binary links.

Flags

Flag Default Meaning
-runs N 3 Runs per scenario; the median is reported
-scenario S all One of no_change, leaf, mid, deep, all
-touch-mode M private private edits an unexported symbol; exported edits an exported one
-tools L nix-nocgo,nix-ca-nocgo Comma-separated tool variants: nix, nix-ca, nix-nocgo, nix-ca-nocgo
-fixture F light light or torture (see below)
-json PATH — Write raw results as JSON
-assert-cascade N — Fail (non-zero exit) if any tool builds more than N derivations on a touch scenario

The nix-ca* variants set contentAddressed = true; the *-nocgo variants set CGO_ENABLED = 0. Comparing nix-nocgo against nix-ca-nocgo with -touch-mode private shows the iface early-cutoff in action.

Fixtures and scenarios

Two synthetic projects under tests/fixtures/:

Fixture Shape leaf edits mid edits deep edits
light small app, a few internal packages app/cmd/app/main.go internal/handler/handler.go internal/core/core.go
torture large app, hundreds of modules app-full/cmd/app-full/main.go internal/aws/aws.go internal/common/common.go

leaf touches the entrypoint (no reverse dependents — only the link rebuilds). mid touches a package roughly halfway up the graph with a moderate reverse-dependency cone. deep touches a package near the bottom of the graph that fans out to most of the app. no_change measures the eval + no-op-build floor.

Using -assert-cascade in CI

nix run .#bench-incremental -- \
  -fixture light -scenario mid -touch-mode private \
  -tools nix-ca-nocgo -assert-cascade 5

This fails if a private-symbol edit to a mid-graph package causes more than five derivations to rebuild — a regression check for the early-cutoff machinery.

Other benchmarks

The flake also exposes coarser-grained harnesses:

  • benchmark-build — wall-clock time for a full cold build of a fixture.
  • benchmark-eval — wall-clock time for a pure nix eval of the package graph (plugin + instantiation cost).
  • benchmark-build-cross-app-isolation — verifies that two apps sharing third-party packages reuse each other's per-package store paths.