nix-plugin: persistent DAG cache for resolveGoPackages

[aldoborrero]

Problem

builtins.resolveGoPackages runs go list -json -deps at Nix eval time. That command must read source files from every module in the transitive closure to parse their import blocks, so the full GOMODCACHE has to be present on disk before the builtin can produce output — even when nothing has changed since the last eval. For large module graphs this is the dominant cost of every nix eval.

Approach

Cache the full output JSON under $XDG_CACHE_HOME/go2nix/resolve/<sha256>.json (sibling of the existing nar_cache impurity). On a hit we return the cached string before resolve_packages runs, so go list -deps — and therefore GOMODCACHE — is never touched.

Why the cache key is correct

go list reads each .go file only through the import block (go/build/read.go readGoInfo, parser.ImportsOnly); function bodies are never read. So a cheap local-only probe captures exactly the local state that can change the third-party closure:

go list -e -mod=mod -json=ImportPath,Imports,GoFiles,CgoFiles,SFiles,CFiles,CXXFiles,FFiles,HFiles,SysoFiles,EmbedPatterns ./...

This runs with an empty GOMODCACHE and GOPROXY=off in milliseconds — Imports is populated because it comes from header parsing, not resolution. (Do not use -find: it sets IgnoreImports and clears Imports.)

The key is sha256 of:

Input	Why
SCHEMA_VERSION constant	Self-invalidate on output-shape changes
go.sum + go.mod bytes	Module graph, replace directives
Local probe output (sorted)	Local imports + file lists
GOOS/GOARCH/CGO_ENABLED	File selection
sorted tags, sorted subPackages, modRoot	Scope
doCheck, resolveHashes	Output shape
go version output	Toolchain identity → ReleaseTags + stdlib

Invalidation

Change	Hit?
Local .go body edited (no import change)	✅ hit
Local import line added/removed	❌ miss
Local file added/renamed	❌ miss
go.sum / go.mod changed	❌ miss
Different platform/tags	❌ miss
Go toolchain bump	❌ miss

Semantics

• Best-effort: any error in probe/key/read/write logs to stderr and falls through to today's path. Never fails the builtin.
• Opt-out: GO2NIX_RESOLVE_CACHE=0 disables it entirely.
• Atomic writes: temp+rename, same as nar_cache.rs.

Caller-side note

nix/dag/default.nix:296-315 passes no modcache argument to the builtin — the plugin reads ambient GOMODCACHE from the evaluator's process. So a cache hit genuinely avoids GOMODCACHE. To realise the full win, whatever wrapper exports GOMODCACHE before nix eval (e.g. benchmark-eval/default.nix:106-107) must become lazy — populate only on miss. That's outside this PR.

Test plan

• cargo build && cargo test — 44/44 (38 pre-existing + 6 new: key determinism, varies-with-tags/subpackages, sort-stable, no-adjacent-field-collision, read/write round-trip)
• cargo clippy — only the 2 pre-existing warnings (nar.rs:57, resolve.rs:232)
• rustfmt-clean on touched files
• End-to-end via vanilla nix + --option plugin-files: cold call populates $XDG_CACHE_HOME/go2nix/resolve/<hash>.json; warm call with GOMODCACHE=/definitely/missing returns identical JSON
• strace: warm call execs only the probe + go version; no -deps invocation. Cold call execs -deps.
• Body-edit → hit (same key); add import "fmt" → miss (new key, -deps runs)
• GO2NIX_RESOLVE_CACHE=0 → eval succeeds, no cache file written
• XDG_CACHE_HOME=/proc/self/nonwritable → write error swallowed, eval succeeds
• doCheck=true vs false → distinct keys

Related

• golist: pass -pgo=off -mod=readonly to go list -deps #47 adds -pgo=off -mod=readonly to the go list -deps invocations — independent quick win for the cold/miss path.

Pre-building the cache (sandboxed environments)

For hermetic/sandboxed evals with no writable cache dir, point GO2NIX_RESOLVE_CACHE_DIR at a pre-built read-only directory — checked before XDG, never written to.

To produce that directory as a Nix derivation, run one eval with XDG_CACHE_HOME=$out:

resolveCache = runCommand "go2nix-resolve-cache" {
  nativeBuildInputs = [ nixWithPlugin go ];
  inherit src;
  # the gomodcache FOD is an input here — paid once, then this drv is
  # substitutable for all downstream evals that share the same key inputs
  GOMODCACHE = goModules;
} ''
  XDG_CACHE_HOME=$out nix eval --impure --expr \
    'builtins.resolveGoPackages { go="${go}/bin/go"; src='"$src"'; ... }'
'';

Downstream evals then export GO2NIX_RESOLVE_CACHE_DIR=${resolveCache}/go2nix/resolve and never realise GOMODCACHE on a hit.