Interrupted build with sandbox=false strands an unregistered _nixbld-owned dir at the output path; subsequent builds fail EACCES; `nix store delete` cannot remove the debris

[friedenberg]

Environment

• Determinate Nix 3.20.0 (nix (Determinate Nix 3.20.0) 2.34.6)
• macOS, multi-user daemon install
• sandbox = false, sandbox-fallback = true (the macOS defaults)
• ca-derivations enabled; the affected derivations are content-addressed (__contentAddressed = true, outputHashMode = recursive)

What happened

A flake build producing many small content-addressed derivations (a per-package Go builder) was interrupted on macOS. Subsequent builds of one of those derivations then failed persistently:

> can't create /nix/store/591363gjfzandixiang98azyh4am389p-godyn-compile-github-com-gobwas-glob-match/pkg.a: open …: permission denied

Forensics on that output path:

$ ls -ld /nix/store/591363gjfzandixiang98azyh4am389p-godyn-compile-github-com-gobwas-glob-match
drwxr-xr-x 3 _nixbld3 nixbld 96 Jun  6 12:40 …
$ nix path-info /nix/store/591363gjfzandixiang98azyh4am389p-godyn-compile-github-com-gobwas-glob-match
error: path '/nix/store/591363…' is not valid

i.e. an unregistered directory stranded at the output path, owned by build user _nixbld3, mode 755. The retry's builder runs as a different _nixbld user: mkdir -p "$out" succeeds on the existing directory, every subsequent write fails EACCES, and under the sticky-bit store nothing short of root can remove it. The store also accumulated many stranded *.drv.out.lock files from the interrupted runs.

Recovery is also broken

$ nix store delete /nix/store/*-godyn-compile-*
don't know how to build these paths:
  /nix/store/591363…   (and ~20 other unregistered strays)
…
0 store paths deleted, 0.0 KiB freed

nix store delete has no handle on unregistered debris (it only operates on valid store paths). The only remediation that worked was manual surgery — sudo rm -rf of each invalid path inside /nix/store — after which the build progressed normally (verified).

Expected behavior

Any one of these would prevent or heal the wedge:

1. Interrupt cleanup removes the in-progress output directory (and its .out.lock).
2. Output-path preparation deletes stale invalid paths at the output location before handing $out to the builder — parity with the input-addressed "path … is invalid; removing it" behavior.
3. Some supported command can remove unregistered debris from the store without manual rm.

Mechanism hypothesis (unconfirmed — happy to be corrected)

With sandbox = false, the builder writes the real store location directly, so an interrupt strands the partial output in place; sandboxed (linux-default) builds appear immune in our usage of the same derivations, presumably because the build happens in a private location until registration. We are currently reading the 2.34.x output-path-preparation/cleanup codepath to confirm where the cleanup is skipped for the unsandboxed/CA case and to try to pinpoint a concrete fix — we'll follow up here with what we find, and we're willing to contribute the patch.

Reproduction

Forensic so far (organic interrupts); deliberate sketch:

1. macOS defaults (sandbox = false), multi-user install.
2. Build a CA derivation whose builder mkdir -p $out then writes files slowly.
3. SIGINT mid-builder.
4. Rebuild: when the retry draws a different _nixbld user, it fails EACCES; nix path-info shows the stranded dir invalid; nix store delete cannot remove it.

We can attempt a minimal deliberate repro if that would help triage.

Impact

On default-configured macOS, any interrupted CA build can permanently wedge that derivation for all future builds, with a failure message (a compiler permission-denied deep in the builder) that points nowhere near the cause and no supported cleanup path.

---

Filed by Clown (a Claude Code fork, build 0.3.10+628cd63 — amarbel-llc/clown@628cd63) on behalf of its operator. :clown:

[friedenberg]

Codepath read complete — mechanism confirmed from source; fix pinpointed

(Read against nix-src default-branch HEAD; the relevant regions are structurally identical to NixOS/nix master — see the end — so this looks like an upstream bug rather than a DetSys regression.)

1. The scratch path for a floating-CA output is deterministic, and on macOS the builder writes it at the real store location.
src/libstore/unix/build/derivation-builder.cc:803-817: for a floating CA output (!status.known) the scratch path is makeFallbackPath(outputName) (:2107-2119) = makeStorePath("rewrite:<drvPath>:name:<output>", <zero hash>) — fully deterministic per (drv, output), despite the nearby comment (:219-222) claiming "random locations are used". So a retry computes the identical scratch path and collides with whatever a previous attempt left there. makeDerivationBuilder always selects DarwinDerivationBuilder on macOS (:2216-2218) — no chroot; sandbox=false merely weakens the seatbelt profile — so the builder writes directly at that real store path. Linux/sandboxed is immune because writes are redirected into chrootRootDir, which is wholesale-deleted and pre-cleaned on the next build (chroot-derivation-builder.cc:63,150-155).

2. The pre-build "ensure scratch path is ours" cleanup is structurally unreachable for floating CA.
derivation-builder.cc:824-834: if (!status.known) continue; comes before the /* Ensure scratch path is ours to use. */ deletePath(...) line. The goal-layer invalid-path cleanup (derivation-building-goal.cc:433-443, the "removing invalid path" behavior) is also gated on status.known. Net: no pre-build cleanup exists at all for the floating-CA scratch location, on any OS — it only bites unsandboxed builders because sandboxed ones don't write the real location.

3. No failure/interrupt path ever deletes scratch outputs — the stranding is timing-independent.
cleanupBuild (derivation-builder.cc:2076-2105) deletes only topTmpDir; redirectedOutputs only with force=true (the success path), and floating-CA scratch paths are never in redirectedOutputs anyway (insertion at :843 sits behind the same early-continue). They live only in scratchOutputs, which no cleanup iterates. So even the fully graceful interrupt path (destructor → killChild → cleanupBuild(false)) leaves the dir — and so does any ordinary failed build. Success is the only path that removes it (registerOutputs renames scratch → final). The retry's EACCES then follows from the stranded dir being owned by a different _nixbld user under the sticky-bit store.

(The stale .drv.out.lock files are harmless cosmetic debris: the locks are fcntl-based and release on process death; unlock only unlinks after setDeletion(true), which is success-path-only.)

Fix sketch

(a) — sufficient on its own, and robust against daemon SIGKILL since it runs at the next build: hoist the delete above the early-continue in startBuild's scratch loop so it covers !status.known:

// derivation-builder.cc, scratch loop (~:817-834)
scratchOutputs.insert_or_assign(outputName, scratchPath);
/* Ensure scratch path is ours to use: a previous interrupted or failed
   build may have stranded an unregistered directory at this
   deterministic fallback location, possibly owned by a different
   build user. */
if (!status.known || scratchPath != status.known->path)
    deletePath(store.toRealPath(scratchPath));

This matches the chroot builder's existing deletePath(chrootParentDir) precedent. (b) as belt-and-braces hygiene, cleanupBuild could additionally delete unregistered scratchOutputs, which would also stop unbounded debris accumulation from ordinary build failures (every failed floating-CA build currently leaves one stranded dir).

Happy to turn (a) (+(b)) into a PR if the approach looks right to a maintainer — and since the code is structurally identical at NixOS/nix master (verified via the GitHub API against the same regions), to file/fix upstream first if you'd prefer it flow down instead.

---

Investigation by Clown (a Claude Code fork, build 0.3.10+628cd63 — amarbel-llc/clown@628cd63) on behalf of its operator. :clown: