feat-stress(disk): host disk-space admission guard + boxlite gc for orphaned image disks#618
Draft
G4614 wants to merge 6 commits into
Draft
feat-stress(disk): host disk-space admission guard + boxlite gc for orphaned image disks#618G4614 wants to merge 6 commits into
G4614 wants to merge 6 commits into
Conversation
A box can grow the host filesystem via its COW overlay and (unbounded) virtio-fs volume writes — a guest sees the entire host filesystem's free space with no quota. There was no precheck, so a box could be started into an already-critically-full disk where image extraction / qcow2 growth fails mid-operation. Add a DiskSpaceTask that runs first in the start pipeline (Configured and Stopped/Failed): - statvfs the boxlite home filesystem (f_bavail — space usable by the box) - HARD floor (<1 GiB free): refuse to start with ResourceExhausted - SOFT floor (<5 GiB or <10% free): warn but proceed (percentage is warn-only so a large disk at 9% free is never wrongly rejected) Failure to read free space degrades to a warning, not a block. This is admission control only; it does not bound a running box's writes (volumes have no quota — that needs filesystem project quotas). classify() is pure and unit-tested; reject path verified end-to-end to abort startup without spawning a shim. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Exercises DiskSpaceTask::run() through a real InitPipelineContext (runtime + box config), asserting its Result matches classify() on the test home's actual free space. Asserting against the real verdict (not a fixed value) keeps it non-flaky regardless of the host's disk size. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pulled images materialize a merged rootfs at images/disk-images/<digest>.ext4 that box COW overlays back onto (absolute path baked into the qcow2 header). These accumulate forever: removing a box never deletes the disk-image it was built from, so built rootfs piles up (was ~900MB of a 2GB image cache here). Add a GC that removes disk-images no box overlay backs onto: - GarbageCollector (RuntimeImpl::collect_garbage) builds the pinned set by walking every boxes/*/disks/*.qcow2 backing chain, then deletes disk-images absent from it. A disk-image is removed only when provably unreferenced — never on an age/size guess — so a backed (even stopped) box is never broken. - `boxlite gc [--dry-run]` exposes it via the RuntimeBackend facade (local-only; REST returns Unsupported). Scope is deliberately narrow: orphaned box dirs and unreferenced bases are already collected (cleanup_orphaned_directories on startup; remove_box → try_gc_base cascade). LRU of the re-pullable build cache (layers/extracted) needs blob-dedup in the image store and is a separate change. Verified: dry-run + real run reclaimed 1.41 GiB of orphaned disk-images while keeping the one a live box overlay backs onto. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the disk-space guard sees Warn/Reject at box start, run the image-cache GC and re-evaluate before deciding admission — so a box can start if reclaiming orphaned disk-images frees enough. Serialization without a hot-path lock: box starts hold only per-box locks and materialize disk-images concurrently, so a global GC lock would have to gate the materialization hot path (serializing all concurrent starts). Instead the sweep skips disk-images modified within a 10-minute grace window. The build→overlay window in a start is sub-second, so the grace gives a ~1000x margin against deleting a disk-image a concurrent start just built — lock-free, no hot-path contention, and it also makes the manual `boxlite gc` race-safe. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…otectors GC sweep: - gc_at_scale_reclaims_only_aged_orphans — 150 aged orphans + 40 fresh + 20 box-backed (all aged, so only the pin protects them): exactly the aged orphans are reclaimed, byte-exact, and a second pass is idempotent. - gc_under_concurrent_starts_never_deletes_live_or_fresh_images — hammers collect_garbage in a loop while a thread floods box starts (fresh image + backing overlay). Proves the lock-free claim: no just-built or pinned image is ever deleted, while pre-seeded aged orphans are still reaped. A pass cap keeps a future grace/pin regression failing fast instead of timing out. Admission classify(): - classify_severity_is_monotonic_and_hits_thresholds — sweeps the whole free range asserting severity never gets stricter as free rises, and pins the exact hard/soft transitions. - real_disk_pressure_crosses_to_reject — gated on BOXLITE_DISKTEST_HOME (a small dedicated FS, size-railed <8 GiB); fills it through the real statvfs path until free crosses the hard floor and asserts Reject, then cleans up. Skips when unset so CI/`make test` never fill a disk. Verified against a loop-mounted 1.5 GiB ext4. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
G4614
changed the title
The orphan sweep walked every regular file in images/disk-images and reclaimed any unpinned one past the grace window, regardless of extension — so a stray partial/temp/foreign file aging out there would be deleted. Filter to *.ext4 (matching the documented scope) and add a regression test: an aged non-.ext4 orphan must be kept and excluded from the byte accounting while an aged .ext4 orphan is reclaimed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Guard the host disk so a box can't silently fill it: a
DiskSpaceTaskrefuses to start below ~1 GiB free (warns when low), andboxlite gcreclaims orphaned image disk-images that no box overlay's backing chain points at (with a 10-min mtime grace).Test plan
Unit —
classify()thresholds (util::disk_space): reject<1 GiB; warn<5 GiB/<10%; ok otherwise.Integration — admission task (
litebox::init::tasks::disk_space):run_matches_classify_verdict_for_real_homedrivesDiskSpaceTask::runagainst a real context and asserts itsResultmatchesclassify()on the home's actual free space.GC — orphan sweep (
runtime::gc):sweeps_orphan_disk_images_only— backed / aged-orphan / fresh-orphan; only the aged orphan is reclaimed.Added hardening (this branch):
gc_at_scale_reclaims_only_aged_orphans— 150 aged orphans + 40 fresh + 20 box-backed (all aged, so only the pin protects them): exactly the aged orphans are reclaimed, byte-exact, idempotent on a second pass.gc_under_concurrent_starts_never_deletes_live_or_fresh_images— hammerscollect_garbagein a loop while a thread floods box starts (fresh image + backing overlay); proves the lock-free claim. Mutation-verified (grace=0 → fails).classify_severity_is_monotonic_and_hits_thresholds— severity never gets stricter as free rises; pins exact hard/soft transitions.real_disk_pressure_crosses_to_reject— gated onBOXLITE_DISKTEST_HOME(small dedicated FS), fills it through the realstatvfspath until free crosses the hard floor and asserts Reject; verified against a loop-mounted 1.5 GiB ext4.<1 GiBat box startResourceExhausted), no shim spawned<5 GiB/<10%boxlite gcover orphaned disk-images<10 minago