Skip to content

fix(boot-profile): close foyer clean cache before TempDir drop (stops tmpfs fd leak)#85

Merged
jaredLunde merged 1 commit into
mainfrom
fix/boot-profiler-foyer-fd-leak
Jun 27, 2026
Merged

fix(boot-profile): close foyer clean cache before TempDir drop (stops tmpfs fd leak)#85
jaredLunde merged 1 commit into
mainfrom
fix/boot-profiler-foyer-fd-leak

Conversation

@jaredLunde

Copy link
Copy Markdown
Contributor

Problem

The boot-set profiler (oci/boot_capture_served.rs::capture_once) opens a per-run FoyerBlockCache in a tempfile::TempDir and never closes it — only Arc-drop at function return. foyer's SSD-tier region files stay open until the storage engine is closed, so the TempDir's remove_dir_all runs while the fds are still held, orphaning them as deleted-but-held files that aren't reclaimed until the process exits.

instd drives a profile run per rootfs compose/create, so a long-lived daemon leaks these unbounded into $TMPDIR. On hosts where /tmp is a tmpfs this fills RAM and ENOSPCs every other process on the node.

Observed in the field (glidefs 1.0.1):

glidefs <pid> root REG 0,37 16777216 0 ... /tmp/.tmp<rand>/foyer/foyer-storage-direct-fs-00000000 (deleted)
5472 deleted 16 MiB regions = 85.5 GiB logical, ~11 GB resident, pinning a 14 GB tmpfs at 100%

A daemon restart reclaimed all of it (confirming held fds, not on-disk files).

Fix

  • Add BlockCache::close() (default no-op; in-memory impls hold no fds).
  • Implement it on FoyerBlockCache via foyer's HybridCache::close().
  • Call it in capture_once before the TempDir drops, on every exit path (the body is wrapped in a result async block so the early None returns still close). The TempDir cleanup then actually frees the regions.

Notes / follow-up

This fixes the fd leak. Separately, the profiler scratch uses std::env::temp_dir() (ignores [cache].dir); placing it on the configured cache filesystem is a follow-up. Operators can currently work around the tmpfs placement with TMPDIR=<real-disk> in the unit.

Test

cargo check -p glidefs clean. The result async block preserves the original control flow (.ok()? / return None) and output type (Option<Vec<u64>>).

The boot-set profiler (`capture_once`) opens a per-run `FoyerBlockCache`
in a `tempfile::TempDir` and never closes it — it only Arc-drops at
function return. foyer's SSD-tier region files stay open until the
storage engine is closed, so the `TempDir`'s `remove_dir_all` runs while
the fds are still held, orphaning them as deleted-but-held files that are
never reclaimed until the daemon process exits.

A long-lived daemon that profiles many images (instd drives a profile run
per rootfs compose/create) leaks these unbounded into `$TMPDIR`. On hosts
where `/tmp` is a tmpfs this fills RAM and ENOSPCs every other process on
the node. Observed in the field: 5472 deleted 16 MiB regions (~85 GiB
logical) pinning a 14 GB tmpfs at 100%; a daemon restart reclaimed it all.

Add `BlockCache::close()` (default no-op; in-memory impls hold no fds),
implement it on `FoyerBlockCache` via foyer's `HybridCache::close()`, and
call it in `capture_once` before the `TempDir` is dropped, on every exit
path (the work is wrapped in a `result` async block so early `None`s still
close). The TempDir cleanup then actually frees the region files.

Note: this fixes the fd leak. Placing the profiler scratch on the
configured `[cache].dir` instead of `std::env::temp_dir()` is a separate
follow-up; operators can currently point `$TMPDIR` at a real-disk path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added 1 comment

}

async fn close(&self) {
if let Err(e) = self.inner.close().await {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[🟡 Medium] [🔵 Bug]

FoyerBlockCache::close logs and suppresses self.inner.close() failures, so capture_once cannot detect that cleanup did not complete and may still leave deleted-but-held region files pinned in $TMPDIR; this weakens the new leak fix under error conditions. Return a Result from BlockCache::close and propagate/handle it at the call site so failed shutdown is explicit and retryable instead of silently continuing.

// glidefs/src/block/cache.rs
async fn close(&self) {
    if let Err(e) = self.inner.close().await {
        tracing::warn!(error = %e, "foyer clean cache close failed");
    }
}

@jaredLunde jaredLunde merged commit 4894f00 into main Jun 27, 2026
38 of 39 checks passed
@jaredLunde jaredLunde deleted the fix/boot-profiler-foyer-fd-leak branch June 27, 2026 17:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant