feat: add directory-level disk usage stats

[blinkagent]

Summary

Adds three new methods to Statter for directory-level disk usage measurement, complementing the existing filesystem-level Disk() method.

New Methods

DiskUsage(prefix Prefix, path string) (*Result, error)

Returns actual disk usage of a directory tree (similar to du -sh). On Unix systems, uses stat.Blocks for accurate block-level accounting and tracks inodes to avoid double-counting hard links.

DiskUsageWithTotal(prefix Prefix, path string) (*Result, error)

Same as DiskUsage but includes the filesystem total capacity from statfs/GetDiskFreeSpaceEx, enabling percentage calculations.

DiskUsageSimple(prefix Prefix, path string) (*Result, error)

A simpler/faster version that uses file sizes instead of blocks. On Windows, this is identical to DiskUsage.

Use Case

These methods are useful in containerized environments where you need to track usage of specific directories (e.g., /home, /var/lib/docker) rather than the entire filesystem. The existing Disk() method reports filesystem-level totals which shows the host filesystem in Docker containers.

Implementation Details

• Uses filepath.WalkDir for efficient directory traversal
• Skips files that can't be accessed (permission errors)
• On Unix: uses syscall.Stat_t.Blocks * 512 for accurate disk block usage
• On Unix: tracks visited inodes to handle hard links correctly
• On Windows: uses file.Size() (blocks not exposed by Windows API)
• Errors only on initial path access failure; inaccessible children are skipped

Performance Note

These operations can be expensive for large directory trees with many small files (e.g., node_modules). Callers should use appropriate refresh intervals (e.g., 1 hour as discussed in the issue).

Fixes #32

---

Created on behalf of @matifali

[blinkagent]

Staff Engineer Review 🔍

Putting on the review goggles, here are potential issues and gotchas to consider:

1. Symlink handling is missing

The code uses d.Info() which follows symlinks by default. If a symlink points outside the target directory, we'll count data that shouldn't be included. Worse, if there are symlink loops, filepath.WalkDir will follow them until it hits the OS path limit.

Suggestion: Use d.Type()&fs.ModeSymlink != 0 check to skip symlinks, or document the behavior explicitly.

2. Inode tracking doesn't consider device IDs

The hard link deduplication uses stat.Ino alone, but inodes are only unique per filesystem. If the directory tree spans multiple mount points, we could incorrectly skip files with coincidentally matching inode numbers on different devices.

Suggestion: Key on (stat.Dev, stat.Ino) tuple instead of just stat.Ino.

3. Silent error swallowing is risky

Returning nil for errors in the walk function means permission errors, I/O errors, and other failures are silently ignored. This could lead to dramatically underreported usage if a large subdirectory is unreadable.

Suggestion: Consider logging warnings, or provide an option to collect/surface these errors.

4. Unbounded memory usage

The visited inode map grows unbounded. For a directory with millions of files (think large mono-repos or node_modules forests), this could consume significant memory.

Consideration: For agent metadata use cases this is probably acceptable, but worth documenting. Could also consider a bounded LRU cache or bloom filter for extreme cases.

5. Race conditions during walk

Files can be created/deleted/modified during the walk. The current code handles some of this gracefully (skips errors), but d.Info() could return stale data from the DirEntry cache.

Consideration: For agent metadata polling (1hr intervals), this is probably fine, but worth documenting.

6. Test coverage gaps

• No test for symlinks (especially external/loop symlinks)
• No test for hard links (the feature we're claiming to handle)
• No test for permission-denied scenarios
• NonExistentPath test is a bit weak - accepts either error or zero result

7. API consistency question

The existing Disk() uses Bfree (blocks available to superuser) for calculating used space. For container use cases, might Bavail (blocks available to unprivileged users, accounting for reserved blocks) be more appropriate? This is an existing behavior, but worth considering for the new methods.

8. Windows hard link inconsistency

Windows also supports hard links (NTFS), but the Windows implementation doesn't deduplicate them. This creates a platform behavioral inconsistency. If Windows hard link deduplication is desired, could use GetFileInformationByHandle to get nFileIndexHigh/nFileIndexLow.

---

Summary: Most of these are edge cases that may not matter for the immediate use case (agent metadata in containers), but worth considering before this becomes a widely-used API. At minimum, I'd suggest addressing #1 (symlinks) and #2 (device ID) as they could cause incorrect results in common scenarios.

[blinkagent]

Addressed the key review feedback in the latest commit:

Fixed

1. Symlink handling - Now explicitly skips symlinks using d.Type()&fs.ModeSymlink != 0 to avoid:

   • Counting files outside the target directory
   • Infinite loops from symlink cycles
   • Double-counting if symlink target is also in the tree

2. Device ID for inode tracking - Changed from map[uint64]struct{} keyed on inode alone to map[devIno]struct{} keyed on (Dev, Ino) tuple. This correctly handles directory trees spanning multiple mount points.

3. Improved documentation - Added clear godoc comments explaining edge case behavior.

New Tests Added

• SymlinksAreSkipped - Verifies external symlinks aren't followed
• SymlinkLoopsDoNotCauseInfiniteLoop - Ensures symlink cycles don't hang
• HardLinksCountedOnce - Verifies hard link deduplication works
• PermissionDeniedSkipped - Confirms inaccessible files are gracefully skipped
• Updated DiskUsageSimple with symlink test

Deferred

Items I chose not to address in this PR (could be follow-ups):

• Windows hard link deduplication (would require GetFileInformationByHandle)
• Logging for skipped files (adds complexity, caller can implement if needed)
• Bounded memory for inode map (unlikely to be an issue for typical use cases)