[dmytrogajewski]

Scope

Analyzer audit fixes, plus the restoration of bounded-memory history processing that the Go → Rust rewrite lost. CHANGELOG.md [Unreleased] lists every change.

• Bounded-memory history core.
  • Every history walk folds through a worker pool (history/pool.rs). Workers claim runs of consecutive commits from a shared cursor, steal from idle peers, and run at most one chunk ahead of the in-order fold.
  • Free heap pages go back to the OS after each chunk (cf_meminfo::release_free_heap).
  • Output is byte-identical for every chunk size.
• --memory-budget / --chunk-size, also as pipeline.memory_budget / pipeline.chunk_size.
  • Each walk plans its chunk size and worker count from its share of the budget and its measured per-walk costs.
  • libgit2 gets 25 % of the budget for its object cache and 7.5 % for mapped pack windows.
  • -v prints each plan.
• UAST analyzers.
  • They parse one change at a time and run as one memoized pass.
  • Their finished results spill to an unlinked temp file while other history walks run.
• history/burndown prepares its per-change work on the pool and reuses the pre-pass tree diffs of heavy commits: 59.4 s → 25.9 s.
• --shotness-keep-ended (opt-in) keeps ended shotness entities, marked ended. The default report is unchanged.
• codefang mcp serves codefang_analyze, uast_parse and codefang_history over stdio. Each tool result is the CLI's own report (cf_commands::render_run).
• Removed: the retired Go differential gate and its oracle data.

Measurements

Workload: 31,923 kubernetes commits, codefang run -a 'history/*' --format json, 24 cores, quiet machine. Output is byte-identical to the previous build (md5 15fbc154f5393227f96f87ad6165cb4a, 538 MB).

Build / flags	Wall	Peak RSS
previous build	399.8 s	12.9 GiB
automatic budget (2 GiB)	334.4 s	3.56 GiB
--memory-budget 4GB	327.8 s	3.79 GiB

Test matrix

• Gates: cargo fmt --check, make lint (0 findings), make deadcode (0 findings), 3138 tests passed.
• Chunk-size equality: bins/codefang/tests/chunked_history_e2e.rs checks 10 history analyzers × 9 formats × chunk sizes {1, 2, 7, 1000}. It also covers the co-selected UAST pass, budget invariance, the plan lines and the spill path.
• Other end-to-end suites:
  • mcp_e2e.rs: the handshake, tool listing, and all three tools against the CLI reports.
  • shotness_e2e.rs: both modes.
  • config_e2e.rs: the new keys.
• Unit tests: the pool (ordering, stealing, failures, panics), the planner, the spill round trip, burndown's change facts against the sequential reference, and libgit2 native limits. Each new mechanism was red first or mutation-checked.

Risks

• libgit2 options are process-global. The memory limits are set once, before the first repository opens (cf_gitlib::limit_native_memory).
• Two new unsafe calls, each behind a safe wrapper with a test: malloc_trim(0) and libgit2's mapped-pack and object-cache limits.
• tmpfs /tmp. On a tmpfs /tmp the spill leaves the process but not RAM; TMPDIR points it at a disk.
• --shotness-keep-ended on long histories: the shotness report grows 3.4× (256 MB → 859 MB on the window above).

Rollback

• --chunk-size 1000000 reproduces the unbounded walk.
• Reverting this commit restores the previous pipeline. The report bytes are unchanged, so consumers are unaffected.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KmiE8YcZ3GtgQW2bDbcgXo