Skip to content

fix(history): port --languages filter (HistoryFilter) + root-cause oracle timeout - #27

Closed
dmytrogajewski wants to merge 2 commits into
mainfrom
fix/history-languages-filter
Closed

fix(history): port --languages filter (HistoryFilter) + root-cause oracle timeout#27
dmytrogajewski wants to merge 2 commits into
mainfrom
fix/history-languages-filter

Conversation

@dmytrogajewski

@dmytrogajewski dmytrogajewski commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes the two known gaps left open after #26.

Gap 1 — history phase ignored --languages (FIXED, oracle-verified)

Ports the Go TreeDiffAnalyzer.checkLanguage to Rust as HistoryFilter, threaded through every history change-filter site (devs, burndown, anomaly, couples, shotness, file-history, imports, typos, quality, sentiment). Oracle-verified byte-identical for devs/burndown/anomaly/couples across language flagsets.

HistoryFilter encodes two Go-history quirks measured against the live oracle:

  • History ignores --include-vendored/--include-generated entirely — the flag channel never reaches the history pipeline (verified on hercules + kubernetes: byte-stable 1851/16692 across flag/no-flag). policy is always Options::default().
  • --languages "" (empty value) restricts everything away (total_commits=0), distinct from --languages "all"/absent (no filter). Empty set ≠ no filter — matches Go's empty Languages map rejecting every change via checkLanguage.

Also fixes a latent burndown findLastTick parity bug exposed by language filtering: last_tick is now the max delta-recorded GlobalHistory key (Go findLastTick), not the max walked commit tick. This was the num_samples 1 vs 6 divergence on --languages Python.

Gap 2 — 6 --include-vendored@hercules static oracle-timeout cells (ROOT-CAUSED, evidence-backed)

Two layers, both diagnosed:

  1. The frozen-oracle timeout is a Go BUG, not untestability: pkg/uast/parser_dsl.go toCanonicalNode re-walks every unmapped subtree twice (2^depth blowup). A perf-only, output-byte-identical Go patch — the same restructure Rust already landed in cf-uast/src/lowering.rs — makes the oracle complete the cell with a full 5,351,785-byte report (proven byte-identical to the frozen binary on 9 diverse cells). Committed as tests/compat/oracle/patches/uast-unmapped-subtree-rewalk-perf.patch.
  2. The residual divergence is REAL and DELIBERATE on the Rust side: Rust's 64 MiB durability cap skips vendor/gopkg.in/vmarkovtsev/BiDiSentiment.v1/assets/bindata.go (70.5 MB; Go has no such cap and parses it) → 21473 vs Go's 21494 total functions (19 top-level funcs + 2 methods), plus tie-order diffs.

These 6 cells stay RED (recorded as tracked_known_failing via a label_glob covering all 6 static cells). Measuring them in-suite needs an explicit oracle re-bless (tamper_check --bless, the oracle SHA is fingerprinted) plus a policy call on the deliberate durability caps and a ~20 min-per-cell runtime — that is the explicit remaining follow-up, not something to patch around.

Gate fix

gate.py _match had a tautology: rec.get("kind", entry["kind"]) defaulted the record's kind to the entry's own kind, so a family+kind allowlist entry matched every FAIL in that family (records carry no kind). Now requires an explicit kind match, so the 6 vendor cells correctly fall through to tracked_known_failing (honest RED) instead of being mis-counted as ALLOWED.

Also added a go_nondeterminism allowlist entry for flag:static/clones--languages:Go (6 distinct live-Go output SHAs; Go MinHash/LSH map-iteration nondeterminism).

Test plan

  • cargo fmt --check — clean
  • cargo clippy -p cf-commands --all-targets — exit 0 (only pre-existing warnings)
  • cargo test --workspace — 163 suites, 0 failures (incl. 9 new history_filter_tests via git2 fixture, +5 flag_parity integration)
  • python3 tests/compat/integrity/tamper_check.py — GREEN (oracle binaries untouched; gate.py/allowlist.json not protected)
  • python3 tests/compat/gate.py --tier full — honest tally: TRACKED_KNOWN_FAILING: 6 (vendor cells stay RED), FAIL (un-allow): 0, ALLOWED: 0; flag:history/devs--languages:Go@hercules now PASSES

Stack

⚠️ Stacks on #26 (fix/static-flag-filter-parity, still open). Merge #26 first, then this one rebases cleanly onto main.

🤖 Generated with Claude Code

drwatsno and others added 2 commits August 2, 2026 18:28
@drwatsno @claude
fix(static): thread static flags into every analyzer and output format
89b1a5e 
Non-json formats of static/complexity and all of halstead/cohesion/
comments/imports/clones walked with a default path policy, silently
ignoring --include-generated/--include-vendored; --languages was never
consumed by static analyzers at all. On generated-code-heavy repos this
made text/json report divergent totals for identical flags (observed:
2178 vs 18653 total functions on a protobuf-heavy repo).

- Add StaticFilter (path policy + language globs) in cf-commands and
  thread it into every static walk, every format arm, multi-analyzer
  merges, and plot builders; language globs use basename fnmatch
  (filepath.Match parity) via cf-langpath
- Match cobra csv semantics: --languages "" means no filter (was an
  unknown-language error); verified byte-identical to the Go oracle
- Fix clones serialization parity: empty non-nil clone_pairs now emits
  [] (vs Go's null for the nil case), via Option<Vec<ClonePair>
- Add 16 regression tests incl. dispatch-level tests through the real
  registry (mutation self-test verified they catch the shipped bug
  shape), plus integration tests in cf-commands/tests/flag_parity.rs
- Extend compat matrix flag family (flags x formats x analyzers,
  --languages axes): 9 -> 63 cells full tier, 0 -> 8 smoke tier;
  re-bless integrity baseline, refresh result artifacts
- Record the known gap in allowlist (tracked_known_failing): the
  history phase ignores --languages (Go TreeDiffAnalyzer.checkLanguage
  not yet ported); entry stays RED in the gate

Verified against the live Go oracle: 303 fixture-cell sweep clean,
smoke flag family 8/8 PASS, tamper-evidence GREEN.

Co-Authored-By: Claude <noreply@anthropic.com>
@drwatsno @claude
fix(history): port TreeDiffAnalyzer.checkLanguage as HistoryFilter; r…
f89f198 
…oot-cause oracle timeout

Port the Go history-phase --languages filter (TreeDiffAnalyzer.checkLanguage)
to Rust as HistoryFilter, threaded through every history change-filter site
(devs, burndown, anomaly, couples, shotness, file-history, imports, typos,
quality, sentiment). Oracle-verified byte-identical for devs/burndown/anomaly/
couples across language flagsets.

HistoryFilter encodes two Go-history quirks measured against the live oracle:
  * The history phase IGNORES --include-vendored/--include-generated (the flag
    channel never reaches the history pipeline; verified on hercules +
    kubernetes: byte-stable across flag/no-flag). policy is always Options::default().
  * --languages "" (empty value) restricts everything away (total_commits=0),
    distinct from --languages "all" / absent (no filter). Empty set != no filter.

Also fixes a latent burndown findLastTick parity bug exposed by language
filtering: last_tick is now the max delta-recorded GlobalHistory key (Go
findLastTick), not the max walked commit tick.

Root-cause the 6 --include-vendored@hercules static oracle-timeout cells:
  (1) The frozen-oracle timeout is a Go BUG (pkg/uast/parser_dsl.go
      toCanonicalNode re-walks every unmapped subtree twice, 2^depth blowup).
      A perf-only, output-byte-identical Go patch (the same restructure Rust
      landed in cf-uast/src/lowering.rs) makes the oracle complete the cell —
      committed as oracle/patches/uast-unmapped-subtree-rewalk-perf.patch.
  (2) The residual divergence is REAL and DELIBERATE: Rust's 64MiB durability
      cap skips vendor/.../bindata.go (70.5MB; Go has no cap) = 21473 vs 21494
      total functions. Cells stay RED pending an oracle re-bless + a durability-
      cap policy call — recorded as tracked_known_failing (label_glob covers all
      6 static cells).

Fix a gate.py _match tautology (rec.get("kind", entry["kind"]) defaulted the
record's kind to the entry's own kind, letting a family+kind entry swallow every
FAIL in that family). Records carry no kind; require an explicit match so the
6 vendor cells fall through to tracked_known_failing (honest RED) instead of
being mis-counted as ALLOWED.

Add go_nondeterminism allowlist entry for flag:static/clones--languages:Go
(6 distinct live-Go output SHAs; Go MinHash/LSH map-iteration nondeterminism).

Tests: +117 cf-commands lib tests incl. 9 history_filter_tests (git2 fixture),
flag_parity integration, 163 workspace suites green. Gate honest: 6 tracked
(still RED), 0 un-allowlisted FAIL.

Co-Authored-By: Claude <noreply@anthropic.com>
@cursor

cursor Bot commented Aug 2, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@dmytrogajewski

dmytrogajewski commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

Folded into #26 (its branch fix/static-flag-filter-parity now contains this commit via fast-forward) — keeping a single MR as requested. Closing in favor of #26.

@dmytrogajewski
dmytrogajewski deleted the fix/history-languages-filter branch August 2, 2026 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dmytrogajewski @drwatsno