feat: unified cross-detector confidence calibration (0.8.0)

[copyleftdev]

What

Confidence was computed three incompatible ways — 1−p (dist/mv), logistic-over-threshold (point/ctx/coll/psi), linear (cadence) — so a 0.9 meant different things per detector and severity / --top / --min-severity couldn't rank across detectors. Now every detector routes through one shared ax_detect::calibrate:

confidence = logistic(STEEPNESS · relative excess past the firing threshold)

• from_exceedance(stat, threshold) — "fires above" (modified z, CUSUM shift, PSI, null-rate)
• from_undercut(stat, threshold) — "fires below" (p-value < alpha, CV < max, point FDR's BH cutoff)

0.5 at the threshold, rising toward 1.0; a finding "2× past threshold" earns the same confidence on any detector (unit test pins this exactly). Removed the duplicated/divergent shift_confidence, psi_confidence, robustz::confidence helpers.

Why it's better

The old 1−p saturated to ~1 (everything "critical"); the logistic gives a real gradient and comparability, so severity ranking across detector families is now meaningful — which is what makes the 0.6.0 --top/--min-severity trustworthy.

Impact / contract

Recalibrates every published confidence and severity. config_version bumped anomalyx-cfg/7 → /8 so agents see it. Envelope shape and PROTOCOL unchanged. Structural schema-diff findings keep their fixed priors (binary facts, not statistics); the null-rate finding joins the unified scale.

Gate

proptest + cargo-mutants 0 missed: calibrate.rs 32/32 caught; touched detectors point/ctx/coll/cadence/structural/config/robustz 178 caught/6 unviable; dist/mv 193 caught/5 unviable + 10 loop-bound timeouts (detected, per the gate rule). calibrate has exact at-threshold, monotonicity, 2×-comparability, and boundary/NaN tests.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Unified confidence calibration across all detectors for consistent scoring methodology.

• Changed

  • Version updated to 0.8.0.
  • Configuration fingerprint version incremented.
  • Confidence values now computed using shared calibration approach.

• Documentation

  • Enhanced documentation explaining how confidence values are calibrated consistently across detectors.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR implements unified confidence calibration across all detectors in the anomaly detection suite. A new calibrate module provides logistic-transform functions (from_exceedance and from_undercut) that compute comparable confidence scores based on how far each detector's statistic sits past or under its threshold, with 0.5 confidence at the threshold boundary. All detector implementations are updated to use this shared calibration, the old detector-specific helpers are removed, and the configuration fingerprint is bumped to anomalyx-cfg/8.

Changes

Unified Confidence Calibration

Layer / File(s)	Summary
Calibration module and public export crates/ax-detect/src/lib.rs, crates/ax-detect/src/calibrate.rs	New public module calibrate defines logistic steepness constant, private logistic helper, and public from_exceedance(statistic, threshold) and from_undercut(statistic, threshold) functions. Both saturate outputs for boundary cases (nonpositive statistics/thresholds) and guarantee confidence in [0, 1] with 0.5 at threshold. Comprehensive test suite validates threshold, monotonicity, saturation, cross-direction comparability, and unit interval constraints.
Point detector calibration in fixed-threshold and FDR modes crates/ax-detect/src/point.rs	PointDetector switches confidence computation from robustz::confidence(...) to calibrate::from_exceedance(modz, cfg.point_threshold) in fixed-cutoff mode and calibrate::from_undercut(p, cutoff) in FDR mode. Internal scan_column_fdr method signature simplified by removing DetectConfig parameter; call sites and test assertions adjusted accordingly.
Distribution detectors (KS, PSI, Chi2) calibration crates/ax-detect/src/dist.rs	KsDetector and Chi2Detector now derive confidence from calibrate::from_undercut(p, cfg.dist_alpha) on p-values; PsiDetector uses calibrate::from_exceedance(value, cfg.psi_threshold). Internal psi_confidence helper removed. Tests updated to validate calibrated outputs instead of prior logistic behavior.
Single-detector calibration updates crates/ax-detect/src/cadence.rs, crates/ax-detect/src/coll.rs, crates/ax-detect/src/ctx.rs, crates/ax-detect/src/mv.rs, crates/ax-detect/src/structural.rs	CadenceDetector uses calibrate::from_undercut(cv, cfg.cad_max_cv), CusumDetector uses calibrate::from_exceedance(shift, cfg.coll_threshold), SeasonalDetector uses calibrate::from_exceedance(modz, cfg.ctx_threshold), MahalanobisDetector uses calibrate::from_undercut(p, cfg.mv_alpha), and SchemaDetector uses calibrate::from_exceedance(frac, cfg.struct_null_rate). All remove internal helper functions; tests updated to expect calibrated confidence values.
Remove old robustz confidence helper crates/ax-detect/src/robustz.rs	Removes public confidence(modz, threshold) function and its logistic-mapping test. Test module refocused on score arithmetic exactness and center_scale MAD→σ→none fallback validation.
Configuration fingerprint and version bump Cargo.toml, crates/ax-detect/src/config.rs	DetectConfig::version() fingerprint prefix updated from anomalyx-cfg/7 to anomalyx-cfg/8. Workspace package version bumped from 0.7.0 to 0.8.0 in [workspace.package] and across ax-core, ax-normalize, ax-detect dependencies.
Documentation and release notes CHANGELOG.md, docs/src/envelope.md	CHANGELOG.md documents unified calibration, config fingerprint bump, and parser robustness test updates; [Unreleased] link adjusted to start from v0.8.0. envelope.md clarifies that confidence is a logistic transform relative to threshold (with 0.5 at threshold), producing comparable values across detectors; severity derives from calibrated confidence while score remains raw/uncalibrated.

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• copyleftdev/anomalyx#56: Modifies crates/ax-detect/src/point.rs point detector logic—both PRs change how per-finding confidence is computed, with main PR introducing calibration and related PR adding role-based column skipping.

• copyleftdev/anomalyx#52: Adds --cad-max-cv CLI flag that sets cad_max_cv in DetectConfig, which directly feeds the cadence detector's new calibrate::from_undercut(cv, cfg.cad_max_cv) calibration call.

Poem

🐰 All detectors now speak the same tongue,

Calibrated confidence, logistic and young,

From KS to PSI, from cadence so fine,

Threshold at 0.5, the confidence line! ✨

One scale to rule them, one truth to unfold.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: unified confidence calibration across detectors with version bump to 0.8.0.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/confidence-calibration

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

crates/ax-detect/src/dist.rs (1)

731-753: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Stale test name and comment contradict the new calibrated confidence.

The name chi2_detector_confidence_is_one_minus_pvalue and the comment confidence≈0.954 describe the removed 1 − p behavior. With calibrate::from_undercut(p, dist_alpha), this case (χ²=4.0, p≈0.0455, α=0.05) now yields confidence ≈ 0.549, not 0.954. The surviving confidence < 0.99 assertion still passes but no longer validates anything meaningful. Consider mirroring ks_detector_confidence_is_calibrated_from_pvalue and asserting against the calibrated value.

💚 Rename and correct the comment (optionally tighten the assertion)

-    #[test]
-    fn chi2_detector_confidence_is_one_minus_pvalue() {
-        // 50/50 → 40/60 gives χ²=4.0, p≈0.0455 (flagged), confidence≈0.954.
+    #[test]
+    fn chi2_detector_confidence_is_calibrated_from_pvalue() {
+        // 50/50 → 40/60 gives χ²=4.0, p≈0.0455 (flagged); confidence is the
+        // unified undercut mapping of (p vs alpha), ≈0.549 here — distinguishable
+        // from the old 1−p value.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/ax-detect/src/dist.rs` around lines 731 - 753, Rename and update the
test chi2_detector_confidence_is_one_minus_pvalue to reflect calibrated behavior
(e.g., chi2_detector_confidence_is_calibrated_from_pvalue), update the inline
comment to describe that χ²=4.0 ⇒ p≈0.0455 and with dist_alpha=0.05 the
confidence is calibrated via calibrate::from_undercut(p, dist_alpha) (≈0.549),
and replace the weak assertion (out.findings[0].confidence < 0.99) with a
tighter check that the reported confidence from Chi2Detector (invoked via
Chi2Detector.detect with ScanContext::compared and DetectConfig::default)
approximately equals the calibrated value (use a small epsilon tolerance); keep
the existing assertion that out.absent.is_empty().

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@crates/ax-detect/src/dist.rs`:
- Around line 731-753: Rename and update the test
chi2_detector_confidence_is_one_minus_pvalue to reflect calibrated behavior
(e.g., chi2_detector_confidence_is_calibrated_from_pvalue), update the inline
comment to describe that χ²=4.0 ⇒ p≈0.0455 and with dist_alpha=0.05 the
confidence is calibrated via calibrate::from_undercut(p, dist_alpha) (≈0.549),
and replace the weak assertion (out.findings[0].confidence < 0.99) with a
tighter check that the reported confidence from Chi2Detector (invoked via
Chi2Detector.detect with ScanContext::compared and DetectConfig::default)
approximately equals the calibrated value (use a small epsilon tolerance); keep
the existing assertion that out.absent.is_empty().

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ba71cc34-4141-4bc2-b6d1-912d85c1ccaa

📥 Commits

Reviewing files that changed from the base of the PR and between 024d059 and 995f8ac.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (14)

• CHANGELOG.md
• Cargo.toml
• crates/ax-detect/src/cadence.rs
• crates/ax-detect/src/calibrate.rs
• crates/ax-detect/src/coll.rs
• crates/ax-detect/src/config.rs
• crates/ax-detect/src/ctx.rs
• crates/ax-detect/src/dist.rs
• crates/ax-detect/src/lib.rs
• crates/ax-detect/src/mv.rs
• crates/ax-detect/src/point.rs
• crates/ax-detect/src/robustz.rs
• crates/ax-detect/src/structural.rs
• docs/src/envelope.md

💤 Files with no reviewable changes (1)

• crates/ax-detect/src/robustz.rs