feat(analysis): add RunTrendAnalyzer — detect score regression across sequential runs#104
Open
nanookclaw wants to merge 2 commits intopinchbench:mainfrom
Open
feat(analysis): add RunTrendAnalyzer — detect score regression across sequential runs#104nanookclaw wants to merge 2 commits intopinchbench:mainfrom
nanookclaw wants to merge 2 commits intopinchbench:mainfrom
Conversation
…etection Detects whether a model's benchmark score is improving, stable, or degrading across sequential runs using OLS slope fitting over a sliding window. - RunPoint dataclass with run_id, timestamp, score, task_count - RunTrendReport with regression_detected, task_count_varies flag - Narrowed exception handling (JSONDecodeError, OSError) in load_points - CLI output warns when suite composition changed across trending window - Pure stdlib (statistics.linear_regression), no new dependencies See Issue pinchbench#101 for full proposal and maintainer review thread.
- test_no_data_returns_empty: empty directory returns [] - test_single_run_returns_empty: needs >= 2 runs for trend - test_regression_detected: declining scores triggers flag - test_improving_not_regression: positive slope not flagged - test_malformed_file_skipped: bad JSON files skipped gracefully - test_task_count_varies_flag: suite expansion sets warning - test_task_count_varies_false_when_equal: consistent suite no warning - test_summary_string_regression: CLI output format - test_summary_string_task_count_warning: warning in summary - test_stable_scores: zero slope flat detection - test_multiple_models: concurrent analysis per model, sorted - test_filter_by_model: single model query
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements RunTrendAnalyzer as discussed in #101.
Detects whether a model's benchmark score is improving, stable, or degrading across sequential runs using OLS slope fitting over a configurable sliding window.
Changes
New file:
scripts/lib_trend.pyRunPoint— dataclass (run_id, timestamp, model, score_pct, task_count)RunTrendReport— analysis result withregression_detectedandtask_count_variesflagRunTrendAnalyzer.load_points()— parses result JSONs, groups by model, sorts by timestamp. Narrowed exception handling:(json.JSONDecodeError, OSError)with silent skip.RunTrendAnalyzer.analyze()— OLS slope viastatistics.linear_regression, configurable window + thresholdRunTrendReport.summary()— CLI-friendly output with task-count-varied warningDesign decisions
statistics.linear_regression, no new dependenciestask_count_variesflag warns when the benchmark suite composition changed across the trending windowTests
12 new tests in
tests/test_lib_trend.pycovering:CLI integration
Can be wired into benchmark.py post-run:
Closes #101.
cc @Soham-o — all review points addressed per our discussion. ✅ Exception narrowed ✅ task_count_varies ✅