Skip to content

feat(forecast): model zoo and backtesting comparison (PRP-36) #302

@w7-mgfcode

Description

@w7-mgfcode

PRP-36 execution issue under epic #295.

Scope. Slice B of the Forecast Intelligence roadmap. Land a disciplined model zoo with fair, leakage-safe backtesting comparison on top of the PRP-35 Feature Frame V2 contract. PRP at PRPs/PRP-36-forecast-intelligence-B-model-zoo-backtesting.md; final contract snapshot at PRPs/ai_docs/prp-35-final-contract-snapshot.md.

Deliverables (18 tasks, V1 only per #299 deferral).

  • New target-only baselines: weighted_moving_average, seasonal_average, trend_regression_baseline (Ridge).
  • New feature-aware optional model: random_forest (sklearn, gated by forecast_enable_random_forest).
  • Backtest extensions: RMSE, per-horizon-bucket metrics (h_1_7, h_8_14, h_15_28, h_29_plus), aggregate across folds.
  • Registry: _find_duplicate distinguishes V1/V2; find_comparable_runs(grain, overlapping window, same V).
  • RunResponse / RunDetailResponse: Optional feature_frame_version + feature_groups from runtime_info.
  • Ops: FEATURE_FRAME_VERSION_MISMATCH stale-reason; comparable-run predicate honours V.
  • Explainability: explainers for every new baseline + RF feature importances; HGBR continues to 422.
  • Docs + CHANGELOG + examples/forecasting/model_zoo_compare.py.

Explicitly out of scope.

  • V2 backtesting dispatch — deferred to feat(forecast): add feature frame v2 (PRP-35) #299 (PRP-35 Task 13 was incompatible with backtesting's per-fold-from-config architecture).
  • PRP-37 UI / interactive dashboard — Slice C, follows this PRP.
  • No new Alembic migration; all new state rides in existing JSONB columns.
  • No agent tool added; agent_require_approval unchanged.

Validation gates. All five must be green: ruff check, ruff format --check, mypy --strict, pyright --strict, pytest -m "not integration" (+ pytest -m integration for slice-touching tests).

Confidence. 7/10 per the PRP's confidence section. Library claims (sklearn HGBR/RF, np.average) verified at PRP-create time.

Epic: #295
Depends on: PRP-35 (merged in #300)
Blocks: PRP-37
Defers V2 backtesting: #299

Metadata

Metadata

Assignees

No one assigned

    Labels

    featNew featurescope:backtestTouches the backtest area (commit-format scope)scope:forecastTouches the forecast area (commit-format scope)scope:registryTouches the registry area (commit-format scope)

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions