Skip to content

feat(runner): successive-halving and Hyperband (#104)#107

Merged
jc-macdonald merged 1 commit into
mainfrom
feat/104-successive-halving
May 12, 2026
Merged

feat(runner): successive-halving and Hyperband (#104)#107
jc-macdonald merged 1 commit into
mainfrom
feat/104-successive-halving

Conversation

@jc-macdonald
Copy link
Copy Markdown
Contributor

Closes #104.

Summary

Adds multi-fidelity early-stopping of trials within a batch via successive-halving and Hyperband. Distinct from #78 (closed), which proposed phase-level fidelity swap; this prunes trials within a single phase based on intermediate observations.

New API

PartialEvaluator protocol (protocols.py)

@runtime_checkable
class PartialEvaluator(Protocol):
    def evaluate(self, config: dict[str, Any], budget: float) -> dict[str, float]: ...

Budget is opaque to the runner — interpret it as epochs, MCMC iterations, dataset fractions, mesh resolutions, or seconds. Implementations are free to cache intermediate state internally.

run_successive_halving (runner.py)

results = run_successive_halving(
    trials,
    sim,
    rungs=[1, 3, 9, 27],
    eta=3,
    metric="val_loss",
    mode="min",
)
  • Evaluates every trial at the lowest rung.
  • Keeps top ceil(n_prev / eta) by metric at each rung; promotes survivors.
  • Records every (trial, rung) evaluation as one row in ResultsTable.
  • Per-row metadata: rung, budget, trial_index, promoted, wall_seconds.

run_hyperband (runner.py)

Implements the Li et al. (2017) bracket schedule on top of successive-halving. Takes a trial_factory(bracket_idx, n) -> trials callable so brackets can sample independently (e.g. with bracket-derived seeds passed to build_grid).

Tests

11 new tests in tests/test_runner.py:

  • ranking correctness (lowest target survives)
  • row count across rungs
  • metadata schema (rung/budget/trial_index/promoted/wall_seconds)
  • mode="max" reverses ranking
  • input validation (empty trials/rungs, non-ascending, eta ≤ 1, bad mode)
  • KeyError on missing metric
  • Hyperband bracket coverage
  • PartialEvaluator runtime-checkable

CI

just ci clean: lint, mypy strict, 284 tests pass, runner.py at 97% coverage.

Follow-ups

Add multi-fidelity early-stopping of trials within a batch.

- New PartialEvaluator protocol in protocols.py: evaluate(config, budget)
  returns observables. Budget is opaque (epochs, iterations, dataset
  fraction, seconds, ...).
- run_successive_halving(trials, sim, *, rungs, eta, metric, mode):
  evaluate all trials at the lowest rung, keep top 1/eta survivors,
  promote to the next budget, repeat. Records every (trial, rung)
  evaluation as a row in ResultsTable with rung/budget/trial_index/
  promoted/wall_seconds in per-row metadata.
- run_hyperband(trial_factory, sim, *, max_budget, eta, metric, mode):
  Li et al. 2017 bracket schedule wrapping successive-halving.
  trial_factory(bracket_idx, n) returns fresh configs per bracket,
  letting callers seed differently across brackets.
- Distinct from #78 (closed) which was phase-level fidelity; this
  prunes trials within a single phase based on intermediate scores.

11 new tests: ranking correctness, row counts, metadata schema,
max-mode, input validation, missing-metric KeyError, Hyperband bracket
coverage, and PartialEvaluator runtime check.
@jc-macdonald jc-macdonald merged commit 8fa9302 into main May 12, 2026
4 checks passed
@jc-macdonald jc-macdonald deleted the feat/104-successive-halving branch May 12, 2026 14:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multi-fidelity early-stopping within a batch (Hyperband / successive-halving)

1 participant