feat(runner): successive-halving and Hyperband (#104)#107
Merged
Conversation
Add multi-fidelity early-stopping of trials within a batch. - New PartialEvaluator protocol in protocols.py: evaluate(config, budget) returns observables. Budget is opaque (epochs, iterations, dataset fraction, seconds, ...). - run_successive_halving(trials, sim, *, rungs, eta, metric, mode): evaluate all trials at the lowest rung, keep top 1/eta survivors, promote to the next budget, repeat. Records every (trial, rung) evaluation as a row in ResultsTable with rung/budget/trial_index/ promoted/wall_seconds in per-row metadata. - run_hyperband(trial_factory, sim, *, max_budget, eta, metric, mode): Li et al. 2017 bracket schedule wrapping successive-halving. trial_factory(bracket_idx, n) returns fresh configs per bracket, letting callers seed differently across brackets. - Distinct from #78 (closed) which was phase-level fidelity; this prunes trials within a single phase based on intermediate scores. 11 new tests: ranking correctness, row counts, metadata schema, max-mode, input validation, missing-metric KeyError, Hyperband bracket coverage, and PartialEvaluator runtime check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #104.
Summary
Adds multi-fidelity early-stopping of trials within a batch via successive-halving and Hyperband. Distinct from #78 (closed), which proposed phase-level fidelity swap; this prunes trials within a single phase based on intermediate observations.
New API
PartialEvaluatorprotocol (protocols.py)Budget is opaque to the runner — interpret it as epochs, MCMC iterations, dataset fractions, mesh resolutions, or seconds. Implementations are free to cache intermediate state internally.
run_successive_halving(runner.py)ceil(n_prev / eta)bymetricat each rung; promotes survivors.(trial, rung)evaluation as one row inResultsTable.rung,budget,trial_index,promoted,wall_seconds.run_hyperband(runner.py)Implements the Li et al. (2017) bracket schedule on top of successive-halving. Takes a
trial_factory(bracket_idx, n) -> trialscallable so brackets can sample independently (e.g. with bracket-derived seeds passed tobuild_grid).Tests
11 new tests in tests/test_runner.py:
mode="max"reverses rankingKeyErroron missing metricPartialEvaluatorruntime-checkableCI
just ciclean: lint, mypy strict, 284 tests pass, runner.py at 97% coverage.Follow-ups
run_successive_halving.