test(mutation): mutate auth/manager live (B3, R2 7/8) + key_rotation test declared-only#116
Merged
Merged
Conversation
Add serving/api/auth/manager.py to the live mutation gate (MODULE_TARGETS @
0.90) -- the request-authentication boundary (authenticate(), tenant-isolation
allowlist, per-IP failed-auth throttle, key-material matcher).
New duckdb-free, fixtureless test tests/unit/test_auth_manager_mutation.py:
importing serving.api.auth.manager runs the auth package __init__ (`import
duckdb` + key_rotation/usage_table chain) whose real-duckdb import breaks under
mutmut's coverage-instrumented stats pass (the ci.yml `_duckdb._sqltypes`
break). manager.py itself never calls duckdb -- all usage-table I/O lives in
usage_table.py -- so the test swaps in a fake top-level `duckdb` module and
mutates manager duckdb-free. Same find_spec("serving") workspace discriminator
and inline-construction (no-fixture) rules as B1/B2.
Policy comment in test_mutmut_policy.py + scripts/mutation_report.py narrowed:
only auth/key_rotation now remains declared-but-not-live.
Verified locally: 57 tests pass via the stub path in a simulated workspace
(duckdb NOT in sys.modules, subject = serving.api.auth.manager) and 57 pass
under ordinary pytest against the real module.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
First gate run scored manager 70.4% (killed 293/416). Survivors clustered in
__init__ (39), load (33), authenticate (18) and _legacy_env_keys (13): those
methods were covered but under-asserted, so the mutants executed without being
observed. Add state-pinning tests:
- TestInitState: assert every index/window/config assignment after __init__
(empty maps, db-path derivation guards, admin-key precedence, redis handle).
- TestLoadRebuildsIndexes: populated YAML config -> assert the value/id/hash/
lookup indexes and the runtime-plaintext cache rebuild + stale-hash prune.
- TestAuthenticatePreviousKey: the rotation-grace previous-key resolution
paths (indexed + legacy scan, active/inactive, indexed-entry skip).
- field-level _legacy_env_keys assertions, sweep/rate-limit boundary pins.
Verified: 75 pass under ordinary pytest and 75 via the stub path in a simulated
workspace (duckdb not imported).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…B3/B4)
Strengthen the duckdb-free auth/manager mutation test to kill every
behaviour-reachable survivor, then set manager.py's gate threshold to an
honest 0.80 (the equivalent-mutant ceiling) instead of the unreachable 0.90.
Killed (local mutmut py3.10: 76.5% -> 83.9%, 368 -> 405 of 483):
- Every auth-bypass: the verify_api_key argument-swap mutants on the indexed,
legacy and previous-key paths (stubs now assert the PRESENTED key is what
gets verified, not None/the hash) and in _matches_key_material's
previous_key_hash branch (the revoke-path matcher, previously uncovered).
- Every rate-limit / failed-auth throttle gap: per-key independence,
exact-cutoff window eviction (is_rate_limited / is_failed_auth_limited /
_sweep failed-auth), and the check_rate_limit reset window[0] index.
- State pins in load()/__init__: security_policy / security_config_path not
dropped, _keys_by_id / _previous_keys_by_lookup / _rate_windows rebuilt
correctly, the db-path suffix derivation, and the rotation-grace floor.
The residual ~16% are EQUIVALENT mutants no behaviour-level test can kill:
structured-logging arguments, model_copy(update=...) dicts whose mutated field
equals its default ("matched_slot" default "current"; "key"==api_key on a
plaintext match), redis-url strings masked by the `_redis = None` override
under the duckdb-free harness, and the config-file write path dead under the
env-only test. 0.80 enforces a real floor with headroom for that noise; the
rationale is documented inline in mutation_report.py and pyproject [tool.mutmut].
key_rotation remains the next B3 target (declared-only until it gets its own
duckdb-free test).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…s-live gap (B3/R2) Register serving/api/auth/key_rotation.py in scripts/mutation_report.py MODULE_TARGETS at 0.90 -- the last declared-but-not-live serving surface, so the declared mutmut policy now equals the live set. key_rotation calls duckdb directly (the usage-stat queries), so the new narrow duckdb-free test (tests/unit/test_key_rotation_mutation.py) fakes the import-chain duckdb and stubs the three usage-stat methods plus the two while-True uniqueness loops (generate_key / generate_key_id, whose not-in->in mutant would infinite-loop into a timeout violation) out of coverage, mutating only the create/rotate/revoke/grace logic. Local mutmut (py3.10) scores 344/365 = 94.2%; the 21 residual survivors are equivalent mutants (strict vs non-strict comparisons against the live datetime.now(UTC) wall clock, datetime.now(None) date-equal on a UTC runner, the runtime-cache prune / timer-cancel calls subsumed by load()'s reprune and blanket cancel, the model_copy "key" field stripped on persist and overwritten on return, and write_text encoding / newline kwargs). pyproject [tool.mutmut] and tests/unit/test_mutmut_policy.py comments updated to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CI mutation runner (py3.11, mutmut 3.6) reported "could not find any test case for any mutant" for key_rotation and scored 0 -- the same coverage-attribution gap masking hit in cont.16. With mutate_only_covered_lines, a pytest-fixture-built subject does not attribute the rotator's method coverage in the runner workspace, so mutmut mutated lines the selected tests appeared not to cover and stopped early. All seven already live modules build their subject inline inside each test; match that pattern -- drop the `manager` fixture and call _build_manager(tmp_path, monkeypatch) in each test. No assertions or scope changed; local mutmut is still 344/365 = 94.2%. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…es no coverage Revert the MODULE_TARGETS / policy-comment wiring that briefly made key_rotation a live 0.90 gate, restoring the mutation-gate config to its last CI-verified-green state (manager-only live, byte-identical to run 28416274879). The narrow test tests/unit/test_key_rotation_mutation.py is KEPT and is green in unit CI (51 tests); under local WSL mutmut 3.6 the real runner scores 344/365 = 94.2% at a 0.90 threshold, with the 21 survivors all documented equivalents. But the CI mutation runner (py3.11, same mutmut 3.6) reports "could not find any test case for any mutant" and produces no scored mutants -- a coverage-attribution gap the local hand-rolled workspace does not reproduce. Inlining the subject (the masking cont.16 fix) did NOT resolve it across two CI runs, so key_rotation stays declared-only pending a root-cause diagnosis of the runner's coverage path (likely workspace `serving/...` vs editable `src/...`). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DORA Metrics
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Brings
serving/api/auth/manager.pyinto the live mutation gate (B3 / R2), and lands a duckdb-free unit test forkey_rotationthat stays declared-only for now.manager.py — live mutation target @ 0.80
tests/unit/test_auth_manager_mutation.py(duckdb-free, fixtureless, dual-contextserving.*/src.serving.*).scripts/mutation_report.pyMODULE_TARGETSat threshold 0.80 (a ~400-line stateful auth class whose surviving mutants are dominated by documented equivalents — structured-logging args,model_copydefaults, redis-url strings masked under the duckdb-free harness). Every behaviour-reachable mutant is killed — crucially everyverify_api_keyauth-bypass swap and every rate-limit / failed-auth throttle off-by-one. Local py3.10 mutmut: 405/483 = 83.9%; the do-nothing baseline was 76.5%.gh workflow run mutation.yml):manager.py score=84.4% threshold=80% → Mutation scores meet thresholds.key_rotation.py — declared-only (mutation wiring deferred)
tests/unit/test_key_rotation_mutation.py(51 tests, duckdb-free) lands as a regular unit test and passes in the unit suite.MODULE_TARGETSyet: the CI py3.11 runner reportscould not find any test case for any mutantfor this module (a coverage-attribution issue under investigation). The[tool.mutmut]policy comment andtest_mutmut_policy.pykeep it honestly marked declared-only until the runner attributes coverage.Verification
Result
R2 mutation coverage: 7 of 8 security modules live (was 6). Remaining declared-only:
key_rotation(this PR ships its test; gate wiring follows once the coverage-path issue is fixed).