test(mutation): mutate sql_builder + nl_queries live (B2)#115
Conversation
Close the declared-vs-live gap for the query surface: sql_builder.py and
nl_queries.py were declared mutation targets but the gate only ran retry,
sql_guard, masking and rate_limiter. Add duckdb-free fixtureless mutation
tests for both and wire them into scripts/mutation_report.py MODULE_TARGETS
@ 0.90, so 6 of 8 security modules are now mutated live (auth manager /
key_rotation remain declared-only pending their own duckdb-free tests).
Both modules live under the query package whose __init__ imports the
duckdb-backed QueryEngine, so each test stubs
serving.semantic_layer.query.{engine,contracts} (plus the src.* helpers)
before importing the module under a top-level `serving` name. nl_queries'
.sql_guard shim is repointed at the real top-level sql_guard so the
validate_nl_sql boundary runs against the genuine guard.
Reproduced on the WSL/mutmut harness (py3.10): sql_builder 96.0%
(killed 167/174), nl_queries 94.4% (killed 323/342). Remaining survivors
are documented equivalents (typing.cast type-string, dialect=duckdb->None
renders, sqlglot/regex table-extraction convergence, telemetry no-ops,
*1001 elapsed). The CI gate (mutation.yml, py3.11) is the source of truth.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DORA Metrics
|
…arness (B2)
The B2 duckdb-free mutation tests gated stub installation on `import src`
succeeding ("workspace has no top-level src"). That assumption is false: the
editable-installed repo keeps the real `src` on sys.path even inside mutmut's
mutants/ workspace, so the stubs were skipped there and the import fell through
to the real `src.serving.semantic_layer.query` package -> `from .engine import
QueryEngine` -> `import duckdb`. Under mutmut's coverage-instrumented stats pass
on py3.11 that trips duckdb's lazy `_duckdb._sqltypes` import (the same break
ci.yml already works around with `coverage run`), so `mutmut run` exited -11 with
no scored mutants and the gate failed.
Discriminate on the real signal instead: the workspace copies src/serving to a
TOP-LEVEL `serving` package, which ordinary pytest never has. Gate stub install
on `importlib.util.find_spec("serving")`. The stub path (engine/contracts/auth
neutered to no-ops) was effectively never exercised before -- WSL py3.10 also had
`src` importable, so its 96.0%/94.4% came from the real modules.
Verified locally in a simulated workspace (top-level `serving` + editable `src`,
as in CI): both modules import with duckdb NOT in sys.modules, subject under test
is `serving.semantic_layer.query.*` (the mutation target), 101 tests pass via the
stub path; the ordinary-pytest path still passes 101.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Update — py3.11 gate green after a harness fix. The first Fix ( Golden gate (py3.11, run
Matches the WSL py3.10 reproduction exactly. |
2 participants
What
Close the mutation declared-vs-live gap for the NL→SQL query surface (roadmap R2 / B2): make
sql_builder.pyandnl_queries.pymutate live in the gate, not just be declared.serving/semantic_layer/query/sql_builder.pyand.../nl_queries.pytoMODULE_TARGETSinscripts/mutation_report.py(threshold 0.90 each, real gate — sub-threshold → non-zero exit).tests/unit/test_sql_builder_mutation.py(583 LOC) andtest_nl_queries_mutation.py(855 LOC): both stubserving.semantic_layer.query.{engine,contracts}+src.*helpers before importing under a top-levelservingpackage;nl_queriesrepoints its.sql_guardshim at the real top-level guard so thevalidate_nl_sqlboundary is genuinely exercised.tests/unit/test_mutmut_policy.py: narrow the declared-only serving surface to just auth (manager/key_rotation) — the remaining B3 item.This follows the proven pattern from
sql_guard/masking/rate_limiter(B1, #114).Verification
sql_builder96.0% (killed 167/174),nl_queries94.4% (killed 323/342); survivors are documented equivalents.mutation.ymldispatched on this branch — see run linked in checks; both modules gate at ≥90%.Scope
Test/config only — no
src/changes. Next: B3 (authmanager+key_rotationlive).