Problem
Docstrings frequently carry executable examples (>>> foo(2) / 4). These rot silently: a signature change or behavior tweak leaves the example wrong, and nothing catches it because the regular test suite doesn't execute docstring examples. Forge already enforces docstring presence (ruff D-rules) and correctness of Args/Returns (verify_docstrings) and coverage (docstring_coverage), but does not execute the examples those docstrings contain.
pytest --doctest-modules runs every >>> example as a test. Wiring it as a forge step closes the loop: docstring examples become verified, not decorative.
Reference design
CI runs it as a non-blocking informational step, then validates results in a later gate:
pytest --doctest-continue-on-failure --doctest-modules src/ \
| tee ./code_health/doctest.log || true
--doctest-continue-on-failure means one broken example doesn't mask the rest — the full list of failures lands in the log in a single pass.
Components to ship
| Component |
Purpose |
Approx LOC |
step_doctest in forge.precommit |
Runs pytest --doctest-modules over configured paths, writes code_health/doctest.log |
small (~40) |
[tool.forge.doctest] config |
paths (default ["src"]), blocking (default false) |
tiny |
Imports / dependencies
pytest>=8.0 — already a forge extra.
- No new dependency.
--doctest-modules is built into pytest.
How consumers use it
Pre-commit (opt-in, non-blocking default)
[tool.forge.doctest]
paths = ["src"]
blocking = false
Absent section → step self-skips. Present → runs doctests over paths, writes the log, advisory by default.
CI
- name: Doctests
continue-on-error: true
run: |
pytest --doctest-continue-on-failure --doctest-modules src/ \
| tee ./code_health/doctest.log || true
- name: Validate Doctest Results
if: always()
run: |
# parse code_health/doctest.log; fail the job if any example failed
# (the earlier step is continue-on-error so the full list is captured first)
The two-step pattern (run continue-on-error → validate-and-gate) lets the log capture every failure before the gate decides pass/fail, instead of bailing on the first.
Behavioral guarantees
- Full failure list, not first-fail.
--doctest-continue-on-failure guarantees the log enumerates every broken example in one run.
- Path-scoped. Only
[tool.forge.doctest].paths are scanned; tests/ excluded by default (test files rarely carry doctests).
- Skip when not opted in. No
[tool.forge.doctest] section → step exits 0 silently.
Acceptance criteria
step_doctest in forge-precommit, gated on [tool.forge.doctest] presence.
- Runs
pytest --doctest-modules --doctest-continue-on-failure over configured paths.
- Writes
code_health/doctest.log.
blocking config key (default false) controls whether failures refuse the commit.
- Tests cover: passing doctest, failing doctest (blocking vs non-blocking), no-config skip, path scoping.
Out of scope
- Doctest namespace fixtures /
conftest-style doctest setup (consumers configure via pytest's own [tool.pytest.ini_options]).
- Running doctests inside the smart-test depth tiers — that's the smart-test issue's concern.
Related
Problem
Docstrings frequently carry executable examples (
>>> foo(2)/4). These rot silently: a signature change or behavior tweak leaves the example wrong, and nothing catches it because the regular test suite doesn't execute docstring examples. Forge already enforces docstring presence (ruff D-rules) and correctness of Args/Returns (verify_docstrings) and coverage (docstring_coverage), but does not execute the examples those docstrings contain.pytest --doctest-modulesruns every>>>example as a test. Wiring it as a forge step closes the loop: docstring examples become verified, not decorative.Reference design
CI runs it as a non-blocking informational step, then validates results in a later gate:
pytest --doctest-continue-on-failure --doctest-modules src/ \ | tee ./code_health/doctest.log || true--doctest-continue-on-failuremeans one broken example doesn't mask the rest — the full list of failures lands in the log in a single pass.Components to ship
step_doctestinforge.precommitpytest --doctest-modulesover configured paths, writescode_health/doctest.log[tool.forge.doctest]configpaths(default["src"]),blocking(defaultfalse)Imports / dependencies
pytest>=8.0— already a forge extra.--doctest-modulesis built into pytest.How consumers use it
Pre-commit (opt-in, non-blocking default)
Absent section → step self-skips. Present → runs doctests over
paths, writes the log, advisory by default.CI
The two-step pattern (run continue-on-error → validate-and-gate) lets the log capture every failure before the gate decides pass/fail, instead of bailing on the first.
Behavioral guarantees
--doctest-continue-on-failureguarantees the log enumerates every broken example in one run.[tool.forge.doctest].pathsare scanned; tests/ excluded by default (test files rarely carry doctests).[tool.forge.doctest]section → step exits 0 silently.Acceptance criteria
step_doctestinforge-precommit, gated on[tool.forge.doctest]presence.pytest --doctest-modules --doctest-continue-on-failureover configured paths.code_health/doctest.log.blockingconfig key (default false) controls whether failures refuse the commit.Out of scope
conftest-style doctest setup (consumers configure via pytest's own[tool.pytest.ini_options]).Related
forge.verify_docstring_coverage(docstring_coveragestep) — presence %; this is execution.