Skip to content

feat: doctest pre-commit step + CI integration #5

@misnaej

Description

@misnaej

Problem

Docstrings frequently carry executable examples (>>> foo(2) / 4). These rot silently: a signature change or behavior tweak leaves the example wrong, and nothing catches it because the regular test suite doesn't execute docstring examples. Forge already enforces docstring presence (ruff D-rules) and correctness of Args/Returns (verify_docstrings) and coverage (docstring_coverage), but does not execute the examples those docstrings contain.

pytest --doctest-modules runs every >>> example as a test. Wiring it as a forge step closes the loop: docstring examples become verified, not decorative.

Reference design

CI runs it as a non-blocking informational step, then validates results in a later gate:

pytest --doctest-continue-on-failure --doctest-modules src/ \
    | tee ./code_health/doctest.log || true

--doctest-continue-on-failure means one broken example doesn't mask the rest — the full list of failures lands in the log in a single pass.

Components to ship

Component Purpose Approx LOC
step_doctest in forge.precommit Runs pytest --doctest-modules over configured paths, writes code_health/doctest.log small (~40)
[tool.forge.doctest] config paths (default ["src"]), blocking (default false) tiny

Imports / dependencies

  • pytest>=8.0 — already a forge extra.
  • No new dependency. --doctest-modules is built into pytest.

How consumers use it

Pre-commit (opt-in, non-blocking default)

[tool.forge.doctest]
paths = ["src"]
blocking = false

Absent section → step self-skips. Present → runs doctests over paths, writes the log, advisory by default.

CI

- name: Doctests
  continue-on-error: true
  run: |
    pytest --doctest-continue-on-failure --doctest-modules src/ \
      | tee ./code_health/doctest.log || true

- name: Validate Doctest Results
  if: always()
  run: |
    # parse code_health/doctest.log; fail the job if any example failed
    # (the earlier step is continue-on-error so the full list is captured first)

The two-step pattern (run continue-on-error → validate-and-gate) lets the log capture every failure before the gate decides pass/fail, instead of bailing on the first.

Behavioral guarantees

  • Full failure list, not first-fail. --doctest-continue-on-failure guarantees the log enumerates every broken example in one run.
  • Path-scoped. Only [tool.forge.doctest].paths are scanned; tests/ excluded by default (test files rarely carry doctests).
  • Skip when not opted in. No [tool.forge.doctest] section → step exits 0 silently.

Acceptance criteria

  • step_doctest in forge-precommit, gated on [tool.forge.doctest] presence.
  • Runs pytest --doctest-modules --doctest-continue-on-failure over configured paths.
  • Writes code_health/doctest.log.
  • blocking config key (default false) controls whether failures refuse the commit.
  • Tests cover: passing doctest, failing doctest (blocking vs non-blocking), no-config skip, path scoping.

Out of scope

  • Doctest namespace fixtures / conftest-style doctest setup (consumers configure via pytest's own [tool.pytest.ini_options]).
  • Running doctests inside the smart-test depth tiers — that's the smart-test issue's concern.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    ci-testingCI / test infrastructurefeatureNew capabilitytier-3-standardNormal features / refactors

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions