Tests: introduce unit/contract/e2e markers + split CI#291
Open
jeandet wants to merge 28 commits into
Open
Conversation
Captures decisions on UV adoption, hatchling build backend, ruff/basedpyright tooling, and three-tier test strategy (unit/contract/e2e). Sequences the work as 17 small PRs ending with a mass reformat. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per-task implementation plan for the first PR of the modernisation effort. Covers pyproject.toml updates, uv.lock generation, CI/RTD switch to uv, deletion of requirements*.txt / tox.ini / setup.cfg, and developer-doc updates to drop the PYTHONPATH=. pattern. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add SPEASY_CORE_HTTP_REWRITE_RULES env to PRs.yml non-3.10 pytest step (previously only on push/scheduled tests.yml — would have hit a non-existent server on PR builds for non-3.10 matrix entries). - Add --with wheel to PRs.yml build step for parity with tests.yml. - Scope flake8 to 'speasy tests' in both workflows (matches Makefile lint target). Avoids silently broadening lint to docs/conf.py and removes the .venv exclusion workaround that was needed when flake8 ran from repo root.
Without UV_PROJECT_ENVIRONMENT, uv creates .venv/ inside the project and RTD's sphinx step (which calls $READTHEDOCS_VIRTUALENV_PATH/bin/python directly) fails with 'python: not found'. Point uv at RTD's venv so the install lands where the runner looks for it.
Classified via devtools/apply_test_markers.py: - 12 files marked unit (pure-logic, no network) - 19 files marked contract (real-server, will be migrated to cassettes in PRs 4-9) Reclassifications during manual review: - test_cache.py: contract -> unit (pure cache-logic, no network or speasy provider use) - test_file_access.py: unit -> contract (uses HTTP via any_loc_open against live servers) test_wasm.py was manually adjusted to place pytestmark at module level (the file's body lives inside a try/except ImportError block, so the script's naive insertion landed at wrong indentation).
…le path and sample across the inventory The flat_inventories.generic_archive lookup uses module attribute access on an instance, not a submodule import. Also, the first N parameters in the flat inventory are clustered by mission, so a fixed time range can miss all of them; sample across the full list instead.
- test_e2e_smoke.test_generic_archive: fail loudly if every candidate raises (was silently skipping, defeating the e2e tier's purpose). - pyproject.toml: drop dead --ignore=setup.py from addopts and document the -m unit override semantics so future contributors don't trip on 'pytest tests/test_amda.py' silently collecting nothing. - contract.yml / e2e.yml: add concurrency groups so a manual run can't overlap with a cron run hammering the same upstream servers. - CONTRIBUTING.rst: add a short note explaining the three test tiers and how to invoke each from local dev.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| if any("import pytest" in ln for ln in lines): | ||
| block = [f"\n{MARKER_LINE_PREFIX}{tier}\n", "\n"] | ||
| lines[insert_at:insert_at] = block | ||
| path.write_text("".join(lines)) |
There was a problem hiding this comment.
Pull request overview
This PR introduces a three-tier pytest test strategy (unit/contract/e2e) and restructures CI to run each tier on an appropriate cadence, while also folding pytest configuration into pyproject.toml and updating dev tooling/docs for the UV-based workflow (stacked on the UV foundation PR).
Changes:
- Register
unit,contract,e2epytest markers (defaulting local/CIpytestruns to-m unit) and mechanically mark existing test modules. - Add a small
tests/test_e2e_smoke.pysuite to exercisespz.get_dataend-to-end across active providers. - Split GitHub Actions workflows into
unit.yml,contract.yml,e2e.yml, andlint.yml; remove legacy workflows and legacy config files (pytest.ini,tox.ini,setup.cfg, requirements files).
Reviewed changes
Copilot reviewed 51 out of 53 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tox.ini | Removed legacy tox configuration (superseded by UV/CI matrix). |
| setup.cfg | Removed obsolete packaging/flake8 aliases config. |
| requirements.txt | Removed legacy runtime requirements list (moved to pyproject/uv). |
| requirements_dev.txt | Removed legacy dev requirements list (moved to pyproject dependency groups). |
| docs/requirements.txt | Removed legacy docs requirements list (moved to pyproject dependency groups). |
| pytest.ini | Removed legacy pytest config (migrated to pyproject.toml). |
| pyproject.toml | Adds pytest marker registration + default -m unit; includes dependency-groups and Python floor bump from stacked PR. |
| devtools/apply_test_markers.py | One-shot script used to apply module-level markers to existing tests. |
| tests/test_amda.py | Adds module-level pytest marker. |
| tests/test_amda_catalog.py | Adds module-level pytest marker. |
| tests/test_amda_parameter.py | Adds module-level pytest marker. |
| tests/test_amda_timetable.py | Adds module-level pytest marker. |
| tests/test_cache.py | Adds module-level pytest marker. |
| tests/test_cdaweb.py | Adds module-level pytest marker. |
| tests/test_cdpp3dview.py | Adds module-level pytest marker. |
| tests/test_codecs.py | Adds module-level pytest marker. |
| tests/test_common.py | Adds module-level pytest marker. |
| tests/test_config_module.py | Adds module-level pytest marker. |
| tests/test_csa.py | Adds module-level pytest marker. |
| tests/test_dataset.py | Adds module-level pytest marker. |
| tests/test_datetimerange.py | Adds module-level pytest marker. |
| tests/test_direct_archive_downloader.py | Adds module-level pytest marker. |
| tests/test_file_access.py | Adds module-level pytest marker. |
| tests/test_filtering.py | Adds module-level pytest marker. |
| tests/test_hapi.py | Adds module-level pytest marker. |
| tests/test_hapi_codecs.py | Adds module-level pytest marker. |
| tests/test_http_module.py | Adds module-level pytest marker. |
| tests/test_inventories.py | Adds module-level pytest marker. |
| tests/test_proxy.py | Adds module-level pytest marker. |
| tests/test_resampling.py | Adds module-level pytest marker. |
| tests/test_speasy.py | Adds module-level pytest marker. |
| tests/test_speasy_catalog.py | Adds module-level pytest marker. |
| tests/test_speasy_timetable.py | Adds module-level pytest marker. |
| tests/test_speasy_variable.py | Adds module-level pytest marker. |
| tests/test_sscweb.py | Adds module-level pytest marker. |
| tests/test_uiowa_eph_tool.py | Adds module-level pytest marker. |
| tests/test_url_utils.py | Adds module-level pytest marker. |
| tests/test_wasm.py | Adds module-level pytest marker (affects how wasm CI invokes pytest). |
| tests/test_zzz_disable_ws.py | Adds module-level pytest marker. |
| tests/test_e2e_smoke.py | New end-to-end smoke test module, marked e2e. |
| .github/workflows/tests.yml | Removed legacy combined CI workflow. |
| .github/workflows/PRs.yml | Removed legacy PR-only CI workflow. |
| .github/workflows/unit.yml | New unit-tier workflow + coverage upload + release-check job. |
| .github/workflows/contract.yml | New scheduled contract-tier workflow (real upstream probes). |
| .github/workflows/e2e.yml | New scheduled/manual e2e-tier workflow (full OS×Python matrix). |
| .github/workflows/lint.yml | New flake8 lint workflow. |
| Makefile | Updates test/coverage/install/readme targets for UV workflow. |
| CONTRIBUTING.rst | Documents UV workflow + tiered test commands. |
| CLAUDE.md | Updates developer commands for UV (needs alignment with new default -m unit). |
| .readthedocs.yaml | Switches RTD build/install to UV dependency groups + lockfile. |
| docs/superpowers/specs/2026-05-08-speasy-modernisation-design.md | Adds the overarching modernization design spec (test tiers/tooling/CI shape). |
| docs/superpowers/plans/2026-05-08-pr1-foundation-uv-py310.md | Adds the PR1 execution plan document (included due to stacking). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1
to
+4
| import pytest | ||
|
|
||
| pytestmark = pytest.mark.contract | ||
|
|
Comment on lines
+16
to
+18
| uv run pytest # all tests | ||
| uv run pytest tests/test_amda.py # single test file | ||
| uv run pytest -k "test_name" # single test by name |
Comment on lines
+32
to
+40
| - name: Install dependencies | ||
| run: uv sync --group dev --group docs | ||
| - name: Run unit tests | ||
| run: uv run pytest -m unit | ||
| - name: Run unit tests with coverage (one runner) | ||
| if: matrix.python-version == '3.13' && matrix.os == 'ubuntu-latest' | ||
| run: | | ||
| uv run pytest -m unit --cov=speasy --cov-config=.coveragerc --cov-report=xml | ||
| make doctest |
CI failure (blocking): - unit.yml: 'make doctest' was using system Python (no sphinx in scope). Prefix with 'uv run' so make uses the project venv. Reviewer findings: - wasm_tests.yml: pytest tests/test_wasm.py without -m collected 0 tests under the new addopts default (test_wasm.py is contract-marked). Add -m '' to override. - CLAUDE.md: examples like 'uv run pytest tests/test_amda.py' silently collected 0 tests under -m unit default. Replaced with tier-aware examples and added -m '' for the all-tests case. - unit.yml: only sync --group docs on the coverage runner that needs it, not on every matrix entry.
nbsphinx requires the system pandoc binary (not the Python pandoc wrapper that's in the docs dependency group). PR 1's tests.yml had 'sudo apt install -y texlive pandoc' before make doctest; my unit.yml rewrite in PR 2 dropped that line, so the doctest job failed with 'nbsphinx.NotebookError: PandocMissing in examples/AMDA.ipynb'. Restored as a separate apt step on the coverage runner.
The doctest step's examples reference all data providers (cdpp3dview included) and live inventories. The job-level SPEASY_CORE_DISABLED_PROVIDERS='cdpp3dview' makes the inventory tree's cdpp3dview attribute missing during doctest, surfacing as 'types.SimpleNamespace object has no attribute cdpp3dview' and a chain of NameErrors for variables defined in earlier doctest blocks. Original tests.yml overrode SPEASY_CORE_DISABLED_PROVIDERS="" on the combined pytest+doctest step, plus set HTTP_REWRITE_RULES (re-routes the placeholder URL used in some examples to LPP's mirror) and USER_AGENT. My PR 2 rewrite dropped the env block; restoring it on the doctest step.
Pandas now prints its public name in type() repr ('pandas.DataFrame')
rather than the internal module path ('pandas.core.frame.DataFrame').
The user/numpy.rst doctest was written against the old form.
Surfaced now that uv.lock pins a recent pandas; pip-installed envs
were getting older pandas where the old form still applied.
|
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## main #291 +/- ##
===========================================
- Coverage 85.10% 61.87% -23.23%
===========================================
Files 69 82 +13
Lines 4834 5089 +255
Branches 668 693 +25
===========================================
- Hits 4114 3149 -965
- Misses 479 1759 +1280
+ Partials 241 181 -60
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This was referenced May 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second PR of the modernisation effort (spec:
docs/superpowers/specs/2026-05-08-speasy-modernisation-design.md, plan:docs/superpowers/plans/2026-05-08-pr2-test-markers.md).Stacked on PR #290 (UV foundation). Until that one merges, this diff will include its commits too — please review/merge #290 first, then this rebases cleanly onto main.
What this PR does
pyproject.toml:unit,contract,e2e. Defaultpytestruns only the unit tier (addopts = "-m unit").pytestmark = pytest.mark.<tier>usingdevtools/apply_test_markers.py(committed for reviewer audit).tests/test_e2e_smoke.py: 5 canonical end-to-end tests, one per active provider (AMDA, CDA, CSA, SSC, GenericArchive).tests.yml+PRs.yml→unit.yml(push to main + PR, full matrix, runs-m unit, includes a release-check job).contract.yml(cron daily, single runner, runs-m contract).e2e.yml(cron weekly + workflow_dispatch, full matrix, runs-m e2e).lint.yml(push to main + PR, single runner, flake8).pytest.iniinto[tool.pytest.ini_options]inpyproject.toml; deletespytest.ini.Test classification (488 unit + 342 contract + 5 e2e = 835)
Reclassified from the plan's heuristic during implementation:
tests/test_cache.py: contract → unit (pure cache-logic against tempdir, no network).tests/test_file_access.py: unit → contract (hits live HTTP servers).Key design choices
addopts = -m unitkeeps local dev fast. Documented inpyproject.tomlandCONTRIBUTING.rst. To run other tiers:pytest -m contract,pytest -m e2e,pytest -m ''.concurrency:groups oncontract.ymlande2e.ymlprevent overlapping runs from hammering upstream servers.Test plan
unit.ymlgreen across Linux/macOS/Windows × Python 3.10–3.14lint.ymlgreenunit.ymlsucceedscontract.ymlvia Actions → confirm runs and reportse2e.ymlvia Actions → confirm 5 tests pass on Linux × 3.13 (other matrix slots will exercise on the weekly cron)uv run pytestruns unit tier only, completes in ~30 s