Tests: CDA cassette migration (unit tier)#294
Open
jeandet wants to merge 59 commits into
Open
Conversation
Captures decisions on UV adoption, hatchling build backend, ruff/basedpyright tooling, and three-tier test strategy (unit/contract/e2e). Sequences the work as 17 small PRs ending with a mass reformat. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Per-task implementation plan for the first PR of the modernisation effort. Covers pyproject.toml updates, uv.lock generation, CI/RTD switch to uv, deletion of requirements*.txt / tox.ini / setup.cfg, and developer-doc updates to drop the PYTHONPATH=. pattern. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add SPEASY_CORE_HTTP_REWRITE_RULES env to PRs.yml non-3.10 pytest step (previously only on push/scheduled tests.yml — would have hit a non-existent server on PR builds for non-3.10 matrix entries). - Add --with wheel to PRs.yml build step for parity with tests.yml. - Scope flake8 to 'speasy tests' in both workflows (matches Makefile lint target). Avoids silently broadening lint to docs/conf.py and removes the .venv exclusion workaround that was needed when flake8 ran from repo root.
Without UV_PROJECT_ENVIRONMENT, uv creates .venv/ inside the project and RTD's sphinx step (which calls $READTHEDOCS_VIRTUALENV_PATH/bin/python directly) fails with 'python: not found'. Point uv at RTD's venv so the install lands where the runner looks for it.
Classified via devtools/apply_test_markers.py: - 12 files marked unit (pure-logic, no network) - 19 files marked contract (real-server, will be migrated to cassettes in PRs 4-9) Reclassifications during manual review: - test_cache.py: contract -> unit (pure cache-logic, no network or speasy provider use) - test_file_access.py: unit -> contract (uses HTTP via any_loc_open against live servers) test_wasm.py was manually adjusted to place pytestmark at module level (the file's body lives inside a try/except ImportError block, so the script's naive insertion landed at wrong indentation).
…le path and sample across the inventory The flat_inventories.generic_archive lookup uses module attribute access on an instance, not a submodule import. Also, the first N parameters in the flat inventory are clustered by mission, so a fixed time range can miss all of them; sample across the full list instead.
- test_e2e_smoke.test_generic_archive: fail loudly if every candidate raises (was silently skipping, defeating the e2e tier's purpose). - pyproject.toml: drop dead --ignore=setup.py from addopts and document the -m unit override semantics so future contributors don't trip on 'pytest tests/test_amda.py' silently collecting nothing. - contract.yml / e2e.yml: add concurrency groups so a manual run can't overlap with a cron run hammering the same upstream servers. - CONTRIBUTING.rst: add a short note explaining the three test tiers and how to invoke each from local dev.
CI failure (blocking): - unit.yml: 'make doctest' was using system Python (no sphinx in scope). Prefix with 'uv run' so make uses the project venv. Reviewer findings: - wasm_tests.yml: pytest tests/test_wasm.py without -m collected 0 tests under the new addopts default (test_wasm.py is contract-marked). Add -m '' to override. - CLAUDE.md: examples like 'uv run pytest tests/test_amda.py' silently collected 0 tests under -m unit default. Replaced with tier-aware examples and added -m '' for the all-tests case. - unit.yml: only sync --group docs on the coverage runner that needs it, not on every matrix entry.
nbsphinx requires the system pandoc binary (not the Python pandoc wrapper that's in the docs dependency group). PR 1's tests.yml had 'sudo apt install -y texlive pandoc' before make doctest; my unit.yml rewrite in PR 2 dropped that line, so the doctest job failed with 'nbsphinx.NotebookError: PandocMissing in examples/AMDA.ipynb'. Restored as a separate apt step on the coverage runner.
The doctest step's examples reference all data providers (cdpp3dview included) and live inventories. The job-level SPEASY_CORE_DISABLED_PROVIDERS='cdpp3dview' makes the inventory tree's cdpp3dview attribute missing during doctest, surfacing as 'types.SimpleNamespace object has no attribute cdpp3dview' and a chain of NameErrors for variables defined in earlier doctest blocks. Original tests.yml overrode SPEASY_CORE_DISABLED_PROVIDERS="" on the combined pytest+doctest step, plus set HTTP_REWRITE_RULES (re-routes the placeholder URL used in some examples to LPP's mirror) and USER_AGENT. My PR 2 rewrite dropped the env block; restoring it on the doctest step.
Pandas now prints its public name in type() repr ('pandas.DataFrame')
rather than the internal module path ('pandas.core.frame.DataFrame').
The user/numpy.rst doctest was written against the old form.
Surfaced now that uv.lock pins a recent pandas; pip-installed envs
were getting older pandas where the old form still applied.
Sister script to the new before_record_response callback in conftest. Scrubs the same patterns (Set-Cookie response headers, 32-char hex auth.php response bodies) from cassettes that were recorded before the callback existed. Idempotent — safe to re-run. Used once to retroactively clean the existing AMDA + CSA cassettes on the modernisation/pr3-mocking-infra branch. Future recordings are automatically scrubbed at record time by the conftest callback.
jeandet
force-pushed
the
modernisation/pr5-cda-cassettes
branch
from
9cfa61f to
93ff138
Compare
The cassette hosting at sciqlop.lpp.polytechnique.fr/data/speasy_cassettes/ is now public-read. Cassettes are content-addressed by sha256 — URLs are unguessable for outsiders and any tampering is caught on download via the existing hash verification in _fetch_cassette. Practical benefits: - Fork PRs can run the cassette-replaying unit tier (previously blocked: GitHub Actions doesn't expose repo secrets to fork PRs). - New contributors need no credential setup to run the tests. - CI workflows lose the SPEASY_CASSETTE_FETCH_USER/PASSWORD env injection (no longer needed). Cassettes are still scrubbed (Set-Cookie response headers and AMDA auth.php hash response bodies) by the before_record_response callback in vcr_config, so no session/credential material reaches the cassette content itself.
jeandet
force-pushed
the
modernisation/pr5-cda-cassettes
branch
2 times, most recently
from
13874fa to
f04e51f
Compare
The conftest is imported by every pytest run, including the wasm_tests.yml workflow that uses a minimal Python env (it installs pytest-pyodide but not the project's full dev group). numpy is only needed by the speasy_variable_factory fixture's _make closure; importing it at module level breaks wasm test collection with ModuleNotFoundError. Move the import inside the fixture body so non-unit-tier callers don't need it.
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## main #294 +/- ##
===========================================
- Coverage 85.10% 74.45% -10.66%
===========================================
Files 69 82 +13
Lines 4834 5092 +258
Branches 668 693 +25
===========================================
- Hits 4114 3791 -323
- Misses 479 1063 +584
+ Partials 241 238 -3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
jeandet
force-pushed
the
modernisation/pr5-cda-cassettes
branch
from
a1c3a57 to
45bc113
Compare
Same rationale as numpy: the wasm_tests.yml workflow uses a minimal Python env that doesn't install requests (only pyodide-py and pytest-pyodide). Importing requests at conftest module level breaks wasm test collection. Moved inside _fetch_cassette so non-fetching callers don't need it.
jeandet
force-pushed
the
modernisation/pr5-cda-cassettes
branch
from
45bc113 to
e172a47
Compare
The wasm test runs in a minimal Python env that doesn't install 'requests'. The conftest's pytest_configure hook calls _fetch_cassette (which lazily imports requests); on this env the import fails and INTERNALERROR's the run. Wasm tests don't replay cassettes anyway — just skip the fetch via the existing --no-cassette-fetch flag.
Generated by devtools/publish_cassettes.py from cassettes recorded
against live amda.irap.omp.eu. Each entry maps a relative path under
tests/cassettes/ to the sha256 of the cassette's uncompressed YAML
content. The .yaml.gz files for these hashes need to be uploaded by
a maintainer to:
https://sciqlop.lpp.polytechnique.fr/data/speasy_cassettes/
via rsync from .publish_staging/. After upload, the conftest fetch
hook will download cassettes on demand for unit-tier test runs that
hit AMDA code paths.
Total: 90.6 MB uncompressed, 10.9 MB compressed across 22 cassettes.
The original recording captured a HEAD if-modified-since request to spdf.gsfc.nasa.gov/pub/catalogs/all.xml because Speasy's diskcache was already warm at recording time. On replay with a clean cache, Speasy issues a full GET which has no matching cassette entry, failing with CannotOverwriteExistingCassetteException. Re-recorded with SPEASY_CACHE_PATH set to a fresh tempdir (the pattern PR 5 established) so the cassette captures the GET path. The 50 MB cassette content is the full AMDA observatory tree.
The scrub_existing_cassettes.py one-off removed Set-Cookie response headers and replaced AMDA auth.php response bodies (32-char hex session tokens) with <SCRUBBED>. Content-addressed cassette hashes changed for the affected files; manifest updated to match.
The earlier re-record only cleared SPEASY_CACHE_PATH but kept the populated SPEASY_INDEX_PATH (Speasy's index path is separate from the cache). With a populated index, Speasy's catalog-loader sees the inventory as 'already known' and issues HEAD if-modified-since revalidations instead of the full GETs that a fresh-state install does. Result: cassette captured HEAD only; CI (fresh state) issues GET and fails to match. Re-recorded with both SPEASY_CACHE_PATH and SPEASY_INDEX_PATH set to fresh tempdirs so the catalog GETs are captured.
- Skip 4 tests whose cassettes blow the 150 MB unit-tier budget (full inventory fetch, FEEPS electron intensity, MMS FGM virtual-parameter fallback, EQ_PP_MAM via inventory) - all keep equivalent live coverage in tests/test_cdaweb_contract.py. - Drop the MMS2_SCM_SRVY_L2_SCSRVY case from both ddt-driven tests (MMS2 SCM survey returns ~76 MB per request because the API serves the day-aligned CDF chunk regardless of the requested time window). - Drop the MMS1_SCM_BRST_L2_SCB case from both ddt-driven tests (10 min of burst SCM is a ~380 MB CDF). - Shrink large API/FILE windows where it preserves the test's intent (PSP ISOIS, MMS FGM sanitised). - Skip ConcurrentRequests.test_get_variable: thread-pool requests are not reliably intercepted by VCR, breaking replay. - Skip test_get_cluster_fgm_data: upstream-data assumption no longer holds (CDA now returns Cluster C1 FGM data for the 2018-03/2016-03 windows the test asserted None on).
Four daily-cron probes covering the live CDA paths whose cassettes were dropped from the unit tier for size reasons (full inventory fetch and the FEEPS electron-intensity request) plus two cheap smoke checks (short THA fetch, inventory dataset presence).
Mocks speasy.data_providers.cda.http.get to return a 500 so the unit tier verifies that a CDA server error surfaces as CdaWebException rather than being silently swallowed.
jeandet
force-pushed
the
modernisation/pr5-cda-cassettes
branch
from
e172a47 to
192958b
Compare
|
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fifth PR of the modernisation effort (spec:
docs/superpowers/specs/2026-05-08-speasy-modernisation-design.md, plan:docs/superpowers/plans/2026-05-10-pr5-cda-cassettes.md). Second per-provider cassette migration after AMDA.Stacked on PR #293 (AMDA cassettes), which stacks on #292 → #291 → #290. Until each predecessor merges, this PR's diff includes them all.
What this PR does
tests/test_cdaweb.py(24 tests) from contract tier to unit tier with cassette-backed replay.cdaweb.gsfc.nasa.gov, uploads them tohttps://sciqlop.lpp.polytechnique.fr/data/speasy_cassettes/, updatestests/cassettes_manifest.json.tests/test_cdaweb_contract.py(4 daily-cron drift probes).tests/test_cdaweb_failures.py(1 failure-path unit test forCdaWebExceptionpropagation).SPEASY_CORE_HTTP_REWRITE_RULEStounit.ymlenv so cassette URLs match the placeholder-rewrite behaviour used by CI.Production fix included
speasy/core/http.py:Response.urlpreviously crashed undervcrpyreplay becauseVCRHTTPResponsedoes not set aurlattribute. One-linetry/except AttributeErrorfallback preserves original behaviour in production and prevents the AttributeError in cassette-replay paths. Without this, 15 of 24 CDA tests fail under replay.Tests dropped from unit tier (kept on contract)
To stay within the per-PR cassette-size budget, six heavyweight or non-cassette-friendly tests are skip-marked or trimmed. All have equivalent or surrogate coverage on the daily contract tier:
test_can_get_full_inventory_without_proxytest_full_inventory_fetch_finds_at_least_47000_parametersprobetest_get_products_with_percent_in_nametest_wrong_time_dependency_axistest_feeps_electron_intensity_returns_dataprobetest_get_virtual_parameter_always_falls_back_to_apitest_get_cluster_fgm_dataNone). Properly skip-marked; was silently failing CI since PR #290.ConcurrentRequests.test_get_variablePlus two
ddt-driven cases pruned (MMS SCM products returning 76 MB / 380 MB chunks) and two time windows shrunk (semantics preserved).Net effect
Test plan
unit.ymlgreen — CDA tests replay from cassettes fetched at session startcontract.yml(manually triggered) — 4 CDA probes pass against real upstreamSPEASY_CORE_HTTP_REWRITE_RULESis set in CI job env (added in this PR)