Skip to content

CLAUDE.md

Latest commit

 

History

History
114 lines (98 loc) · 6.04 KB

CLAUDE.md

File metadata and controls

114 lines (98 loc) · 6.04 KB
Raw

ShapePipe

ShapePipe is a galaxy shape-measurement pipeline for weak-lensing cosmology. It runs the full chain from raw survey images to calibrated shear catalogues — object detection, PSF modelling, and shape measurement — and was used to produce the first UNIONS cosmic-shear release.

The project is now entering a substantial rework of the shape-measurement pipeline, with the near-term goal of a tight loop between the pipeline and image simulations for validation and calibration. Development is by a small team; reproducibility and clarity matter more than breadth.

Environment — the container is the source of truth

ShapePipe is not a stand-alone library: it needs system tools (Source Extractor, PSFEx, WeightWatcher), MPI, and a specific scientific-Python stack. The supported way to get all of that is the container.

  • Dependencies are declared in pyproject.toml — abstract minimums, the single source of truth for what ShapePipe needs — and pinned exactly in uv.lock, the reproducible manifest generated by uv (never hand-edited). System-level tools live in the Dockerfile.
  • Images publish to ghcr.io/cosmostat/shapepipe on every push to an integration branch, in two targets:
    • :<tag> — the dev image: the full stack plus interactive tooling and the test / lint / doc extras.
    • :<tag>-runtime — a slim image for batch jobs and downstream FROM clauses.
  • On a cluster, run with Apptainer: 
    apptainer build --sandbox shapepipe docker://ghcr.io/cosmostat/shapepipe:develop
apptainer shell --writable shapepipe
cd /app && shapepipe_run -c /app/example/config.ini
  • For development, work inside the dev image — it carries vim, ripgrep, pytest, and the rest. A common pattern is a long-lived --writable Apptainer sandbox with a host clone of the repo bind-mounted in and pip install -e pointed at it, so edits on the host are live inside the container.

Shadow a library build with PYTHONPATH (no image rebuild). When you need the container to run a different build of a pure-Python library than the one baked into its venv — a feature worktree, an unreleased branch, this repo's own src/ against a stale image — prepend the host checkout to PYTHONPATH. Python resolves the prepended path first, so the on-disk version shadows /app/.venv/... without touching the image. This is the local-testing counterpart of a git-ref dependency (e.g. cs_util @ develop in pyproject.toml): the dep change makes CI build the right version; the shadow lets you test that version now, before any rebuild. The recipe:

apptainer exec --bind /n17data,/automnt <image.sif> bash -c \
  "cd <repo-worktree> && \
   PYTHONPATH=/path/to/libfoo-checkout:<repo-worktree>/src \
   python -m pytest <targets> -o addopts='' -q"

Notes: the checkout path is the parent of the importable package dir (the dir containing foo/, not foo/ itself); list several :-separated to stack shadows; -o addopts='' clears pyproject.toml's pytest defaults when a plugin they reference (e.g. pytest-cov) isn't in the image. Use this for a quick verify; land the real fix as the pyproject.toml / uv.lock dep change so CI and the next image agree.

Testing container changes: build remotely, pull locally. Don't apptainer build images on a cluster — quotas are tight and the build is slow. The loop for any change to Dockerfile / pyproject.toml / uv.lock is: edit → push → let GitHub Actions build and publish to GHCR → apptainer pull docker://ghcr.io/cosmostat/shapepipe:<branch>[-runtime] on the cluster → test. Watch the remote build with gh run watch (or gh run list --branch <branch>). The only things that run locally are the pull and the test. On a quota-limited cluster, keep SIFs and Apptainer's scratch off $HOME: point APPTAINER_TMPDIR / APPTAINER_CACHEDIR at a roomy data partition and pull SIFs there.

Full detail: docs/source/installation.md and docs/source/container.md.

Layout

  • src/shapepipe/ — the package (src-layout). modules/ holds the pipeline modules and their *_runner.py wrappers; pipeline/ is execution and file I/O; utilities/; canfar/ is CANFAR/cluster job orchestration. Console entry points (shapepipe_run, summary_run, canfar_*) are defined under [project.scripts].
  • tests/ — the whole test suite, one discovery root: module/ (per-module unit/property/integration tests), unit/ (structural), science/ (fast guardrails), cluster/ (candide-only), helpers/ (shared library code). See tests/README.md.
  • example/ — a runnable example pipeline (example/config.ini) on a single CFIS tile; doubles as the CI smoke test.
  • scripts/ — shell / Python / notebook helpers (sh/, python/, jupyter/), symlinked onto $PATH inside the image.
  • docs/ — Sphinx sources; the API docs are generated from docstrings.

Development workflow

  • develop is the integration branch — open PRs against it. main / master are release branches.
  • Tests run with pytest. CI runs them inside the dev image, so the suite exercises exactly what ships — run them the same way, in the dev container.
  • CI (.github/workflows/deploy-image.yml): every PR and push builds the image and runs the test suite + the example pipeline + binary smokes; pushes to develop / main / master additionally publish the images. API docs deploy from master (cd.yml).
  • Style: PEP 8; numpydoc docstrings on public modules, classes, and methods. Match the surrounding code.

Project knowledge — felt

This repo carries a version-controlled .felt/ store: the team's shared notes on decisions, architecture, and plans, written as markdown "fibers" (readable directly; the felt CLI indexes and searches them). It's where the why lives, for humans and AI agents alike. Start at the shapepipe root fiber; the container/CI architecture and the in-progress test-suite work each have their own. When you make a durable decision or learn something worth keeping, leave a fiber so the next person finds it.