Pgpkgmigrations

.github/copilot-instructions.md

@@ -2,3 +2,6 @@
 
 See `CLAUDE.md` for comprehensive project instructions, architecture, and workflows.
 See `AGENTS.md` for specialized agent definitions (sql-developer, migration-engineer, loader-developer).
+
+When migration workflows, script entry points, or developer commands change, update `CLAUDE.md`, `AGENTS.md`, and any relevant files under `.github/instructions/` in the same change.
+Use `uv` for Python execution, dependency installation, and standalone helper scripts; avoid direct `pip` commands.

.github/instructions/migrations.instructions.md

@@ -7,7 +7,7 @@ applyTo: "src/pgstac/migrations/**"
 These files are **generated** — see CLAUDE.md "Migration Process" for the full workflow.
 
 - **DO NOT** create, edit, or hand-modify migration files
-- Base (`pgstac.X.Y.Z.sql`) = full schema at that version
-- Incremental (`pgstac.X.Y.Z-A.B.C.sql`) = upgrade diff
+- Base (`pgstac--X.Y.Z.sql`) = full schema at that version
+- Incremental (`pgstac--X.Y.Z--A.B.C.sql`) = upgrade diff
 - Staged (`*.sql.staged`) = needs review before removing `.staged` suffix
 - Test: `scripts/test --migrations`

.github/instructions/pypgstac.instructions.md

@@ -7,6 +7,9 @@ applyTo: "src/pypgstac/**"
 See CLAUDE.md "pypgstac Loader Internals" for patterns. See AGENTS.md "loader-developer" for critical rules.
 
 - Uses psycopg v3 (not psycopg2), orjson (not json), tenacity, plpygis, fire
+- `pypgstac migrate` is a thin wrapper over `pgstac-migrate`; put new migration runtime behavior in `src/pgstac-migrate/`, not `src/pypgstac/src/pypgstac/migrate.py`
+- `src/pypgstac/pyproject.toml` keeps a local `[tool.uv.sources]` override for `pgstac-migrate`, while `pgpkg` resolves from PyPI
+- In Docker-backed dev runs, `scripts/runinpypgstac` can mount a local `pgpkg` checkout at `/pgpkg` and export `PGPKG_REPO_DIR` so container scripts can force that checkout when needed
 - Materialize generators before retry boundaries
 - Query `partition_sys_meta` (live VIEW), never `partitions` (stale MATERIALIZED VIEW)
 - Test: `scripts/runinpypgstac --build test --pypgstac`

.github/instructions/scripts.instructions.md

@@ -7,6 +7,10 @@ applyTo: "scripts/**"
 See CLAUDE.md "Development Workflow" for usage. All scripts require the Docker compose environment.
 
 - `runinpypgstac` is the foundation — most scripts delegate to it
+- `runinpypgstac` uses the published-package path by default; set `PGPKG_LOCAL_REPO_DIR` to mount a local `pgpkg` checkout at `/pgpkg` when you need an override
 - `scripts/container-scripts/` contains the in-container script payload copied into the pypgstac image; keep host wrappers in `scripts/`
 - `stageversion` modifies version files AND generates migrations — see CLAUDE.md "Migration Process"
+- `scripts/container-scripts/stageversion` and `scripts/container-scripts/makemigration` now shell through `pgpkg` inside the container rather than assembling/diffing SQL directly
+- Set `PGPKG_LOCAL_REPO_DIR` on the host when you need to force a local pgpkg checkout for `stageversion`, `makemigration`, or related container-script testing
+- Tagged releases run `.github/workflows/release.yml`, which publishes both `pypgstac` and `pgstac-migrate` to PyPI via the GitHub `pypi` environment; PyPI trusted publishers must exist for both projects
 - DO NOT run `stageversion` without understanding its side effects

.github/workflows/release.yml

@@ -217,12 +217,33 @@ jobs:
         with:
           python-version: "3.x"
       - name: Install build
-        working-directory: /home/runner/work/pgstac/pgstac/src/pypgstac
+        working-directory: src/pypgstac
         run: pip install build
       - name: Build
-        working-directory: /home/runner/work/pgstac/pgstac/src/pypgstac
+        working-directory: src/pypgstac
         run: python -m build
       - name: Publish to PyPI
         uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # release/v1
         with:
-          packages-dir: /home/runner/work/pgstac/pgstac/src/pypgstac/dist
+          packages-dir: src/pypgstac/dist
+
+  releasepgstacmigratetopypi:
+    name: Release pgstac-migrate to PyPI
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write
+    environment:
+      name: pypi
+      url: https://pypi.org/p/pgstac-migrate
+    steps:
+      - name: Checkout
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - name: Setup uv
+        uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
+      - name: Build
+        working-directory: src/pgstac-migrate
+        run: uvx --from build pyproject-build
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # release/v1
+        with:
+          packages-dir: src/pgstac-migrate/dist

.gitignore

@@ -16,3 +16,6 @@ src/pypgstac/python/pypgstac/*.so
 .plans/
 .env
 src/pgstacrust/target/
+src/pgstac-migrate/dist/
+src/pgstac-migrate/src/pgstac_migrate/migrations.tar.zst
+src/pypgstac/uv.lock

AGENTS.md

@@ -23,9 +23,13 @@ Migration specialist for PgSTAC. See CLAUDE.md "Migration Process" for full work
 ### Quick Reference
 
 1. Edit SQL in `src/pgstac/sql/*.sql`
-2. `scripts/stageversion VERSION` → generates base + incremental `.staged` migration
-3. Review `.staged` file (watch for DROPs, unsafe ALTERs, missing `CREATE OR REPLACE`)
-4. Remove `.staged` suffix → `scripts/test --migrations`
+2. `src/pgstac/pyproject.toml` is the `pgpkg` project config for the SQL + migrations tree
+3. `uv run --directory src/pgstac-migrate pgstac-migrate info|versions|plan` inspects the baked migration artifact during wrapper work
+4. `uv run --directory src/pypgstac pypgstac migrate -- --help` remains a backwards-compatible wrapper over `pgstac-migrate`; put new runtime migration behavior in `src/pgstac-migrate/`, not `src/pypgstac/`
+5. `scripts/stageversion VERSION` → generates canonical `pgstac--VERSION.sql` plus an incremental `.staged` migration; set `PGPKG_LOCAL_REPO_DIR` when `stageversion` or `makemigration` should run against a local pgpkg checkout. The Docker-backed flow mounts that override at `/pgpkg` and exports `PGPKG_REPO_DIR` to the container scripts.
+6. Review `.staged` file (watch for DROPs, unsafe ALTERs, missing `CREATE OR REPLACE`)
+7. Remove `.staged` suffix → `scripts/test --migrations`
+8. Tagged releases publish both `pypgstac` and `pgstac-migrate` to PyPI from `.github/workflows/release.yml`; keep the PyPI trusted publisher registration aligned with the `pypi` environment and workflow path
 
 ### Review Checklist
 

CHANGELOG.md

@@ -9,6 +9,10 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 ## [Unreleased]
 
 ### Added
+- New `pgstac-migrate` package under `src/pgstac-migrate/` with a standalone
+  CLI, Python API, and tests for migration planning and execution.
+- `src/pgstac/pyproject.toml` `tool.pgpkg` project metadata for canonical SQL +
+  migration staging.
 - `scripts/makemigration` host wrapper for the in-container `makemigration` helper.
 - `.env.example` documenting all supported environment variables for local development.
 - All host-facing scripts (`test`, `format`, `migrate`, `server`, `stageversion`,
@@ -31,6 +35,20 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
   ecosystems with grouped update policies).
 
 ### Changed
+- `pypgstac migrate` now delegates runtime migration planning and apply logic to
+  `pgstac-migrate`; `src/pypgstac/src/pypgstac/migrate.py` remains as a
+  compatibility wrapper.
+- Migration filenames are now canonicalized to
+  `pgstac--<version>.sql` / `pgstac--<from>--<to>.sql` in
+  `src/pgstac/migrations/` and `src/pypgstac/src/pypgstac/migrations/`.
+- `scripts/container-scripts/stageversion` and
+  `scripts/container-scripts/makemigration` now shell through `pgpkg`
+  (`uv run --no-project --with "pgpkg>=0.1.1,<0.2"` and
+  `uv run --no-project --with "pgpkg[diff]>=0.1.1,<0.2"`) with optional
+  `PGPKG_REPO_DIR` override support.
+- `scripts/runinpypgstac` now supports a `PGPKG_LOCAL_REPO_DIR` mount override
+  for local pgpkg development while keeping the default flow PyPI-first.
+- Tagged releases now publish the new `pgstac-migrate` package to PyPI alongside `pypgstac` via trusted publishing in `.github/workflows/release.yml`.
 - In-container helper scripts moved from `docker/pypgstac/bin/` to
   `scripts/container-scripts/`; container `PATH` updated accordingly.
 - `docker/pgstac/Dockerfile` and `docker/pypgstac/Dockerfile` base images updated from
@@ -68,6 +86,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 - `flake8`, `black`, and `mypy` removed from dev dependencies.
 
 ### Fixed
+- `scripts/container-scripts/test` now refreshes collation metadata for the
+  `postgres` database during setup to avoid noisy warning output.
 - `load.py`: Use timezone-aware `MIN_DATETIME_UTC` / `MAX_DATETIME_UTC` sentinel
   constants (instead of naive `datetime.min` / `datetime.max`) to avoid
   `TypeError: can't compare offset-naive and offset-aware datetimes`.

CLAUDE.md

@@ -11,10 +11,12 @@ PgSTAC is a PostgreSQL extension (SQL functions + schema) for Spatio-Temporal As
 ## Architecture
 
 ```
+src/pgstac/pyproject.toml ← pgpkg project config for SQL + migration artifacts
 src/pgstac/sql/          ← ALL SQL source files (edit ONLY here)
 src/pgstac/pgstac.sql    ← Assembled output (DO NOT edit directly)
 src/pgstac/migrations/   ← Base + incremental migration files
 src/pgstac/tests/        ← PGTap and basic SQL tests
+src/pgstac-migrate/      ← Standalone pgstac-migrate wrapper package + baked artifact
 src/pypgstac/src/pypgstac/ ← Python package source
 src/pypgstac/tests/        ← pytest tests
 scripts/                 ← Host-facing entrypoint scripts
@@ -95,6 +97,8 @@ All tests run inside Docker via `scripts/runinpypgstac`. Use `--build` to rebuil
 
 - **pgstac** container: PostgreSQL 17 + PostGIS 3 + extensions, port 5439→5432
 - **pypgstac** container: Python + Rust build tools, runs scripts
+- `scripts/runinpypgstac` uses the published-package path by default; set `PGPKG_LOCAL_REPO_DIR` to mount a local `pgpkg` checkout at `/pgpkg` and export `PGPKG_REPO_DIR` when `stageversion` or `makemigration` should run against a local checkout
+- When no local checkout is mounted, the in-container `stageversion` / `makemigration` helpers resolve `pgpkg>=0.1.1,<0.2` from PyPI with `uv run --no-project --with ...`
 - Credentials: `username` / `password`, database: `postgis`
 
 ## Migration Process
@@ -108,21 +112,20 @@ scripts/stageversion 0.9.11
 This runs inside Docker and:
 1. Removes old `*unreleased*` migration files
 2. Writes `SELECT set_version('0.9.11');` to `999_version.sql`
-3. Concatenates all `sql/*.sql` → `migrations/pgstac.0.9.11.sql` (base migration)
-4. Copies the base migration to `pgstac.sql`
+3. Runs `pgpkg stageversion` against `src/pgstac/pyproject.toml` → `migrations/pgstac--0.9.11.sql`
+4. Uses `--also-write` to keep `pgstac.sql` synchronized with the latest base migration
 5. Updates `version.py` and `pyproject.toml` version strings
-6. Runs `makemigration -f 0.9.10 -t 0.9.11` to generate incremental migration
+6. Runs `makemigration -f 0.9.10 -t 0.9.11` to generate the wrapped incremental migration via `pgpkg`
 
 ### How makemigration Works
 
-`makemigration` (copied from `scripts/container-scripts/makemigration` into the image) generates incremental migrations by diffing schemas:
+`makemigration` (copied from `scripts/container-scripts/makemigration` into the image) now prefers a local checkout via `PGPKG_REPO_DIR`, otherwise it resolves the pinned published package with `uv run --no-project --with "pgpkg[diff]>=0.1.1,<0.2" pgpkg makemigration`:
 
-1. Creates two temp databases: `migra_from`, `migra_to`
-2. Loads old base migration into `migra_from`
-3. Loads new base migration into `migra_to`
-4. Runs `migra --schema pgstac --unsafe` to calculate the SQL diff
-5. Wraps the diff with `000_idempotent_pre.sql`, `998_idempotent_post.sql`, and `set_version()`
-6. Output: `migrations/pgstac.0.9.10-0.9.11.sql`
+1. Uses `src/pgstac/pyproject.toml` to locate the canonical staged base files
+2. Uses `results.temporary_local_db` via `pgpkg` to diff the source and target staged bases
+3. Prepends `000_idempotent_pre.sql`
+4. Appends `998_idempotent_post.sql` and `SELECT set_version(...)`
+5. Writes `migrations/pgstac--0.9.10--0.9.11.sql`
 
 **Important**: The generated migration is created with a `.staged` suffix. You MUST:
 1. Review the `.staged` file for correctness
@@ -132,11 +135,17 @@ This runs inside Docker and:
 ### Running Migrations
 
 ```bash
-pypgstac migrate                    # Migrate to current pypgstac version
-pypgstac migrate --toversion 0.9.10 # Migrate to specific version
+pypgstac migrate                    # Backwards-compatible wrapper over pgstac-migrate
+pypgstac migrate --toversion 0.9.10 # Backwards-compatible wrapper over pgstac-migrate
+uv run --directory src/pgstac-migrate pgstac-migrate build-artifact
+uv run --directory src/pgstac-migrate pgstac-migrate info
+uv run --directory src/pgstac-migrate pgstac-migrate versions
 ```
 
-The `Migrate` class (in `migrate.py`) builds a directed graph of all available migration files and uses BFS to find the shortest path from the current DB version to the target.
+`pgstac-migrate` owns runtime migration planning and apply logic. `pypgstac migrate` delegates to the same Python API for backwards compatibility and does not execute source-tree SQL files directly.
+The source-tree `pgstac-migrate` package prefers the baked artifact at `src/pgstac-migrate/src/pgstac_migrate/migrations.tar.zst` and rebuilds it from the source tree when that file is missing.
+`src/pgstac-migrate/pyproject.toml` resolves `pgpkg>=0.1.1,<0.2` from PyPI. The standalone `src/pgstac-migrate/scripts/build_artifact.py` helper does not use that lockfile; it carries its own inline `pgpkg>=0.1.1,<0.2` dependency.
+`src/pypgstac/pyproject.toml` keeps a local `[tool.uv.sources]` override to the sibling `../pgstac-migrate` project so `uv run --directory src/pypgstac ...` resolves the wrapper stack from the source tree, while `pgpkg` resolves from PyPI. In the Docker-backed dev flow, `scripts/runinpypgstac` can mount a local pgpkg checkout at `/pgpkg` and export `PGPKG_REPO_DIR` for container-script testing.
 
 ## Testing Details
 
@@ -178,7 +187,7 @@ Tests create `pgstac_test_db_template` from `pgstac.sql`, then clone it per test
 5. Copy updated `CHANGELOG.md` to `docs/src/release-notes.md` (keep identical)
 6. Create PR, merge
 7. `git tag vVERSION && git push origin vVERSION`
-8. CI publishes to PyPI + ghcr.io
+8. CI publishes `pypgstac` and `pgstac-migrate` to PyPI plus the ghcr.io images (requires trusted publishers for both PyPI projects on `.github/workflows/release.yml` with the `pypi` environment)
 
 ## Common Patterns
 

CONTRIBUTING.md

@@ -122,7 +122,7 @@ Individual tests can be run with any combination of the following flags `--forma
 6) Once the PR has been merged, start the release process.
 7) Create a git tag `git tag v0.2.8` using new version number
 8) Push the git tag `git push origin v0.2.8`
-9) The CI process will push pypgstac to PyPi, create a docker image on ghcr.io, and create a release on github.
+9) The CI process will push `pypgstac` and `pgstac-migrate` to PyPI, create docker images on ghcr.io, and create a release on GitHub. Register PyPI trusted publishers for both projects before the first tagged release.
 
 
 ### Get Involved

README.md

@@ -24,7 +24,7 @@
 
 ---
 
-**PgSTAC** is a set of SQL functions and schema to build highly performant databases for Spatio-Temporal Asset Catalogs ([STAC](https://stacspec.org/)). The project also provides **pypgstac** (a Python module) to help with database migrations and document ingestion (collections and items).
+**PgSTAC** is a set of SQL functions and schema to build highly performant databases for Spatio-Temporal Asset Catalogs ([STAC](https://stacspec.org/)). The project also provides **pgstac-migrate** (a focused migration package) and **pypgstac** (a Python module for compatibility migration commands and document ingestion).
 
 PgSTAC provides functionality for STAC Filters, CQL2 search, and utilities to help manage the indexing and partitioning of STAC Collections and Items.
 
@@ -34,6 +34,8 @@ PgSTAC Documentation: https://stac-utils.github.io/pgstac/pgstac
 
 pyPgSTAC Documentation: https://stac-utils.github.io/pgstac/pypgstac
 
+pgstac-migrate package: `src/pgstac-migrate`
+
 ## Project structure
 
 ```

docker/pypgstac/Dockerfile

@@ -4,7 +4,7 @@ ENV PYTHONWRITEBYTECODE=1
 ENV PYTHONBUFFERED=1
 ENV PIP_ROOT_USER_ACTION=ignore
 ENV UV_BREAK_SYSTEM_PACKAGES=1
-ENV PYTHONPATH="/opt/src/pypgstac:$PYTHONPATH"
+ENV PYTHONPATH="/opt/src/pypgstac:/opt/src/pgstac-migrate:$PYTHONPATH"
 ENV PATH="/opt/pgstac/container-scripts:$PATH"
 ENV UV_CACHE_DIR=/root/.cache/uv
 ARG PG_MAJOR=17
@@ -24,34 +24,26 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
     && rm -rf /var/lib/apt/lists/*
 
 FROM pyrustbase AS pypgstac
-COPY ./src/pypgstac/pyproject.toml /tmp/pyproject.toml
-WORKDIR /tmp
-RUN --mount=type=cache,target=/root/.cache/uv,sharing=locked \
-    uv pip compile /tmp/pyproject.toml \
-    --extra dev \
-    --extra test \
-    --extra psycopg \
-    --extra migrations \
-    >/tmp/requirements.txt \
-    && uv pip install --system -r /tmp/requirements.txt
+ENV UV_CACHE_DIR=/home/user/.cache/uv
 COPY scripts/container-scripts /opt/pgstac/container-scripts
 COPY src/pypgstac /opt/src/pypgstac
 COPY src/pgstac /opt/src/pgstac
+COPY src/pgstac-migrate /opt/src/pgstac-migrate
 WORKDIR /opt/src/pypgstac
-RUN --mount=type=cache,target=/root/.cache/uv,sharing=locked \
-    uv pip install --system -e . \
-    && rm -rf /usr/local/cargo/registry
+RUN rm -rf /usr/local/cargo/registry
 
 RUN addgroup --gid 1000 user && \
     adduser --uid 1000 --gid 1000 --disabled-password --gecos "" --home /home/user user && \
-    chown -R user:user /opt/src/pypgstac /opt/src/pgstac
+    mkdir -p /home/user/.cache/uv && \
+    chown -R user:user /home/user /opt/src/pypgstac /opt/src/pgstac /opt/src/pgstac-migrate
 USER user
 
 # Optional runtime-optimized image: no build toolchain, only pypgstac package + runtime deps.
 FROM python:3.13-slim-trixie AS pypgstac-runtime
 ENV PYTHONWRITEBYTECODE=1
 ENV PYTHONBUFFERED=1
 ENV UV_BREAK_SYSTEM_PACKAGES=1
+ENV PYTHONPATH="/opt/src/pypgstac:/opt/src/pgstac-migrate:$PYTHONPATH"
 ENV PATH="/opt/pgstac/container-scripts:$PATH"
 ENV UV_CACHE_DIR=/root/.cache/uv
 ARG PG_MAJOR=17
@@ -65,16 +57,8 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
         postgresql-client-${PG_MAJOR} \
     && curl -LsSf https://astral.sh/uv/install.sh | UV_INSTALL_DIR=/usr/local/bin sh \
     && apt-get clean && rm -rf /var/lib/apt/lists/*
-COPY ./src/pypgstac/pyproject.toml /tmp/pyproject.toml
-RUN --mount=type=cache,target=/root/.cache/uv,sharing=locked \
-    uv pip compile /tmp/pyproject.toml \
-    --extra psycopg \
-    --extra migrations \
-    >/tmp/requirements.txt \
-    && uv pip install --system -r /tmp/requirements.txt
 COPY scripts/container-scripts /opt/pgstac/container-scripts
 COPY src/pypgstac /opt/src/pypgstac
 COPY src/pgstac /opt/src/pgstac
+COPY src/pgstac-migrate /opt/src/pgstac-migrate
 WORKDIR /opt/src/pypgstac
-RUN --mount=type=cache,target=/root/.cache/uv,sharing=locked \
-    uv pip install --system .