Harden search cache and add pg_stat_statements/pg_cron support

.github/instructions/scripts.instructions.md

@@ -10,6 +10,7 @@ See CLAUDE.md "Development Workflow" for usage. All scripts require the Docker c
 - `runinpypgstac` uses the published-package path by default; set `PGPKG_LOCAL_REPO_DIR` to mount a local `pgpkg` checkout at `/pgpkg` when you need an override
 - `scripts/container-scripts/` contains the in-container script payload copied into the pypgstac image; keep host wrappers in `scripts/`
 - `stageversion` modifies version files AND generates migrations — see CLAUDE.md "Migration Process"
+- `stageversion` regenerates `*unreleased*` migrations each run; if you hand-edit incremental SQL, rebuild the baked artifact with `uv run --directory src/pgstac-migrate pgstac-migrate build-artifact` and avoid rerunning `stageversion` unless you intend to overwrite edits
 - `scripts/container-scripts/stageversion` and `scripts/container-scripts/makemigration` now shell through `pgpkg` inside the container rather than assembling/diffing SQL directly
 - Set `PGPKG_LOCAL_REPO_DIR` on the host when you need to force a local pgpkg checkout for `stageversion`, `makemigration`, or related container-script testing
 - Tagged releases run `.github/workflows/release.yml`, which publishes both `pypgstac` and `pgstac-migrate` to PyPI via the GitHub `pypi` environment; PyPI trusted publishers must exist for both projects

.github/workflows/continuous-integration.yml

@@ -225,4 +225,4 @@ jobs:
       - name: Set search_path
         run: psql -c "ALTER ROLE username SET search_path TO pgstac, public;"
       - name: Test
-        run: cargo test -p pgstac --all-features --manifest-path rust/Cargo.toml
+        run: cargo test -p pgstac --all-features --manifest-path src/pgstac-rs/Cargo.toml

.github/workflows/release.yml

@@ -261,7 +261,7 @@ jobs:
       - uses: rust-lang/crates-io-auth-action@bbd81622f20ce9e2dd9622e3218b975523e45bbe # v1.0.4
         id: auth
       - name: Publish
-        working-directory: rust
+        working-directory: src/pgstac-rs
         env:
           CARGO_REGISTRY_TOKEN: ${{ steps.auth.outputs.token }}
         run: cargo publish

.gitignore

@@ -19,3 +19,4 @@ src/pgstacrust/target/
 src/pgstac-migrate/dist/
 src/pgstac-migrate/src/pgstac_migrate/migrations.tar.zst
 src/pypgstac/uv.lock
+*_PLAN.md

AGENTS.md

@@ -26,10 +26,11 @@ Migration specialist for PgSTAC. See CLAUDE.md "Migration Process" for full work
 2. `src/pgstac/pyproject.toml` is the `pgpkg` project config for the SQL + migrations tree
 3. `uv run --directory src/pgstac-migrate pgstac-migrate info|versions|plan` inspects the baked migration artifact during wrapper work
 4. `uv run --directory src/pypgstac pypgstac migrate -- --help` remains a backwards-compatible wrapper over `pgstac-migrate`; put new runtime migration behavior in `src/pgstac-migrate/`, not `src/pypgstac/`
-5. `scripts/stageversion VERSION` → generates canonical `pgstac--VERSION.sql` plus an incremental `.staged` migration; set `PGPKG_LOCAL_REPO_DIR` when `stageversion` or `makemigration` should run against a local pgpkg checkout. The Docker-backed flow mounts that override at `/pgpkg` and exports `PGPKG_REPO_DIR` to the container scripts.
-6. Review `.staged` file (watch for DROPs, unsafe ALTERs, missing `CREATE OR REPLACE`)
-7. Remove `.staged` suffix → `scripts/test --migrations`
-8. Tagged releases publish both `pypgstac` and `pgstac-migrate` to PyPI from `.github/workflows/release.yml`; keep the PyPI trusted publisher registration aligned with the `pypi` environment and workflow path
+5. `scripts/stageversion VERSION` regenerates canonical `pgstac--VERSION.sql` plus incremental `pgstac--FROM--TO.sql`; set `PGPKG_LOCAL_REPO_DIR` when `stageversion` or `makemigration` should run against a local pgpkg checkout. The Docker-backed flow mounts that override at `/pgpkg` and exports `PGPKG_REPO_DIR` to the container scripts.
+6. Review the generated incremental migration (watch for DROPs, unsafe ALTERs, missing `CREATE OR REPLACE`)
+7. If you hand-edit the incremental migration, rebuild the baked artifact: `uv run --directory src/pgstac-migrate pgstac-migrate build-artifact`
+8. Run `scripts/test --migrations` (or full `scripts/test` gate)
+9. Tagged releases publish both `pypgstac` and `pgstac-migrate` to PyPI from `.github/workflows/release.yml`; keep the PyPI trusted publisher registration aligned with the `pypi` environment and workflow path
 
 ### Review Checklist
 

CHANGELOG.md

@@ -11,6 +11,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 ### Added
 - New `pgstac-migrate` package under `src/pgstac-migrate/` with a standalone
   CLI, Python API, and tests for migration planning and execution.
+- New Rust crate under `src/pgstac-rs/` with updated CI/release wiring,
+  README guidance, and test coverage.
 - `src/pgstac/pyproject.toml` `tool.pgpkg` project metadata for canonical SQL +
   migration staging.
 - `scripts/makemigration` host wrapper for the in-container `makemigration` helper.
@@ -29,6 +31,12 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 - `workflow_dispatch` trigger for manual CI runs.
 - `pg_tle` v1.5.2 built and pre-loaded in the `pgstacbase` image; database init runs
   `CREATE EXTENSION IF NOT EXISTS pg_tle`.
+- `pg_stat_statements` and `pg_cron` are now installed in the pgstac Docker image,
+  added to `shared_preload_libraries`, and initialized during container bootstrap
+  (`pg_stat_statements` in the app database, `pg_cron` in `postgres`).
+- `scripts/container-scripts/test` now includes extension smoke tests that verify
+  preload configuration plus basic runtime behavior for both
+  `pg_stat_statements` and `pg_cron`.
 - `pypgstac-runtime` Docker target: slim Python 3.13-trixie image without the Rust/build
   toolchain, for production deployments where the Rust build environment is not needed.
 - Dependabot coverage expanded to Docker base images and pip packages (two new
@@ -48,6 +56,15 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
   `PGPKG_REPO_DIR` override support.
 - `scripts/runinpypgstac` now supports a `PGPKG_LOCAL_REPO_DIR` mount override
   for local pgpkg development while keeping the default flow PyPI-first.
+- Search cache hashing, storage, and concurrency control were reworked: SHA-256
+  cache keys, canonical where-clause inputs, `searches`-backed lifecycle,
+  retention-driven GC, and less blocking row touch / update behavior.
+- Search context stats updates now use optimistic compare-and-update guards on
+  `statslastupdated`, reducing stale overwrites when concurrent workers refresh
+  counts.
+- GitHub Actions and release automation were refreshed for the current layout:
+  Rust crate path updates, workflow/action version bumps, and Dependabot group
+  adjustments.
 - Tagged releases now publish the new `pgstac-migrate` package to PyPI alongside `pypgstac` via trusted publishing in `.github/workflows/release.yml`.
 - In-container helper scripts moved from `docker/pypgstac/bin/` to
   `scripts/container-scripts/`; container `PATH` updated accordingly.
@@ -61,22 +78,23 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
   `--build` flag; `PGSTAC_BUILD_POLICY` env var provides a persistent default.
 - Dev tooling: `flake8`, `black`, and `mypy` removed in favour of `ruff==0.15.11` and
   `ty==0.0.31`. `pre-commit` pinned to `3.5.0`. `pre-commit-hooks` updated to v5.0.0.
-- `pypgstac` package floor raised to Python 3.11; metadata now advertises 3.11-3.14.
-- `pypgstac` settings now use `pydantic-settings` (`BaseSettings` from
-  `pydantic_settings`) and require `pydantic>=2,<3`.
 - `cachetools` upper bound removed (`cachetools>=5.3.0`) since `pypgstac` only uses
   `cachetools.func.lru_cache`; no known incompatible API changes affect this usage.
 - `pypgstac` developer tooling config now consistently targets Ruff + ty:
   removes stale mypy config, pins Ruff to `0.15.11` to match pre-commit,
   and adds minimal `[tool.ty]` project settings.
+- `pypgstac` now requires Python 3.11+ and advertises support through 3.14;
+  settings now use `pydantic-settings` and require `pydantic>=2,<3`.
 - Formatting/type-check pipeline now uses `scripts/test --formatting` as the
   single pre-commit entry point (removing duplicate direct Ruff pre-commit hooks)
   and aligns Ruff line-length handling with the formatter (`E501` ignored;
   explicit `line-length = 88`).
-- GitHub Actions updated: `dorny/paths-filter` v2→v3, `docker/build-push-action`
-  v4→v6, `astral-sh/setup-uv` v8.0.0→v8.1.0; all SHA pins refreshed.
-- Dependabot groups reworked: `actions-all` (replaces `minor-and-patch`), new
-  `docker-base-images`, `python-dev-tooling`, and `python-runtime` groups.
+- GitHub Actions and release automation were refreshed for the current layout:
+  Rust crate path updates, `dorny/paths-filter` v2→v3,
+  `docker/build-push-action` v4→v6, `astral-sh/setup-uv` v8.0.0→v8.1.0,
+  refreshed SHA pins, and Dependabot group updates (`actions-all` replaces
+  `minor-and-patch`, with new `docker-base-images`, `python-dev-tooling`, and
+  `python-runtime` groups).
 - `docker-compose.yml` removes explicit `container_name` entries to avoid conflicts
   between concurrent local instances.
 
@@ -86,8 +104,11 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 - `flake8`, `black`, and `mypy` removed from dev dependencies.
 
 ### Fixed
+- Explicit search stats refresh now propagates through cached and uncached search paths when `updatestats` is requested, keeping `numberMatched`/context counts current.
 - `scripts/container-scripts/test` now refreshes collation metadata for the
   `postgres` database during setup to avoid noisy warning output.
+- Read-only search with context now returns `numberMatched` without requiring
+  cache writes, reducing failure risk for replica/read-only deployments.
 - `load.py`: Use timezone-aware `MIN_DATETIME_UTC` / `MAX_DATETIME_UTC` sentinel
   constants (instead of naive `datetime.min` / `datetime.max`) to avoid
   `TypeError: can't compare offset-naive and offset-aware datetimes`.
@@ -97,6 +118,9 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
   the broken `3.9.0` sdist under `--resolution lowest-direct`.
 - `pydantic` minimum raised to `>=2.10` so `--resolution lowest-direct` on Python 3.13
   does not resolve to `pydantic-core==2.0.1`, which fails to build.
+- `scripts/container-scripts/test` now derives the active database from
+  `PGDATABASE`/`POSTGRES_DB` when checking server extensions and refreshing
+  collation versions, instead of assuming `postgis`.
 
 
 ## [v0.9.11]

CLAUDE.md

@@ -127,10 +127,12 @@ This runs inside Docker and:
 4. Appends `998_idempotent_post.sql` and `SELECT set_version(...)`
 5. Writes `migrations/pgstac--0.9.10--0.9.11.sql`
 
-**Important**: The generated migration is created with a `.staged` suffix. You MUST:
-1. Review the `.staged` file for correctness
-2. Remove the `.staged` suffix to enable it
-3. Run `scripts/test --migrations` to validate
+**Important**:
+1. `scripts/stageversion` regenerates `*unreleased*` migration files on each run.
+2. If you hand-edit an incremental migration, do not rerun `stageversion` unless you want those edits overwritten.
+3. After hand-editing an incremental migration, rebuild the baked artifact:
+	`uv run --directory src/pgstac-migrate pgstac-migrate build-artifact`
+4. Validate with `scripts/test --migrations` (or `scripts/test` for the full gate).
 
 ### Running Migrations
 
@@ -181,13 +183,14 @@ Tests create `pgstac_test_db_template` from `pgstac.sql`, then clone it per test
 ## Release Checklist
 
 1. `scripts/stageversion VERSION`
-2. Review `.staged` migration, remove suffix
-3. `scripts/test --migrations`
-4. Move CHANGELOG "Unreleased" → new version
-5. Copy updated `CHANGELOG.md` to `docs/src/release-notes.md` (keep identical)
-6. Create PR, merge
-7. `git tag vVERSION && git push origin vVERSION`
-8. CI publishes `pypgstac` and `pgstac-migrate` to PyPI plus the ghcr.io images (requires trusted publishers for both PyPI projects on `.github/workflows/release.yml` with the `pypi` environment)
+2. Review generated incremental migration for correctness
+3. If hand-edited, run `uv run --directory src/pgstac-migrate pgstac-migrate build-artifact`
+4. `scripts/test --migrations`
+5. Move CHANGELOG "Unreleased" → new version
+6. Copy updated `CHANGELOG.md` to `docs/src/release-notes.md` (keep identical)
+7. Create PR, merge
+8. `git tag vVERSION && git push origin vVERSION`
+9. CI publishes `pypgstac` and `pgstac-migrate` to PyPI plus the ghcr.io images (requires trusted publishers for both PyPI projects on `.github/workflows/release.yml` with the `pypi` environment)
 
 ## Common Patterns
 

docker/pgstac/Dockerfile

@@ -16,6 +16,8 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
     && apt-get install -y --no-install-recommends \
         postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR \
         postgresql-$PG_MAJOR-postgis-$POSTGIS_MAJOR-scripts \
+        postgresql-$PG_MAJOR-cron \
+        postgresql-contrib-$PG_MAJOR \
         postgresql-$PG_MAJOR-pgtap \
         postgresql-$PG_MAJOR-plpgsql-check \
         postgresql-$PG_MAJOR-partman \
@@ -31,8 +33,8 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
     && make -C /tmp/pg_tle \
     && make -C /tmp/pg_tle install \
     && rm -rf /tmp/pg_tle \
-    && sed -i "s/^#shared_preload_libraries = .*/shared_preload_libraries = 'pg_tle'/" /usr/share/postgresql/$PG_MAJOR/postgresql.conf.sample \
-    && sed -i "s/^#shared_preload_libraries = .*/shared_preload_libraries = 'pg_tle'/" /usr/share/postgresql/postgresql.conf.sample \
+    && sed -i "s/^#shared_preload_libraries = .*/shared_preload_libraries = 'pg_tle,pg_stat_statements,pg_cron'/" /usr/share/postgresql/$PG_MAJOR/postgresql.conf.sample \
+    && sed -i "s/^#shared_preload_libraries = .*/shared_preload_libraries = 'pg_tle,pg_stat_statements,pg_cron'/" /usr/share/postgresql/postgresql.conf.sample \
     && apt-get purge -y --auto-remove \
         postgresql-server-dev-$PG_MAJOR \
         build-essential \

docker/pgstac/dbinit/pgstac.sh

@@ -3,11 +3,14 @@ SHARED_BUFFERS=$(( $SYSMEM/4 ))
 EFFECTIVE_CACHE_SIZE=$(( $SYSMEM*3/4 ))
 MAINTENANCE_WORK_MEM=$(( $SYSMEM/8 ))
 WORK_MEM=$(( $SHARED_BUFFERS/50 ))
-
 psql -X -q -v ON_ERROR_STOP=1 <<EOSQL
 CREATE EXTENSION IF NOT EXISTS pg_tle;
+CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
+\connect postgres
+CREATE EXTENSION IF NOT EXISTS pg_cron;
 ALTER SYSTEM SET search_path TO pgstac, public;
 ALTER SYSTEM SET client_min_messages TO WARNING;
+ALTER SYSTEM SET cron.database_name='postgres';
 ALTER SYSTEM SET shared_buffers='${SHARED_BUFFERS}kB';
 ALTER SYSTEM SET effective_cache_size='${EFFECTIVE_CACHE_SIZE}kB';
 ALTER SYSTEM SET maintenance_work_mem='${MAINTENANCE_WORK_MEM}kB';

scripts/container-scripts/stageversion

@@ -1,4 +1,6 @@
 #!/bin/bash
+set -euo pipefail
+
 SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
 SRCDIR=${PGSTAC_REPO_DIR:-/opt/src}
 cd $SRCDIR
@@ -29,13 +31,23 @@ Usage: $(basename "$0") [version]
 Create a new base migration, update pypgstac version metadata, and generate the
 incremental migration via makemigration.
 
+Important workflow note:
+        This command regenerates *unreleased* migrations each time it runs.
+        If you hand-edit an incremental migration, do NOT rerun stageversion unless
+        you want those edits overwritten. After hand edits, rebuild the baked
+        artifact with:
+            uv run --directory src/pgstac-migrate pgstac-migrate build-artifact
+
 Environment:
     PGSTAC_VERSION   Default version when no positional version is provided.
     PGPKG_REPO_DIR   Optional local pgpkg checkout to use instead of the installed package.
 EOF
 }
 
-if [[ "$1" == "-h" || "$1" == "--help" ]]; then
+ARG1=${1:-}
+VERSION=""
+
+if [[ "$ARG1" == "-h" || "$ARG1" == "--help" ]]; then
         usage
         exit 0
 fi
@@ -45,8 +57,8 @@ fi
 find $MIGRATIONSDIR -name "*unreleased*" -exec rm {} \;
 
 # Get Version
-if [[ -n "$1" ]]; then
-    VERSION=$1
+if [[ -n "$ARG1" ]]; then
+    VERSION=$ARG1
 elif [[ -n "${PGSTAC_VERSION:-}" ]]; then
     VERSION=$PGSTAC_VERSION
 fi
@@ -83,8 +95,13 @@ cd $PGSTACDIR
 echo "Setting pypgstac version to $PYVERSION"
 cat <<EOD > $PYPGSTACDIR/src/pypgstac/version.py
 """Version."""
+
 __version__ = "${PYVERSION}"
 EOD
 sed -i "s/^version[ ]*=[ ]*.*$/version = \"${PYVERSION}\"/" $PYPGSTACDIR/pyproject.toml
 
 makemigration -f $OLDVERSION -t $VERSION
+
+echo "Stageversion complete."
+echo "If you hand-edit the incremental migration, rebuild the baked artifact with:"
+echo "  uv run --directory src/pgstac-migrate pgstac-migrate build-artifact"

scripts/container-scripts/test

@@ -65,9 +65,73 @@ EOSQL
 
 function refresh_collation_versions(){
     # Newer container libc versions can make template collation metadata stale.
-    psql -X -q -d postgres -c "ALTER DATABASE template1 REFRESH COLLATION VERSION;" >/dev/null 2>&1 || true
-    psql -X -q -d postgres -c "ALTER DATABASE postgres REFRESH COLLATION VERSION;" >/dev/null 2>&1 || true
-    psql -X -q -d postgres -c "ALTER DATABASE postgis REFRESH COLLATION VERSION;" >/dev/null 2>&1 || true
+    local appdb="${PGDATABASE:-${POSTGRES_DB:-postgres}}"
+    local db
+
+    for db in template1 postgres "$appdb"; do
+        psql -X -q -d postgres -c "ALTER DATABASE ${db} REFRESH COLLATION VERSION;" >/dev/null 2>&1 || true
+    done
+}
+
+function test_server_extensions(){
+    local appdb="${PGDATABASE:-${POSTGRES_DB:-postgres}}"
+    local pgss_count
+
+    # CI test jobs use the pgstacbase image (no init scripts), so create the
+    # extensions in the active databases before validating preload/runtime behavior.
+    psql -X -q -v ON_ERROR_STOP=1 -d "$appdb" -c "CREATE EXTENSION IF NOT EXISTS pg_stat_statements;" >/dev/null
+    psql -X -q -v ON_ERROR_STOP=1 -d postgres -c "CREATE EXTENSION IF NOT EXISTS pg_cron;" >/dev/null
+
+    psql -X -q -v ON_ERROR_STOP=1 -d "$appdb" <<EOSQL
+DO \$\$
+DECLARE
+    preload_libs text;
+BEGIN
+    preload_libs := current_setting('shared_preload_libraries');
+    IF preload_libs NOT LIKE '%pg_stat_statements%' THEN
+        RAISE EXCEPTION 'pg_stat_statements is not in shared_preload_libraries: %', preload_libs;
+    END IF;
+    IF preload_libs NOT LIKE '%pg_cron%' THEN
+        RAISE EXCEPTION 'pg_cron is not in shared_preload_libraries: %', preload_libs;
+    END IF;
+
+    IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pg_stat_statements') THEN
+        RAISE EXCEPTION 'pg_stat_statements extension is not installed in %', current_database();
+    END IF;
+END
+\$\$;
+EOSQL
+
+    psql -X -q -v ON_ERROR_STOP=1 -d "$appdb" -c "SELECT pg_stat_statements_reset();" >/dev/null
+    psql -X -q -v ON_ERROR_STOP=1 -d "$appdb" -c "SELECT count(*) FROM pg_class;" >/dev/null
+    pgss_count=$(psql -X -q -t -A -v ON_ERROR_STOP=1 -d "$appdb" -c "SELECT count(*) FROM pg_stat_statements;")
+    if [[ -z "$pgss_count" || "$pgss_count" -eq 0 ]]; then
+        echo "pg_stat_statements did not record statements for ${appdb}" >&2
+        exit 1
+    fi
+
+    psql -X -q -v ON_ERROR_STOP=1 -d postgres <<EOSQL
+DO \$\$
+DECLARE
+    preload_libs text;
+BEGIN
+    preload_libs := current_setting('shared_preload_libraries');
+    IF preload_libs NOT LIKE '%pg_cron%' THEN
+        RAISE EXCEPTION 'pg_cron is not in shared_preload_libraries: %', preload_libs;
+    END IF;
+    IF current_setting('cron.database_name') <> 'postgres' THEN
+        RAISE EXCEPTION 'cron.database_name expected postgres but was %', current_setting('cron.database_name');
+    END IF;
+    IF NOT EXISTS (SELECT 1 FROM pg_extension WHERE extname = 'pg_cron') THEN
+        RAISE EXCEPTION 'pg_cron extension is not installed in %', current_database();
+    END IF;
+
+    PERFORM cron.unschedule(cron.schedule('* * * * *', 'SELECT 1'));
+END
+\$\$;
+EOSQL
+
+    echo "Server extension tests passed for ${appdb} + postgres."
 }
 
 function test_formatting(){
@@ -410,7 +474,11 @@ then
 fi
 
 [ $FORMATTING -eq 1 ] && test_formatting
-[ $SETUPDB -eq 1 ] && refresh_collation_versions && setuptestdb
+if [ $SETUPDB -eq 1 ]; then
+    refresh_collation_versions
+    test_server_extensions
+    setuptestdb
+fi
 [ $PGTAP -eq 1 ] && test_pgtap
 [ $BASICSQL -eq 1 ] && test_basicsql
 [ $PYPGSTAC -eq 1 ] && test_pypgstac