Fix pre-existing tools test failures: RPM signing no-op, docker PG-version drift, EOL images, and PG16/17/18 package-test retarget

[ihalatci]

Summary

This PR consolidates three independent pre-existing fixes into a single branch so CI exercises them together. It supersedes / folds in #408 (whose commit is cherry-picked here). All of this is independent of the GH_TOKEN → GitHub App migration — no app-token/secret logic is touched.

Five commits (on top of develop 9322601):

1. 4c862ce — RPM signing was silently skipped (path-doubling) + the verifier was blind.
2. df93bb1 — test_sign_packages now signs the real built rpm (in rpm_build) instead of an empty folder.
3. 8ac87b4 — Retarget citus-package-tests to PG16/17/18 (citus 14.1.0) + bump test-image HLL/TOPN.
4. 4695e3a — Drop EOL centos/8; add published ubuntu/noble to the package-test matrix.
5. a6d04f8 — (Fix pre-existing test failures: align docker PG versions to source pkgvars; retarget EOL build images #408) align update_docker.py to the docker pkgvars source-of-truth (PG14/15 → 16/17/18) + retarget EOL debian-stretch build image → debian-bookworm.

---

Root causes

A. RPM signing silently skipped (citus_package.py) + blind verifier (common_tool_methods.py)

build_packages() pre-mutates output_dir to {base}/{sub_folder}, but sign_packages() appended {sub_folder} again → globbed {base}/{sub_folder}/{sub_folder} → 0 files → signing skipped with no error. The verifier is_rpm_file_signed() used rpm -K (returncode), which returns 0 for unsigned rpms too, so the regression was invisible — published community RPMs have been unsigned since this no-op landed (#175, 2021), surfacing in CI only once packagecloud turned on per-package gpgcheck enforcement (a new signing subkey 723C4197ED5519E8 was provisioned 2026-02-12).

Fix: restore the base output_dir before sign_packages() (call-site fix, leaves the sub_folder-append contract intact so test_sign_packages stays valid); make the empty-package case loud; replace the verifier with a real 3-tag PGP check (%{SIGPGP}|%{SIGGPG}|%{RSAHEADER} — unsigned iff all three (none), back-compatible with legacy V3 sigs and future RSAHEADER-only sigs); add a regression assertion that produced rpms must be signed.

B. docker pkgvars schema drift (#408)

citusdata/docker pkgvars ships only postgres_16/17/18 now (14/15 removed). read_postgres_versions() still returned 5 keys → KeyError at runtime, and test_pkgvar_postgres_version_existence asserted 14/15. Fix: align both production code and tests to PG16/17/18.

C. EOL / unavailable build & package-test images

• test_build_package_debian used debian-stretch (long EOL) → image gone. Retargeted to debian-bookworm (Fix pre-existing test failures: align docker PG versions to source pkgvars; retarget EOL build images #408).
• citus-package-tests matrix tested PG14/15/16 via the stale PROJECT_VERSION=12.1.5 default and pinned HLL 2.18 / TOPN 2.6.0. Retargeted the default to 14.1.0 → PG16/17/18, and bumped the test-image HLL_VERSION/TOPN_VERSION ARGs to 2.19.citus-1 / 2.7.0.citus-1 (the versions actually published for the 14.1.0 era; pg17/18 don't ship 2.18/2.6.0).
• Dropped centos/8 (no longer in citusdata/packaging's published matrix; base image is EOL → vault) and added ubuntu/noble (published, was missing).

---

What changed

• .github/workflows/package-tests.yml — PROJECT_VERSION default 12.1.5 → 14.1.0 (metadata + test_execution); matrix: - centos/8 removed, - ubuntu/noble added.
• packaging_automation/citus_package.py — restore base output_dir before signing; loud-warn on 0 packages.
• packaging_automation/common_tool_methods.py — real 3-tag is_rpm_file_signed.
• packaging_automation/tests/test_citus_package.py — regression assert: built rpms must be signed.
• packaging_automation/tests/test_citus_package_utils.py — debian-stretch → debian-bookworm; rpm sign target centos-8 → rpm_build.
• packaging_automation/tests/test_update_docker.py, packaging_automation/update_docker.py — PG14/15 → 16/17/18 alignment.
• test-images/{almalinux-9,ol-9,debian-bullseye,debian-bookworm,ubuntu-jammy,ubuntu-noble,centos-8}/Dockerfile — HLL 2.18 → 2.19, TOPN 2.6.0 → 2.7.0.

Validated locally (WSL + Docker Desktop)

• test_update_docker.py → 9 passed (Failure Remove socket directory permissions fix #1 fixed).
• Full tests collect cleanly (76 tests; only test_publish_docker.py errors, due to no docker-socket access for the python SDK in this env — out of scope, unchanged).
• black --check and prospector clean on all touched files (the 2 reported findings — pypi_stats_collector.py black drift, upload_to_package_cloud.py too-many-args — are pre-existing on develop in files this PR does not touch).
• Real Docker installs against the actual test-image Dockerfiles: debian/bookworm pg17 & pg18, ubuntu/noble pg16/17/18 all install citus 14.1.0 + hll 2.19 + topn 2.7.0 cleanly → deb/ubuntu install legs green.
• RPM signing verifier proven by negative control: signed → True, --delsign unsigned → False, re-signed (RSAHEADER) → True.

Known-red, by design (not a merge gate)

The rpm test_execution install legs (el/9, ol/9 × PG16/17/18) stay RED: the already-published community RPMs are unsigned per-package while the repo enforces gpgcheck=1. This PR fixes go-forward signing in the pipeline; turning those legs green additionally requires a one-time packaging-side re-sign/re-upload backfill of the published RPMs — a separate track, not fixable in this repo. Deb legs authenticate via signed repo metadata and are green.

Note

This folds in #408 (cherry-picked as a6d04f8); #408 can be closed in favor of this PR.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com