Implement promotion-based CI/CD pipeline (TEST → SANDBOX → PROD)

[arielr-lt]

Context

Per CI/CD principles defined by the Design Document 2026:

• TEST — automatic continuous deployment from master. Free-for-all for developer testing; communication required between engineers.
• SANDBOX — manual promotion of the last-good build from TEST. No rebuilds, only promotions. Public bake area for features to "smoke" before going live.
• PRODUCTION — manual promotion of the last-good build from SANDBOX. No rebuilds, only promotions.

Underlying principle: a single immutable image artifact flows TEST → SANDBOX → PROD, gated by manual approvals at each promotion step.

Goals

• One build per merge to master. The same image digest is what runs in TEST, SANDBOX, and PROD.
• Promotions are explicit, audited, manual, and reversible.
• No path by which a code merge becomes a live PROD release without a human action.
• All deployment manifests reference immutable tags only.

Proposed pipeline

1. build.yaml — Build & publish

• Trigger: push to master (and optional workflow_dispatch for manual rebuild).
• Output: single multi-arch image pushed to ECR.
• Tags: immutable only — YYYY.MM.DD.NNNN and the 40-char commit SHA. No floating branch tags.
• On success: triggers deploy-test.yaml with the new tag.

2. deploy-test.yaml — Auto-deploy to TEST

• Trigger: workflow_call from build.yaml (also workflow_dispatch for ad-hoc redeploys of any immutable tag).
• Action: kubectl set image against credreg-test for main-app and worker-app. Wait for rollout. Run DB migrations job.
• Gate: CI test suite must have passed for the commit (configurable).
• Slack notification: posts the deployed tag, digest, and a link to the promote-to-sandbox workflow.

3. promote-sandbox.yaml — Promote TEST → SANDBOX

• Trigger: workflow_dispatch only. No inputs.
• Action: read the image digest currently running in credreg-test/main-app, apply the same digest (resolved to its date.build tag) to credreg-sandbox. Wait for rollout. Run DB migrations.
• Guards:
  • TEST rollout must be healthy.
  • Resolved digest must differ from sandbox's current.
• Slack notification: posts the promoted tag and a link to the promote-to-prod workflow.

4. promote-prod.yaml — Promote SANDBOX → PROD

• Trigger: workflow_dispatch only. No inputs.
• Action: read the image digest currently running in credreg-sandbox/main-app, apply the same digest to credreg-prod. Wait for rollout. Run DB migrations.
• Guards: sandbox rollout healthy; differs from prod's current; optional minimum bake time in SANDBOX.
• Slack notification: posts the deployed tag and digest.

5. rollback.yaml — Rollback any env

• Trigger: workflow_dispatch. Inputs: target environment, target tag (optional — defaults to previous).
• Action: kubectl rollout undo (or explicit set image to a chosen prior tag). Migrations not re-run.

Cluster-side requirements

• Deployment manifests for all three envs reference immutable tags. Image field is mutated by kubectl set image from the workflows; manifests in git carry a placeholder or the most recently promoted tag.
• imagePullPolicy: IfNotPresent on main-app and worker-app deployments in all envs.

Acceptance criteria

• build.yaml triggers only on master and produces only immutable tags
• deploy-test.yaml exists and auto-deploys every successful master build to credreg-test
• promote-sandbox.yaml exists, requires manual dispatch, and promotes the running TEST digest
• promote-prod.yaml exists, requires manual dispatch + N-reviewer approval, and promotes the running SANDBOX digest
• rollback.yaml exists and works for any env
• Manifests in k8s-manifests-{test,sandbox,prod}/ use immutable tags + IfNotPresent
• Slack notifications fire for build, TEST deploy, SANDBOX promotion, PROD promotion, and rollback
• Documentation in docs/ (or top-level DEPLOY.md) describing the flow and how to use each workflow

Open questions

1. feature-branch deploy workflow — currently retargets feature branches to credreg-test. Keep as-is, or fold into this new model?