[User Story] Secure JIT Backup Credentials

[noahwhite]

Story Summary

As a Site Reliability Engineer, I want the nightly backup cron job to fetch R2 credentials dynamically from the Aembit proxy, so that no long-lived S3 keys are stored on the server.

Phase: 3 - Hardened Ongoing Maintenance

---

✅ Acceptance Criteria

• Nightly rclone script fetches R2 credentials from Aembit proxy at runtime
• No R2 credentials stored on disk or in environment files
• Backup completes successfully using JIT credentials
• Aembit audit logs show distinct request for R2 credential provider at scheduled time
• Credentials are short-lived (valid only for backup window)
• Failed credential fetch prevents backup and alerts (doesn't use stale creds)

---

📝 Additional Context

Backup Flow with JIT Credentials

systemd timer triggers (2:00 AM daily)
    │
    ├─► ghost-backup.service starts
    │
    ├─► Fetch R2 credentials from Aembit proxy
    │   curl -sf http://localhost:8080/credentials/r2-backup
    │   Returns: { access_key_id, secret_access_key, expires_at }
    │
    ├─► Configure rclone with temporary credentials
    │   - Write to temp config in /run/ghost/rclone.conf
    │   - Or use environment variables
    │
    ├─► Execute rclone sync
    │   rclone sync /var/mnt/storage r2:ghost-backups-dev/
    │
    └─► Cleanup
        - Remove temp rclone config
        - Credentials expire automatically

Backup Script Update

#!/bin/bash
# /opt/bin/ghost-backup.sh

set -euo pipefail

PROXY_URL="http://localhost:8080"
BACKUP_BUCKET="${BACKUP_BUCKET:?}"

# Fetch JIT credentials from Aembit
log "Fetching R2 credentials from Aembit..."
CREDS=$(curl -sf "$PROXY_URL/credentials/r2-backup")

export AWS_ACCESS_KEY_ID=$(echo "$CREDS" | jq -r '.access_key_id')
export AWS_SECRET_ACCESS_KEY=$(echo "$CREDS" | jq -r '.secret_access_key')

# Verify credentials were fetched
if [[ -z "$AWS_ACCESS_KEY_ID" ]]; then
  log "ERROR: Failed to fetch R2 credentials"
  exit 1
fi

# Run backup with JIT credentials
rclone sync /var/mnt/storage "r2:$BACKUP_BUCKET" \
  --s3-provider Cloudflare \
  --s3-endpoint "https://${CF_ACCOUNT_ID}.r2.cloudflarestorage.com" \
  --exclude "ghost-compose/.env.*"

# Credentials cleared when script exits (environment vars)
log "Backup completed"

Aembit Credential Provider Configuration

resource "aembit_credential_provider" "r2_backup" {
  name = "R2 Backup Credentials"
  type = "cloudflare_r2"
  
  cloudflare_r2 {
    account_id    = var.cloudflare_account_id
    access_key_id = var.r2_access_key_id  # Stored in Aembit, not on instance
    secret_key    = var.r2_secret_key
  }
}

resource "aembit_access_policy" "backup_r2" {
  client_workload_id     = aembit_client_workload.ghost_dev.id
  server_workload_id     = aembit_server_workload.r2.id
  credential_provider_id = aembit_credential_provider.r2_backup.id
  
  # Time-based restriction (see GHO-71)
}

Security Benefits

Without JIT	With JIT
R2 keys in rclone.conf on disk	No keys on disk
Keys valid 24/7	Keys valid only during backup
Compromised server = permanent access	Compromised server = limited window

Dependencies

• GHO-69: Docker secrets integration (Aembit proxy running)
• GHO-62: rclone sysext package (from backup epic)

Connection to Backup Epic

This story provides the secure credential delivery mechanism for the backup epic (GHO-62, GHO-63, GHO-64). After this story:

• GHO-62 (rclone sysext) remains unchanged
• GHO-63 (backup implementation) uses JIT credentials from this story
• GHO-64 (restore procedure) documentation updated

---

📦 Definition of Ready

• Acceptance criteria defined
• Blocked by GHO-69 (Docker secrets / Aembit proxy working)
• Story is estimated
• Team has necessary skills and access
• Priority is clear
• Business value understood

---

✅ Definition of Done

• All acceptance criteria met
• Nightly backup runs successfully with JIT credentials
• Aembit audit log shows credential request at backup time
• No R2 credentials on disk (verified)
• Backup epic stories updated to reference this approach

[linear]

GHO-70