gitops: add the in-cluster monitoring stack (Prometheus/Grafana) GitOps-native

[ausbru87]

Goal

Add the new in-cluster monitoring stack (Prometheus/Grafana) GitOps-native from
day one, so there is no CLI-installed release to adopt later. Plan:
docs/plans/gitops-adoption.md.

Approach

This is greenfield, not an adoption. Install through the GitOps controller
directly.

Tasks

• Choose kube-prometheus-stack (Prometheus, Alertmanager, Grafana, kube-state-metrics, node-exporter) installed via an Argo Application (Helm source), namespace monitoring.
• Mirror all required images into ECR (no pull-through in GovCloud); add them to scripts/images.txt and run scripts/mirror-images.sh.
• Pin chart + app versions in versions.lock.yaml; commit a values file under deploy/platform/monitoring/.
• Persist Prometheus/Grafana on the gp3 default StorageClass (encrypted).
• Source the Grafana admin secret from ASM via ESO (an ExternalSecret), never git.
• If Grafana is exposed, add a grafana.usgov.coderdemo.io Ingress (ingressClassName: nginx) under the existing *.usgov.coderdemo.io ACM cert, plus a Route53 alias to the NLB.
• Keep coder-workspaces scraping consistent with the demo posture; document what is and is not scraped.

Notes

Observability was previously out of scope (STATUS.md); this issue introduces it under GitOps cleanly. Place it after the secret plumbing (ESO) is adopted so the Grafana admin secret flows from ASM.

Generated by Coder Agents.