Add OpenTelemetry metrics for package resources size

[JamesMcDermott]

Title

Feat: Add OpenTelemetry metrics for package resources size

---

Description

• What changed:
  • new OpenTelemetry metrics: histogram- and gauge-type metrics to track package revision resource sizes
  • minimal Prometheus monitoring stack for manual testing
• Why it's needed:
  • provides observability into package resource sizes across Porch operations, enabling monitoring and alerting on package growth.
• How it works:
  • New metrics (porch_package_size_bytes_bucket, porch_package_size_bytes_count, porch_package_size_bytes_sum, porch_package_size_bytes_total) are recorded in Porch flows that update package revision resources:
    • create
    • delete
    • discover/sync from a registered repository
    • delete on unregistering a repository
    • direct update of PackageRevisionResources in rpkg push

---

Related Issue(s)

• Closes Expose package resource file size as a Prometheus metric #991
• Prometheus setup copied from WIP: Refactor Performance Tests and add Grafana, Pyroscope and Prometheus monitoring stack #561

---

Type of Change

• Bug fix
• New feature
• Enhancement
• Refactor
• Documentation
• Tests
• Other: ________

---

Checklist

• Code follows project style guidelines
• Self-reviewed changes
• Tests added/updated
• Documentation added/updated
• All tests and gating checks pass

---

Testing Instructions (Optional)

1. Deploy the monitoring stack using scripts/deploy-monitoring.sh
2. Prometheus GUI will be available at http://localhost:9092
3. Create/delete package revisions and verify new porch_package_size_bytes_* metrics can be queried in Prometheus
4. Run go test ./internal/telemetry/... ./test/e2e/api/ to execute unit and e2e tests
5. Verify new porch_package_size_bytes_* metrics for e2e tests' packages can be queried in Prometheus

---

Additional Notes (Optional)

• Review notes: The internal/otel package has been refactored into internal/telemetry

---

AI Disclosure

• I have used AI in the creation of this PR.

If so, please describe how:

• Kiro/Amazon Q to:
  • generate the pull request description from the commit message
  • determine locations to add new metrics in code picked from PR 561
  • generate new unit tests to cover code updates
  • suggest rearrangement in internal/telemetry/otel.go.setupMetrics() to properly gate off Prometheus exporter setup

[netlify]

✅ Deploy Preview for kpt-porch ready!

Name	Link
🔨 Latest commit	cf910a7
🔍 Latest deploy log	https://app.netlify.com/projects/kpt-porch/deploys/6a2fbf6ad5032c0008c90ec7
😎 Deploy Preview	https://deploy-preview-1018--kpt-porch.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR introduces a new internal/telemetry package (replacing internal/otel) that provides a unified OpenTelemetry setup with proper lifecycle management, adds Prometheus-exporter support, and instruments package revision resource size as a new porch_package_size_bytes metric. Deployment manifests are extended with metrics ports/services and a new Prometheus/Grafana monitoring stack is added.

Changes:

• New internal/telemetry package with SetupOpenTelemetry returning lifecycle-managed OTelResources and recording package size histograms/gauges.
• Wiring of the new telemetry setup and deferred shutdown across cmd/porch, controllers, func/server, and func/wrapper-server, along with metric recording calls in dbcache.
• Deployment additions: metrics ports/services on porch components, Prometheus/Grafana kpt package, and deploy/cleanup script; new e2e test for the package size metric.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
internal/telemetry/otel.go	New unified OTel setup with explicit shutdown lifecycle and optional Prometheus HTTP server.
internal/telemetry/metrics.go	New porch_package_size_bytes histogram + gauge instruments and recorder.
internal/telemetry/metrics_test.go	Tests for RecordPackageSizeUpdate including nil-instrument guards.
internal/otel/otel.go	Old package removed (replaced by internal/telemetry).
internal/otel/otel_test.go	Tests moved to telemetry package; new Prometheus server tests added.
go.mod	Promote otel/exporters/prometheus from indirect to direct dependency.
cmd/porch/main.go	Use new telemetry package with deferred shutdown.
controllers/main.go	Move OTel init out of newManager; add deferred shutdown.
func/server/server.go	Switch to new telemetry package; add deferred shutdown.
func/wrapper-server/main.go	Switch to new telemetry package; add deferred shutdown.
pkg/cache/dbcache/dbpackage.go	Record package size on delete.
pkg/cache/dbcache/dbrepository.go	Record package size on close-draft and delete.
pkg/cache/dbcache/dbreposync.go	Record package size on external PR cache/delete.
deployments/porch/2-function-runner.yaml	Add metrics port to function-runner service.
deployments/porch/3-porch-server.yaml	Add 9464 metrics port and service entry.
deployments/porch/9-controllers.yaml	Add Service exposing controller metrics port.
deployments/porch/22-function-templates.yaml	Add metrics env/port for wrapper-server templates.
deployments/metrics/Kptfile	New kpt package for monitoring deployment.
deployments/metrics/prometheus-deployment.yaml	New Prometheus deployment, RBAC and Service.
deployments/metrics/grafana-deployment.yaml	New Grafana deployment, datasources and Service.
deployments/metrics-resources/prometheus-config.yaml	Prometheus scrape configuration for porch components.
scripts/deploy-monitoring.sh	New script to deploy/cleanup monitoring stack via kpt.
docs/.../opentelemetry.md	Documentation for new metrics.
test/e2e/api/metrics_test.go	Fix regex globs, switch to suite assertions, add package-size metric test.

[Copilot]

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 10 comments.

[Copilot]

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 9 comments.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 7 comments.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 9 comments.

[Copilot]

Pull request overview

Copilot reviewed 27 out of 27 changed files in this pull request and generated 9 comments.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 7 comments.

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 7 comments.

[Copilot]

Pull request overview

Copilot reviewed 29 out of 29 changed files in this pull request and generated 8 comments.

[liamfallon]

Do we need this file? Members can retrigger CI on the Github UI.

[JamesMcDermott]

removed

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 12 comments.