Add Grafana Pyroscope Continuous Support

[gpreviatti]

TL;DR: Add OSS Pyroscope service to Docker Compose, configure WebApp Dockerfiles to include Pyroscope native binaries, set environment variables for profiling types, integrate Pyroscope datasource into Grafana, and enable profiles in Alloy.

Steps

Phase 1: Docker Compose Infrastructure (can run in parallel with Phase 2)

1. Add Pyroscope service to templates/Bff/docker-compose-local.yml

   • Service: pyroscope image pyroscope/pyroscope:latest (or pinned version like 0.14.2)
   • Port: 4040 (HTTP API)
   • Volumes: Mount data directory for profile persistence
   • Environment: Basic config, CORS enabled for Grafana access

2. Add Pyroscope service to templates/Full/docker-compose-local.yml

   • Identical to BFF setup

Phase 2: WebApp Dockerfile Configuration (can run in parallel with Phase 1)

3. Update templates/Bff/Dockerfile to include Pyroscope native binaries

   • Add multi-stage build or layer to copy Pyroscope.Profiler.Native.so and Pyroscope.Linux.ApiWrapper.x64.so from pyroscope/pyroscope-dotnet:0.14.2-glibc image
   • Ensure binaries are in /dotnet directory (required by Pyroscope)
   • Set LD_LIBRARY_PATH in runtime environment

4. Update templates/Full/Dockerfile with identical Pyroscope binary setup

Phase 3: Environment Variables & Runtime Configuration (depends on Phase 2)

5. Add Pyroscope environment variables to templates/Bff/docker-compose-local.yml

   • In WebApp service section:
     • PYROSCOPE_APPLICATION_NAME=hexagonal.solution.template.bff
     • PYROSCOPE_SERVER_ADDRESS=http://pyroscope:4040 (internal container network)
     • PYROSCOPE_PROFILING_ENABLED=1
     • PYROSCOPE_PROFILING_CPU_ENABLED=1
     • PYROSCOPE_PROFILING_WALLTIME_ENABLED=1
     • PYROSCOPE_PROFILING_ALLOCATION_ENABLED=1
     • PYROSCOPE_PROFILING_LOCK_ENABLED=1
     • PYROSCOPE_PROFILING_EXCEPTION_ENABLED=1
     • PYROSCOPE_PROFILING_HEAP_ENABLED=1
     • CLR profiler environment variables (CORECLR_ENABLE_PROFILING, CORECLR_PROFILER, etc.)
     • Diagnostics variables for .NET 8+ compatibility (DOTNET_EnableDiagnostics, DOTNET_EnableDiagnostics_Profiler)

6. (Optional) Add to MockApi service if want to profile mock API responses

7. Update templates/Full/docker-compose-local.yml with Full template service names

Phase 4: Grafana Integration (depends on Phase 1)

8. Add Pyroscope datasource to templates/Bff/scripts/grafana/datasources.yaml

   • Type: grafana-pyroscope-datasource
   • URL: http://pyroscope:4040
   • Default datasource for profiles: yes
   • Enable service graph integration if supporting (optional but recommended)

9. Add Pyroscope datasource to templates/Full/scripts/grafana/datasources.yaml

Phase 5: Alloy Configuration (depends on Phase 1)

10. Update templates/Bff/scripts/grafana/config.alloy to forward profiles

    • Add discovery of profiles (if Alloy supports); Pyroscope can act as standalone receiver
    • Alternatively: Document that Pyroscope runs independently and apps send profiles directly via environment variables

11. Update templates/Full/scripts/grafana/config.alloy similarly

Phase 6: Documentation & Verification (final step)

12. Update templates/Bff/Readme.md

    • Add Pyroscope section explaining what profiling types are enabled
    • Document how to access Pyroscope UI (http://localhost:4040)
    • Explain labels visible in profiles (application name, service, environment)
    • Link to Grafana Pyroscope documentation for interpretation

13. Update templates/Full/Readme.md similarly

14. Verification checklist:

    • Pyroscope service starts and is healthy in Docker Compose
    • WebApp Dockerfile builds successfully with Pyroscope binaries included
    • WebApp container starts without profiler errors (check logs)
    • Profiles appear in Pyroscope UI at http://localhost:4040
    • Grafana Pyroscope datasource connects and displays profiles
    • All profiling types (CPU, memory, lock contention, etc.) show data
    • Profiles include proper labels and context
    • No performance degradation in baseline metrics

Relevant Files

• templates/Bff/docker-compose-local.yml — Add Pyroscope service + environment variables
• templates/Bff/scripts/grafana/datasources.yaml — Add Pyroscope datasource
• templates/Bff/scripts/grafana/config.alloy — Document profiles path (if applicable)
• templates/Bff/Readme.md — Document Pyroscope usage and access
• templates/Full/docker-compose-local.yml — Add Pyroscope service + Full service names
• templates/Full/scripts/grafana/datasources.yaml — Add Pyroscope datasource
• templates/Full/scripts/grafana/config.alloy — Document (if applicable)
• templates/Full/Readme.md — Document usage

Decisions

• Native Profiler Only: Using Pyroscope .NET native profiler (not SDK integration) for real-time CPU/memory/lock/exception profiling without code changes
• Docker-Only Configuration: Pyroscope only in docker-compose-local.yml; users doing local development without Docker won't profile (acceptable trade-off)
• All Profiling Types Enabled: CPU, wall time, allocations, lock contention, exceptions, live heap (requires .NET 7+) all enabled by default; can be toggled per deployment via environment variables
• Standalone Pyroscope Service: Runs independently; apps connect directly via environment variables, no Alloy passthrough needed
• Same Config in BFF & Full: Identical Pyroscope setup for consistency, reducing future drift

Further Considerations

1. Build Time: Copying Pyroscope binaries in Dockerfile increases image layer size and build time by ~10-15MB. Trade-off: acceptable for dev environments, consider multi-stage build for production.
2. Linux Requirement: Pyroscope only works on Linux. Windows/Mac developers may want alternative profiling. Consider documenting fallback tools.
3. Profile Retention: Pyroscope keeps profiles in memory; add data volume to docker-compose for persistence if needed between container restarts.