Skip to content

feat(profiling): internal metrics for overhead#3616

Merged
morrisonlevi merged 6 commits intomasterfrom
levi/stack-walk-metric
Feb 4, 2026
Merged

feat(profiling): internal metrics for overhead#3616
morrisonlevi merged 6 commits intomasterfrom
levi/stack-walk-metric

Conversation

@morrisonlevi
Copy link
Collaborator

@morrisonlevi morrisonlevi commented Feb 2, 2026

Description

This is an alternative to #3591. Instead of putting the time we spend stack walking into the timeline and flamegraph, we instead aggregate it into counters, which we emit into internal metrics. This addresses some of the concerns I received about that PR while still providing something we believe is useful: the ability to understand our overhead for a specific profile. This internal metric could be re-exported in the future if we desire to do per-organization or other type of aggregation, for example, to determine which customers are hitting the highest amounts of overhead.

In addition to collecting CPU time walking the stack, this also aggregates CPU time spent in our two background threads, ddprof_time and ddprof_upload. The former is a bit of a misnomer because it also aggregates CPU samples, it doesn't just track time.

Manually verified it's working (practically idle service), this is on a slightly older version of the PR so it looks a little different now:

Screenshot 2026-02-03 at 11 15 04 AM

Reviewer checklist

  • Test coverage seems ok.
  • Appropriate labels assigned.

@datadog-official

This comment has been minimized.

@morrisonlevi morrisonlevi added the profiling Relates to the Continuous Profiler label Feb 2, 2026
@codecov-commenter
Copy link

codecov-commenter commented Feb 3, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.11%. Comparing base (32aaf0a) to head (76483bb).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3616      +/-   ##
==========================================
- Coverage   62.21%   62.11%   -0.11%     
==========================================
  Files         141      141              
  Lines       13387    13387              
  Branches     1753     1753              
==========================================
- Hits         8329     8315      -14     
- Misses       4260     4273      +13     
- Partials      798      799       +1     

see 3 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 32aaf0a...76483bb. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pr-commenter
Copy link

pr-commenter bot commented Feb 3, 2026

Benchmarks [ profiler ]

Benchmark execution time: 2026-02-03 19:57:47

Comparing candidate commit aed7a13 in PR branch levi/stack-walk-metric with baseline commit 32aaf0a in branch master.

Found 2 performance improvements and 3 performance regressions! Performance is the same for 24 metrics, 7 unstable metrics.

scenario:php-profiler-timeline-memory-with-profiler

  • 🟥 execution_time [+54.337ms; +75.399ms] or [+5.855%; +8.124%]
  • 🟩 cpu_usage_percentage [-4.885%; -3.086%]

scenario:php-profiler-timeline-memory-with-profiler-and-timeline

  • 🟥 cpu_system_time [+38.635ms; +61.757ms] or [+7.237%; +11.568%]
  • 🟥 execution_time [+93.618ms; +115.961ms] or [+7.583%; +9.393%]
  • 🟩 cpu_usage_percentage [-6.534%; -5.345%]

@morrisonlevi morrisonlevi marked this pull request as ready for review February 3, 2026 21:00
@morrisonlevi morrisonlevi requested a review from a team as a code owner February 3, 2026 21:00
realFlowControl and others added 3 commits February 4, 2026 11:40
…ddprof_upload` for current profile exported

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
@realFlowControl
Copy link
Member

I was reviewing this and made a few additions that build on your work in #3618, let me know what you think

@morrisonlevi morrisonlevi merged commit 6843f96 into master Feb 4, 2026
410 of 511 checks passed
@morrisonlevi morrisonlevi deleted the levi/stack-walk-metric branch February 4, 2026 15:42
@github-actions github-actions bot added this to the 1.17.0 milestone Feb 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

profiling Relates to the Continuous Profiler tracing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants