Skip to content

Fix flaky: CpuAndWallTimeWorker "always simulates signal delivery"#5980

Closed
p-datadog wants to merge 1 commit into
masterfrom
fix-flaky-cwtw-simulates-signal-delivery
Closed

Fix flaky: CpuAndWallTimeWorker "always simulates signal delivery"#5980
p-datadog wants to merge 1 commit into
masterfrom
fix-flaky-cwtw-simulates-signal-delivery

Conversation

@p-datadog

@p-datadog p-datadog commented Jul 1, 2026

Copy link
Copy Markdown
Member

What does this PR do?

Fixes the flaky spec spec/datadog/profiling/collectors/cpu_and_wall_time_worker_spec.rb:678 (CpuAndWallTimeWorker#start when using the no signals workaround always simulates signal delivery) by waiting on a reliable condition instead of samples.any?.

Motivation:

Observed flaking on Ruby 2.7 / build & test (standard) [0] (commit ab2601db), an unrelated PR. In that job profiling:main ran 3 times and this spec failed once and passed twice on the same commit; it passed on all other Ruby versions.

Failure:

Failure/Error: expect(sample_count).to be > 0
  expected: > 0
       got:   nil
# ./spec/datadog/profiling/collectors/cpu_and_wall_time_worker_spec.rb:699

Root cause:

recorder.serialize! drains the recorder, so the all_samples captured inside try_wait_until is only the single batch drained by the iteration whose wait condition first became truthy. The condition samples.any? becomes truthy on the first surviving (non-GC, non-overhead) sample of any thread.

Since the on-serialize flush was added in #5955, every serialize! emits catch-up samples for the profiler-internal threads (the worker and the idle-sampling helper). Those samples are not GC and not profiler overhead, so they survive the filter. When a serialize! happens before the worker's first per-tick sample has hit the main thread, the drained batch contains only profiler-internal-thread samples (and/or another application thread) with no main-thread sample. samples_for_thread(all_samples, Thread.current) is then empty, [].reduce(:+) returns nil, and expect(sample_count).to be > 0 fails with got: nil.

This is the same unreliable wait pattern that #5955 migrated away from in the sibling specs of this file (switching to stats.fetch(:cpu_sampled) > 0), but this spec was not migrated.

Fix:

Wait until the drained batch actually contains a main-thread sample, so the batch kept as all_samples is guaranteed to have one. Test-synchronization change only; the assertions are unchanged (no weakening).

Change log entry

None.

How to test the change?

Reproduced deterministically and verified locally (Ruby 3.2.3):

  • Naturally, on stock Ruby, by adding background CPU-bound threads (so the worker samples them instead of the main thread at the first tick): ~1-2% failure rate on the unmodified spec; 0 failures with this fix.
  • Deterministically, on a locally patched Ruby that delays the profiler-internal threads' startup so the first serialize! flushes only profiler-internal-thread samples: the unmodified spec fails 20/20 with the exact got: nil; with the knob off it passes; with this fix applied it passes 30/30.

Root cause: `recorder.serialize!` drains the recorder, so the `all_samples`
captured inside `try_wait_until` is only the single batch drained by the
iteration whose wait condition first became truthy. The condition was
`samples.any?`, which becomes truthy on the first surviving (non-GC,
non-overhead) sample of ANY thread. When that first batch contains a sample
for another thread but no sample for the main thread,
`samples_for_thread(all_samples, Thread.current)` is empty, so
`[].reduce(:+)` returns nil and `expect(sample_count).to be > 0` fails with
"got: nil".

This is the same unreliable wait pattern that commit 3431b01 migrated
away from in the sibling specs of this file (switching to an observable
stat), but this spec was not migrated.

Fix: wait until the drained batch actually contains a main-thread sample, so
the batch kept as `all_samples` is guaranteed to have one. This is a test
synchronization fix; the assertions are unchanged.

Co-Authored-By: Claude <noreply@anthropic.com>
@p-datadog p-datadog added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Jul 1, 2026
@dd-octo-sts dd-octo-sts Bot added the dev/testing Involves testing processes (e.g. RSpec) label Jul 1, 2026
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Jul 1, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 90.02% (-0.03%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 56ee2b9 | Docs | Datadog PR Page | Give us feedback!

@p-datadog p-datadog marked this pull request as ready for review July 1, 2026 22:53
@p-datadog p-datadog requested a review from a team as a code owner July 1, 2026 22:53
@p-datadog p-datadog requested a review from Copilot July 1, 2026 22:53
@p-datadog p-datadog requested a review from a team as a code owner July 1, 2026 22:53
@p-datadog

Copy link
Copy Markdown
Member Author

@codex review

1 similar comment
@p-datadog

Copy link
Copy Markdown
Member Author

@codex review

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a flaky profiling spec (CpuAndWallTimeWorker#start when using the no signals workaround always simulates signal delivery) by making the wait condition align with what the test later asserts on: the presence of main-thread samples in the specific batch returned by recorder.serialize! (which drains the recorder).

Changes:

  • Update the try_wait_until condition to wait for at least one main-thread sample (samples_for_thread(..., Thread.current).any?) instead of any sample at all, preventing the drained batch from containing only profiler-internal-thread samples.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 56ee2b96a8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ivoanjo

ivoanjo commented Jul 2, 2026

Copy link
Copy Markdown
Member

Ahh my bad I missed this one and basically did the same in #5983 . Sorry for the duplicated work :(

We can close off this one now.

@ivoanjo ivoanjo closed this Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos dev/testing Involves testing processes (e.g. RSpec)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants