Skip to content

Reproduce flaky: SymDB scheduler thread logger.debug#5942

Draft
p-datadog wants to merge 1 commit into
masterfrom
reproduce-symdb-scheduler-logger-boom
Draft

Reproduce flaky: SymDB scheduler thread logger.debug#5942
p-datadog wants to merge 1 commit into
masterfrom
reproduce-symdb-scheduler-logger-boom

Conversation

@p-datadog

@p-datadog p-datadog commented Jun 23, 2026

Copy link
Copy Markdown
Member

What does this PR do?

Reproducer for the flaky test does not propagate exceptions when logger.debug itself raises at spec/datadog/symbol_database/component_spec.rb:752, observed on Ruby 2.6 in PR #5798. Forces the underlying race condition deterministically so CI fails in <1s on every Ruby version.

Do not merge — this PR exists for CI validation only.

Motivation:

Flaky test Ruby 2.6 / build & test (standard) [0] on 2026-06-15.

Hypothesis:

The PR #5871 test name "does not propagate exceptions when logger.debug itself raises" expresses a property of @logger.debug everywhere in Datadog::SymbolDatabase::Component, but only install_hot_load_hook has the inner-rescue protection. When allow(raw_logger).to receive(:debug).and_raise(...) is set up, every @logger.debug call in the component raises — including on the scheduler thread, where there is no inner rescue:

  • extract_and_upload success-path log at lib/datadog/symbol_database/component.rb:500
  • extract_and_upload's rescue handler at line 511
  • scheduler_loop's rescue handler at line 458
  • start_upload's rescue handler at line 202

If raw_logger.debug is stubbed to raise while the scheduler thread is inside extract_and_upload, the exception escapes line 500, then escapes line 511's rescue (calling @logger.debug again), then escapes line 458's rescue (calling @logger.debug again), terminating the scheduler thread. The test's ensure calls component.shutdown!, which invokes @scheduler_thread&.join(5). Thread#join re-raises the joined thread's unhandled exception in the caller — standard Ruby semantics across every supported version. The exception surfaces in the test thread with the scheduler thread's backtrace, and RSpec records the failure.

Not Ruby-2.6-specific structurally — every link in the chain (missing rescues, Thread#join re-raising, etc.) is the same on all MRI Rubies. The 2.6-only CI flake is a timing artefact: extract_all's ObjectSpace walk is slow enough on 2.6 in a loaded test environment that wait_for_idle(timeout: 5) regularly times out, leaving the scheduler thread inside extract_and_upload when the test sets up the stub.

Reproducer technique:

A new spec at spec/datadog/symbol_database/logger_boom_reproducer_spec.rb forces the race deterministically with a Queue:

  1. stub_const overrides EXTRACT_DEBOUNCE_INTERVAL to 0 so the scheduler fires immediately.
  2. allow_any_instance_of(Extractor).to receive(:extract_all) holds the scheduler inside extract_all on a Queue until the test releases it.
  3. While the scheduler is held, the test installs allow(raw_logger).to receive(:debug).and_raise(...).
  4. The test releases extract_all. The scheduler proceeds to line 500 and raises.
  5. expect { component.shutdown! }.not_to raise_error — without the fix, shutdown!'s Thread#join re-raises RuntimeError('logger boom').

Verified failing locally on Ruby 2.6.10, 3.0.7, 3.1.7, 3.3.10, 3.4.8.

Change log entry

None.

How to test the change?

CI should show the new spec failing deterministically on every Ruby version. If it doesn't fail (especially on Rubies where the original CI flake is rarely seen), the hypothesis is wrong.

Companion PRs:

Hypothesis: when raw_logger.debug is stubbed to raise (as in PR #5871's
"does not propagate exceptions when logger.debug itself raises" test),
the scheduler thread's @logger.debug calls in extract_and_upload,
scheduler_loop's rescue, and start_upload's rescue have no inner rescue
and terminate the thread. Component#shutdown!'s @scheduler_thread.join
then re-raises the exception in the test thread (standard Thread#join
semantics, all Ruby versions).

Forces the race deterministically by holding the scheduler in
extract_all on a Queue, activating the stub, then releasing.

Reproduces on Ruby 2.6.10, 3.0.7, 3.1.7, 3.3.10, 3.4.8 locally.
@p-datadog p-datadog added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Jun 23, 2026
@dd-octo-sts dd-octo-sts Bot added the dev/testing Involves testing processes (e.g. RSpec) label Jun 23, 2026
@datadog-prod-us1-3

datadog-prod-us1-3 Bot commented Jun 23, 2026

Copy link
Copy Markdown

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 19 Pipeline jobs failed

Check Pull Request CI Status | all-jobs-are-green   View in Datadog   GitHub Actions

Test Nix | Test Nix (aarch64-linux, 24.05)   View in Datadog   GitHub Actions

Test Nix | Test Nix (arm64-darwin, 24.05)   View in Datadog   GitHub Actions

View all 19 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 7afeca3 | Docs | Datadog PR Page | Give us feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos dev/testing Involves testing processes (e.g. RSpec)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants