Reproduce flaky: SymDB scheduler thread logger.debug#5942
Draft
p-datadog wants to merge 1 commit into
Draft
Conversation
Hypothesis: when raw_logger.debug is stubbed to raise (as in PR #5871's "does not propagate exceptions when logger.debug itself raises" test), the scheduler thread's @logger.debug calls in extract_and_upload, scheduler_loop's rescue, and start_upload's rescue have no inner rescue and terminate the thread. Component#shutdown!'s @scheduler_thread.join then re-raises the exception in the test thread (standard Thread#join semantics, all Ruby versions). Forces the race deterministically by holding the scheduler in extract_all on a Queue, activating the stub, then releasing. Reproduces on Ruby 2.6.10, 3.0.7, 3.1.7, 3.3.10, 3.4.8 locally.
|
This was referenced Jun 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Reproducer for the flaky test
does not propagate exceptions when logger.debug itself raisesatspec/datadog/symbol_database/component_spec.rb:752, observed on Ruby 2.6 in PR #5798. Forces the underlying race condition deterministically so CI fails in <1s on every Ruby version.Do not merge — this PR exists for CI validation only.
Motivation:
Flaky test Ruby 2.6 / build & test (standard) [0] on 2026-06-15.
Hypothesis:
The PR #5871 test name "does not propagate exceptions when logger.debug itself raises" expresses a property of
@logger.debugeverywhere inDatadog::SymbolDatabase::Component, but onlyinstall_hot_load_hookhas the inner-rescue protection. Whenallow(raw_logger).to receive(:debug).and_raise(...)is set up, every@logger.debugcall in the component raises — including on the scheduler thread, where there is no inner rescue:extract_and_uploadsuccess-path log atlib/datadog/symbol_database/component.rb:500extract_and_upload's rescue handler at line 511scheduler_loop's rescue handler at line 458start_upload's rescue handler at line 202If
raw_logger.debugis stubbed to raise while the scheduler thread is insideextract_and_upload, the exception escapes line 500, then escapes line 511's rescue (calling@logger.debugagain), then escapes line 458's rescue (calling@logger.debugagain), terminating the scheduler thread. The test'sensurecallscomponent.shutdown!, which invokes@scheduler_thread&.join(5).Thread#joinre-raises the joined thread's unhandled exception in the caller — standard Ruby semantics across every supported version. The exception surfaces in the test thread with the scheduler thread's backtrace, and RSpec records the failure.Not Ruby-2.6-specific structurally — every link in the chain (missing rescues,
Thread#joinre-raising, etc.) is the same on all MRI Rubies. The 2.6-only CI flake is a timing artefact:extract_all'sObjectSpacewalk is slow enough on 2.6 in a loaded test environment thatwait_for_idle(timeout: 5)regularly times out, leaving the scheduler thread insideextract_and_uploadwhen the test sets up the stub.Reproducer technique:
A new spec at
spec/datadog/symbol_database/logger_boom_reproducer_spec.rbforces the race deterministically with aQueue:stub_constoverridesEXTRACT_DEBOUNCE_INTERVALto 0 so the scheduler fires immediately.allow_any_instance_of(Extractor).to receive(:extract_all)holds the scheduler insideextract_allon a Queue until the test releases it.allow(raw_logger).to receive(:debug).and_raise(...).extract_all. The scheduler proceeds to line 500 and raises.expect { component.shutdown! }.not_to raise_error— without the fix,shutdown!'sThread#joinre-raisesRuntimeError('logger boom').Verified failing locally on Ruby 2.6.10, 3.0.7, 3.1.7, 3.3.10, 3.4.8.
Change log entry
None.
How to test the change?
CI should show the new spec failing deterministically on every Ruby version. If it doesn't fail (especially on Rubies where the original CI flake is rarely seen), the hypothesis is wrong.
Companion PRs: