ci(tsan): timeout-bound the race gate — seeded-race UB hung 1/3 of overnight runs for 6h each#451
Merged
Merged
Conversation
…ly hang Overnight 2026-07-05, 5 of ~15 CI runs hung in tsan_seeded_race.eigs and burned the 6h default job timeout before being auto-cancelled — including code-touching runs (#443) and pure docs pushes, so the hang is content-independent. All 9 race-free slice tests had passed in every hung run; the hang is downstream of the DELIBERATE lost-update race (two threads set_at the same slot; both can decref the displaced element — double-decref corruption may legally spin). 0/30 local repros on the 2-core box vs ~1/3 on 4-core runners — consistent with the CI-TSan-sees-more rule. The gate now bounds every run with timeout (default 120s, TSAN_RUN_TIMEOUT to override): - a race-free slice test that times out FAILs loudly in minutes as a liveness bug (was: silent 6h runner burn), - the seeded race may be killed after its warnings are counted — TSan emits warnings as they happen, so detection (the property the gate asserts) survives the kill, - ci.yml adds timeout-minutes: 20 as the job-level backstop. tsan_warnings returns via globals, deliberately not echo-and-capture: w=$(fn) runs fn in a subshell and drops LAST_RC, silently disabling the hang branch — caught by a planted-hang probe, which now validates both branches (clean pass exit 0, planted hang exit 1). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The incident. Overnight 2026-07-05, 5 of ~15 CI runs hung in the TSan job and burned the 6h default job timeout before auto-cancel — including the #443 VM fix and pure docs pushes, so the hang is content-independent. In every hung run all 9 race-free slice tests had already passed; the hang is always in
tsan_seeded_race.eigs, the deliberately-racy planted fault. Orphan-process teardown in the logs shows the interpreter still alive at kill time.Mechanism (hypothesis, consistent with the evidence). The seeded race does unsynchronized
set_aton one slot from two threads: both can read the same displaced element and both decref it — double-decref corruption downstream of deliberate UB may legally spin forever. 0/30 local repros on the 2-core dev box vs ~1/3 on 4-core runners, consistent with the CI-TSan-sees-more rule. No evidence of any liveness bug in race-free code.The fix is gate design, not runtime. An intentionally-UB program may hang; the gate must bound it:
timeout(default 120s,TSAN_RUN_TIMEOUToverrides)timeout-minutes: 20on the job as backstopPlanted-fault validation. The first draft returned
LAST_RCfrom a command-substitution subshell — the hang branch could never fire; a planted-hang probe caught it. Final version validated both ways: clean gate 10/10 exit 0; planted hang →FAIL: … HUNG (killed after 3s)exit 1.🤖 Generated with Claude Code