test: feature_llmq_signing.py --spork21 intermittent timeout in wait_for_sigs

[thepastaclaw]

Summary

feature_llmq_signing.py --spork21 intermittently fails with a wait_until() timeout in wait_for_sigs(). The failure is timing-dependent and not reproducible — the same test passes on develop and the non---spork21 variant passes consistently on the same CI run.

Failure Details

Test: feature_llmq_signing.py --spork21
Error: AssertionError: Predicate '' not true after N seconds
Location: feature_llmq_signing.py:111 → wait_for_sigs(True, False, True, 15) (15s timeout)
Traceback:

File "feature_llmq_signing.py", line 111, in run_test
    wait_for_sigs(True, False, True, 15)
File "feature_llmq_signing.py", line 60, in wait_for_sigs
    self.wait_until(lambda: check_sigs(hasrecsigs, isconflicting1, isconflicting2), timeout = timeout)
AssertionError: Predicate '' not true after <timeout> seconds

Retry behavior: Failed all 3 CI retry attempts (attempt 1: 128s, attempt 2: 114s, attempt 3: 131s)

CI Log Links

• Failing run on PR fix: register shareman recovery interface in ActiveContext #7244: https://github.com/dashpay/dash/actions/runs/23441584574/job/68207659502
• Passing run on develop (same commit after merge): https://github.com/dashpay/dash/actions/runs/23505322590 — linux64_multiprocess-test / Test source passed

Evidence of Flakiness

1. The non---spork21 variant (feature_llmq_signing.py) passed on the exact same CI run
2. The --spork21 variant passed on develop after the PR was merged (CI run 23505322590)
3. PR fix: register shareman recovery interface in ActiveContext #7244 only modifies src/active/context.cpp (adds shareman->RegisterRecoveryInterface()) — no test files or LLMQ signing logic changed

Root Cause Analysis

The 15-second timeout in wait_for_sigs() appears insufficient for the multiprocess test environment under CI load. The --spork21 variant likely exercises a slightly different code path that takes marginally longer, hitting the timeout under resource contention.

Suggested Fix

Consider increasing the timeout in wait_for_sigs() calls within the --spork21 path, or using self.options.timeout_factor to scale timeouts appropriately for CI environments.

[thepastaclaw]

Another occurrence of this flaky test failure:

CI run: https://github.com/dashpay/dash/actions/runs/23780028947/job/69294959989
PR: #7251 (ci: convert clang-format diff check to informational PR comment — only changes .github/workflows/clang-diff-format.yml)
Build target: linux64_multiprocess
Date: 2026-03-31

The test failed 3/3 attempts with the same wait_until() timeout:

AssertionError: Predicate '' not true after N seconds

Other LLMQ tests also failed on first attempt during this run (feature_llmq_data_recovery, feature_llmq_simplepose, feature_llmq_dkgerrors) but passed on retry — suggesting general LLMQ test instability on ARM runners.

[thepastaclaw]

Another occurrence from PR #7242 (wallet deprecation + RPC type checking backport), but this one is unrelated to the PR changes.

• CI run: https://github.com/dashpay/dash/actions/runs/24064815059/job/70192274452
• Build target: linux64_multiprocess
• Date: 2026-04-07

The failure was again feature_llmq_signing.py --spork21, timing out in wait_for_sigs(True, False, True, 15) after retrying 3/3 times:

AssertionError: Predicate "" not true after 60.0 seconds

Same run also had separate in-scope addressindex RPC type-check failures in other jobs; those were fixed on the PR branch independently. This multiprocess failure still looks like the existing intermittent LLMQ signing flake.

[thepastaclaw]

Another occurrence from PR #7232, again unrelated to the PR diff itself (the PR only changes CI/workflow/env files plus test/get_previous_releases.py).

• CI run: https://github.com/dashpay/dash/actions/runs/24081094106
• Job: linux64-test / Test source (70248433408)
• Date: 2026-04-07

This run hit the same feature_llmq_signing.py DKG phase timeout in wait_for_quorum_phase():

AssertionError: Predicate ... not true after 120.0 seconds

Same job also had separate collateral failures from a missing shared RPC coverage temp dir (dashpay/dash#7273), but the LLMQ signing timeout is still the existing intermittent failure tracked here.

[thepastaclaw]

Another occurrence from PR #7242 (wallet deprecation + RPC type checking backport), current head 24daadcf6d, and it is still unrelated to the PR diff itself.

• CI run: https://github.com/dashpay/dash/actions/runs/24580460650
• Job: https://github.com/dashpay/dash/actions/runs/24580460650/job/71879977938
• Build target: linux64_sqlite-test / Test source
• Date: 2026-04-17

Failure was again feature_llmq_signing.py --spork21, timing out in wait_for_sigs(True, False, True, 15) after retrying 3/3 times:

AssertionError: Predicate "" not true after 60.0 seconds

I rechecked the current PR head and git log upstream/develop..HEAD -- test/functional/feature_llmq_signing.py is empty, so this test file is still untouched by the branch. The only functional-test changes on the PR are unrelated RPC bool/type-check adaptations plus deprecatedrpc=create_bdb test config plumbing. This still looks like the existing intermittent LLMQ signing flake tracked here, not a regression from #7242.

[thepastaclaw]

Another occurrence from dashpay/dash#7232 after the multiprocess lane had already been moved back to amd64:

• PR head: ff5ff3536502914506ce5c912783a1e070f8a7f1
• Failing job: linux64_multiprocess-test / Test source
• Run/job: https://github.com/dashpay/dash/actions/runs/24600056092/job/71938370655
• Failing variant: feature_llmq_signing.py --spork21
• Failure mode: same wait_until() timeout in wait_for_sigs(True, False, True, 15) at feature_llmq_signing.py:111

This is useful because the latest failure is no longer tied to the PR's ARM-runner migration path for multiprocess: on this head, linux64_multiprocess depends/build/test all run on amd64 again, yet the same intermittent timeout still reproduces in CI.

I also re-ran the exact failing variant locally on upstream/develop:

python3 test/functional/feature_llmq_signing.py --spork21

That local develop run passed, which fits the existing intermittent/timing-sensitive diagnosis rather than a deterministic regression from dash#7232.

[thepastaclaw]

Another confirmation from tracker #1015 / #7232 current head ff5ff3536502914506ce5c912783a1e070f8a7f1:

• failing job: https://github.com/dashpay/dash/actions/runs/24600056092/job/71938370655
• failing variant: feature_llmq_signing.py --spork21
• failure mode: same wait_until() timeout in wait_for_sigs(True, False, True, 15) at feature_llmq_signing.py:111

I rechecked the current branch diff and it still does not touch test/functional/feature_llmq_signing.py (git log upstream/develop..HEAD -- test/functional/feature_llmq_signing.py is empty). I also confirmed the latest linux64_multiprocess-test lane on this PR is already running on amd64 again, so this occurrence is not specific to the ARM-runner migration path anymore.

I reran the exact failing variant locally on upstream/develop again:

python3 test/functional/feature_llmq_signing.py --spork21

That local develop run passed, which is consistent with the existing intermittent/timing-sensitive flake diagnosis rather than a deterministic regression from #7232.

[thepastaclaw]

Another datapoint from tracker #1023 / PR #7232 current head ff5ff3536502914506ce5c912783a1e070f8a7f1:

• failing job: https://github.com/dashpay/dash/actions/runs/24600056092/job/71938370655
• failing variant: feature_llmq_signing.py --spork21
• branch diff still does not touch test/functional/feature_llmq_signing.py or test/functional/test_framework/ (git log upstream/develop..HEAD -- <path> is empty for both)

I also reran the exact failing variant locally on upstream/develop just now:

python3 test/functional/feature_llmq_signing.py --spork21

This time it failed locally on develop with the same timeout in wait_for_sigs(True, False, True, 15) at feature_llmq_signing.py:111:

AssertionError: Predicate """"
            self.wait_until(lambda: check_sigs(hasrecsigs, isconflicting1, isconflicting2), timeout = timeout)
""" not true after 15 seconds

So the latest #7232 occurrence is not only unrelated to the PR diff, it is now directly reproducible on develop as the same underlying test bug.

[thepastaclaw]

Another recheck from tracker #1029 / PR #7232 current head ff5ff3536502914506ce5c912783a1e070f8a7f1:

• failing job: https://github.com/dashpay/dash/actions/runs/24600056092/job/71938370655
• failing variant: feature_llmq_signing.py --spork21
• branch diff still only touches CI/workflow/env files plus test/get_previous_releases.py; it does not touch test/functional/feature_llmq_signing.py or test/functional/test_framework/
• the current linux64_multiprocess-test lane on this PR head is already back on amd64, so this is still not specific to the ARM-runner migration path

I reran the exact failing variant locally on upstream/develop again today:

python3 test/functional/feature_llmq_signing.py --spork21

That rerun passed on develop, which keeps matching the intermittent/timing-sensitive flake diagnosis rather than a deterministic regression from dash#7232. Existing comments on this issue already show mixed local pass/fail outcomes for the same test, which strengthens the flake diagnosis.

[knst]

@thepastaclaw feature_llmq_signing.py --spork21 has intermittent failure; it may fail or not depends to some multi-thread race or miscommunications between nodes / communications with wrong unexpected timing.

Single run of test python3 test/functional/feature_llmq_signing.py --spork21 can not be used to determine a successful fix or still having issue and separate a fixed version from the broken.

Do at least 10 runs and validate a result, count amount of failures with --spork21:
test/functional/test_runner.py -j20 feature_llmq_signing.py feature_llmq_signing.py feature_llmq_signing.py feature_llmq_signing.py feature_llmq_signing.py feature_llmq_signing.py feature_llmq_signing.py feature_llmq_signing.py feature_llmq_signing.py feature_llmq_signing.py

consider reducing -j20 to smaller amount if you have not enough RAM or less than 4 cpu cores and increasing amount of runs from 10 to 20, 50 if results are not clear.

Drop failure rate to the half of current failure rate is a considered as a proper fix worth to get merged.

[thepastaclaw]

@knst agreed — single-run pass/fail isn't a useful validation signal here.

I ran a 10x batch on current upstream/develop (cfad4141d60961e5d4f27e11a8d30deccf18250a) with:

python3 test/functional/test_runner.py -j10 --timeout-factor=1 \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21" \
  "feature_llmq_signing.py --spork21"

Result: 6/10 failures, 4/10 passes.

Failure breakdown from this batch:

• 5/6 failures timed out in the same place already tracked here:
  • feature_llmq_signing.py:111
  • wait_for_sigs(True, False, True, 15)
• 1/6 failures made it further, then failed at:
  • feature_llmq_signing.py:162
  • assert_sigs_nochange(True, False, True, 3)

So the current baseline on develop is roughly a 60% failure rate under repeated runs, and it is not limited to a single lucky/unlucky invocation. I'll use repeated-run batches like this for validation going forward, and any candidate fix should reduce that failure rate materially (per your "drop it by at least half" bar).

[thepastaclaw]

Another occurrence from thepastaclaw/dash#21 (backport: bitcoin#27594 — refactor: Remove unused GetTimeMillis), and this one is again unrelated to the PR diff itself.

• PR head: d0e55ea865ef510106abb597feae88dd4f68507d
• failing jobs:
  • https://github.com/thepastaclaw/dash/actions/runs/24655377399/job/72093830252 (linux64-test / Test source)
  • https://github.com/thepastaclaw/dash/actions/runs/24655377399/job/72093138446 (linux64_sqlite-test / Test source)
  • https://github.com/thepastaclaw/dash/actions/runs/24655375875/job/72098129353 (linux64_multiprocess-test / Test source)
• failing variant in all three jobs: feature_llmq_signing.py --spork21
• failure mode:
  • feature_llmq_signing.py:111 / wait_for_sigs(True, False, True, 15) in linux64-test and linux64_multiprocess-test
  • feature_llmq_signing.py:201 / wait_for_sigs(True, False, True, 2) in linux64_sqlite-test

Scope check on the PR branch:

• exact PR diff only touches src/rpc/net.cpp and src/util/time.cpp
• git log upstream/develop..HEAD -- test/functional/feature_llmq_signing.py test/functional/test_framework/ is empty on the PR head

Also useful: the same head already has successful runs for those same job names in the other matrix run attached to the PR, so this occurrence again shows pass/fail behavior on an unchanged commit.

I reran the exact failing variant locally on current develop:

python3 test/functional/feature_llmq_signing.py --spork21

That single run passed for me today, which is still consistent with the existing intermittent diagnosis rather than a deterministic regression from this PR.

[thepastaclaw]

Another occurrence from tracker #1035 / dashpay/dash#7232 current head ff5ff3536502914506ce5c912783a1e070f8a7f1:

• failing job: https://github.com/dashpay/dash/actions/runs/24600056092/job/71938370655
• failing variant: feature_llmq_signing.py --spork21
• failure mode: same wait_until() timeout in wait_for_sigs(True, False, True, 15) at feature_llmq_signing.py:111

I rechecked the PR diff on this head and it still only touches CI/workflow/env files plus test/get_previous_releases.py; it does not touch test/functional/feature_llmq_signing.py or test/functional/test_framework/ (git log upstream/develop..HEAD -- <path> is empty for both).

I also reran the exact failing variant locally on current upstream/develop just now:

python3 test/functional/feature_llmq_signing.py --spork21

That run passed for me this time. Together with the existing mixed pass/fail data already on this issue, this still points to the same intermittent/timing-sensitive flake rather than a deterministic regression from dash#7232.

[thepastaclaw]

Another occurrence on #7242 (head 24daadcf6d, run https://github.com/dashpay/dash/actions/runs/24580460650):

• failing job: https://github.com/dashpay/dash/actions/runs/24580460650/job/71879977938
• failure: feature_llmq_signing.py --spork21 timed out again in wait_for_sigs(True, False, True, 15) after failing all 3 CI retry attempts

Scope re-check on this PR still points away from branch-caused regression:

• git diff upstream/develop..HEAD --name-only does not touch test/functional/feature_llmq_signing.py
• the non---spork21 variant passed in the same CI job
• I also ran python3 test/functional/feature_llmq_signing.py --spork21 on current develop locally and it passed

So this remains consistent with the intermittent timeout/flakiness already tracked here, not something introduced by PR #7242.

[thepastaclaw]

Another occurrence from tracker #1067 / dashpay/dash#7232 current head ff5ff3536502914506ce5c912783a1e070f8a7f1:

• failing job: https://github.com/dashpay/dash/actions/runs/24600056092/job/71938370655
• failing variant: feature_llmq_signing.py --spork21
• failure mode: same wait_until() timeout in wait_for_sigs(True, False, True, 15) at feature_llmq_signing.py:111

I rechecked the exact PR diff on this head with git diff upstream/develop..HEAD --name-only; it still only touches CI/workflow/env files plus test/get_previous_releases.py, and does not touch test/functional/feature_llmq_signing.py or test/functional/test_framework/.

I also reran the exact failing variant locally on current upstream/develop just now:

python3 test/functional/feature_llmq_signing.py --spork21

This time it failed locally on develop with the same timeout in wait_for_sigs(True, False, True, 15) at feature_llmq_signing.py:111:

AssertionError: Predicate ''''
            self.wait_until(lambda: check_sigs(hasrecsigs, isconflicting1, isconflicting2), timeout = timeout)
''' not true after 15 seconds

That makes this latest #7232 failure directly reproducible on develop again, so it is still the pre-existing intermittent LLMQ signing test bug tracked here, not a regression from the PR branch.

[thepastaclaw]

Another occurrence from tracker #1074 / dashpay/dash#7232 current head ff5ff3536502914506ce5c912783a1e070f8a7f1:

• failing job: https://github.com/dashpay/dash/actions/runs/24600056092/job/71938370655
• failing variant: feature_llmq_signing.py --spork21
• failure mode in CI: same wait_until() timeout in wait_for_sigs(True, False, True, 15) at feature_llmq_signing.py:111

I rechecked the exact PR diff on this head with git diff upstream/develop..HEAD --name-only; it still only touches CI/workflow/env files plus test/get_previous_releases.py, and does not touch test/functional/feature_llmq_signing.py or test/functional/test_framework/ (git log upstream/develop..HEAD -- test/functional/feature_llmq_signing.py test/functional/test_framework/ is empty).

I also reran the exact failing variant locally on current upstream/develop just now:

./test/functional/feature_llmq_signing.py --spork21

This time it failed locally on develop too, with the same underlying signing-timeout predicate (different assertion site later in the script, but same check_sigs(...) wait):

File "/Users/claw/Projects/dash/./test/functional/feature_llmq_signing.py", line 201, in run_test
    wait_for_sigs(True, False, True, 2)
...
AssertionError: Predicate '\\\\n            self.wait_until(lambda: check_sigs(hasrecsigs, isconflicting1, isconflicting2), timeout = timeout)\n'\\ not true after 2 seconds\n```\n\nSo the latest `#7232` failure is again reproducible on plain develop and remains the pre-existing intermittent `feature_llmq_signing.py --spork21` bug tracked here, not a regression from the PR branch.