Skip to content

BUG-035: test_scenario_bc serial wait_link_ready times out early relays (BUG-032 fix incomplete) #14

@Kickflip73

Description

@Kickflip73

Summary

BUG-032 introduced wait_link_ready(retries=30, interval=0.5) to replace a hard-coded time.sleep(5). However, BUG-034 established that sandbox IP detection takes ~31s. When wait_link_ready is called serially for 3 relays, each only gets 15s max wait — so the first two always return None.

Observed Output

Orchestrator: None   ← waited 0-15s → timeout, IP not ready
Worker1:      None   ← waited 15-30s → timeout, IP not ready  
Worker2:      acp://33.229.113.196:7852/tok_...  ← waited 30-45s → success!

Result: Scenario B+C: 21/33 PASS (12 failures, all link-dependent)

Root Cause

run_bc_tests() calls:

orch_link = wait_link_ready(7950)  # 0-15s
w1_link   = wait_link_ready(7951)  # 15-30s  
w2_link   = wait_link_ready(7952)  # 30-45s ← gets link because ~31s has passed since all started

Total cumulative wait when reaching W2 = 30s, exceeding the 31s IP detection threshold.
But each individual call only waits 15s max, so Orch and W1 always miss.

Fix

Introduced wait_all_links_parallel() using ThreadPoolExecutor to wait for all relays concurrently. Total wait = max(individual waits) ≈ 31s instead of N×15s each.

def wait_all_links_parallel(port_map, retries=120, interval=0.5):
    import concurrent.futures
    with concurrent.futures.ThreadPoolExecutor(max_workers=len(port_map)) as ex:
        futures = {name: ex.submit(wait_link_ready, port, retries, interval)
                   for name, port in port_map.items()}
    return {name: fut.result() for name, fut in futures.items()}

Applied to both Scenario B (ports 7950/7951/7952) and Scenario C (7953/7954/7955).

Fix Commit

78ae426 — pushed to main

Affected Tests

  • B1.1, B3.1, B7.1 (Scenario B)
  • C1.1, C1.3, C2.1, C3.1, C5.1, C7.1, C7.2, C8.1, C8.2 (Scenario C)

Labels

test-infra, bug, P2

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions