Skip to content

test(integration): harden post-create exec test vs echo-proxy bring-up race (bd openlock-eh8)#55

Merged
vessux merged 1 commit into
mainfrom
fix/eh8-post-create-exec-retry
May 30, 2026
Merged

test(integration): harden post-create exec test vs echo-proxy bring-up race (bd openlock-eh8)#55
vessux merged 1 commit into
mainfrom
fix/eh8-post-create-exec-retry

Conversation

@vessux
Copy link
Copy Markdown
Owner

@vessux vessux commented May 30, 2026

What

Fixes the recurring exit-56 live-integration flake (bd openlock-eh8) that went red on three consecutive main pushes (#52, #53, #54), each time passing on retry.

Root cause (prior diagnosis was wrong)

The earlier theory was an "ssh transport delivered no data / supervisor stdout-over-ssh delivery race." That's incorrect:

  • ssh relays the remote command's exit status. ssh's own transport failures exit 255, never 56. So ssh exited with status 56 ⇒ the remote curl exited 56 = CURLE_RECV_ERROR (and the historical 35 = CURLE_SSL_CONNECT_ERROR).
  • The "empty stdout AND stderr" was just curl -s muting curl's error message — not a dropped transport. The supervisor/ssh path worked fine; it carried curl's exit code back faithfully.

The real cause is curl racing the in-container echo-proxy + CA-bundle bring-up on the first egress. This was already diagnosed and mitigated in the two sibling echo-mode tests (harness-cred-inject, openrouter-opencode-cred-inject) via --retry 5 --retry-all-errors --retry-delay 1 (added in #38) — their comment literally says "transient TLS/network failures (exit 35/56) … races the supervisor's CA-bundle + echo-proxy bring-up."

post-create-exec-proxy.test.ts (added in #34 for openlock-hnp, the post-create sandbox exec path) never received that hardening — it was the one test still on bare curl -sf, which is exactly why it became the new flake locus. It even waits for waitForSandboxReady first, confirming that readiness (/bin/true execs) ≠ egress/proxy ready.

Change

  • Add --retry 5 --retry-all-errors --retry-delay 1 to post-create-exec-proxy.test.ts, matching its siblings. (Doesn't weaken the routing assertion — a real proxy-bypass regression still fails after retries.)
  • Switch all three flaking tests -sf-sSf so a retries-exhausted failure surfaces curl's real error instead of muting it.

Verification

  • bun run lint ✅ (exit 0) · bun run typecheck ✅ · bun test unit delta zero (the three edited tests are skipIf(!LIVE)).
  • The fix is exercised by live-integration(podman) on this PR — the job where the flake actually reproduces.

Follow-up

Filed openlock-g4x (P3): the test --retry masks the bring-up race, but the production attach path may expose a real harness to the same first-egress transient. Worth an egress-aware readiness probe — but pin whether prod is actually exposed before adding product-side retries.

…p race (bd openlock-eh8)

post-create-exec-proxy.test.ts was the one echo-mode integration test still on
bare `curl -sf`, missing the `--retry 5 --retry-all-errors` hardening its
siblings (harness-cred-inject, openrouter-opencode-cred-inject) received in #38.
That gap is why it became the recurring exit-56 flake locus on #52/#53/#54.

exit 56 = curl CURLE_RECV_ERROR from the in-container echo proxy on first
egress, relayed faithfully through ssh (ssh's own transport failures are 255,
never 56) — NOT an ssh transport drop as previously suspected. `curl -s` was
muting curl's error, which presented as "empty stdout/stderr".

- add `--retry 5 --retry-all-errors --retry-delay 1` to the post-create exec test
- switch all three flaking tests `-sf` -> `-sSf` so a retries-exhausted failure
  surfaces curl's real error instead of being silently muted

bd openlock-eh8
@vessux vessux merged commit c9e3e1c into main May 30, 2026
5 checks passed
@vessux vessux deleted the fix/eh8-post-create-exec-retry branch May 30, 2026 19:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant