Skip to content

test: Retry CliRunner apply subprocess on timeout to reduce flakiness#6592

Open
nikolauspschuetz wants to merge 1 commit into
feast-dev:masterfrom
nikolauspschuetz:test/stabilize-cli-runner-apply-retry
Open

test: Retry CliRunner apply subprocess on timeout to reduce flakiness#6592
nikolauspschuetz wants to merge 1 commit into
feast-dev:masterfrom
nikolauspschuetz:test/stabilize-cli-runner-apply-retry

Conversation

@nikolauspschuetz

Copy link
Copy Markdown
Contributor

What changed

CliRunner.run() (the helper that runs feast apply in a fresh child interpreter for unit tests) now retries once on subprocess.TimeoutExpired instead of treating the first timeout as terminal. The per-attempt timeout is unchanged (120s); the timeout message now includes the attempt counter.

Why

The unit-test workflow flaked on unit-test-python (3.11, macos-14) while setting up test_feature_server.py::test_push_and_get:

ERROR sdk/python/tests/unit/test_feature_server.py::test_push_and_get
  AssertionError: stdout: b''
  stderr: b"Command timed out after 120s: [\x27apply\x27]"
2259 passed, 47 skipped, 1 error

run() spawns a new Python interpreter (sys.executable feast.cli apply) — deliberately, for feature-repo import isolation — which has to import Feast (pandas/pyarrow/protobuf/...) from cold. On a loaded macOS runner that cold start can occasionally exceed the 120s guard. The same commit passed on macos-14 / 3.12 and on all three Ubuntu legs, which is the signature of an environmental cold-start stall rather than a code fault.

Crucially, commands routed through run() (e.g. apply) do not materialize, so they cannot trigger the Dask atexit hang that run_with_output() and the module docstring describe. A timeout here is therefore almost always transient slowness, not a deadlock — so a single retry with warm OS/page caches clears it, while a genuine repeated stall still fails once the attempts are exhausted. Retries are intentionally not added to run_with_output(), whose commands can legitimately hang and whose existing machinery recovers partial output rather than retrying.

This builds directly on the CliRunner subprocess stabilization in #6560.

Validation

  • uv run ruff check sdk/python/tests/utils/cli_repo_creator.py — passed
  • uv run ruff format --check sdk/python/tests/utils/cli_repo_creator.py — already formatted
  • Retry logic verified by monkeypatching subprocess.run:
    • transient timeout on attempt 1 → recovered on attempt 2 (returncode == 0)
    • always timing out → returncode == -1, message ... (attempt 2/2): [...], no exception raised
    • success on first attempt → exactly one subprocess call (no behavior change for the passing path)

Note: the full unit suite could not be collected on my machine (local venv lacks the google.cloud test extra pulled by conftest.py); the change is confined to the retry loop, which is covered by the targeted verification above.

Separate from #6591 (which surfaced this flake).

CliRunner.run() (used for 'feast apply' in unit tests) spawns a fresh child
interpreter that must import Feast (pandas/pyarrow/...) from cold. On loaded
CI runners — observed on unit-test-python (3.11, macos-14) — this can exceed
the 120s timeout, failing the test with:

    AssertionError: Command timed out after 120s: ['apply']

Unlike the materializing paths guarded by run_with_output(), commands routed
through run() do not materialize and so cannot trigger the Dask atexit hang the
timeout was added for. A timeout here is therefore almost always transient cold
-start slowness, not a genuine deadlock, so retrying once (with warm OS/page
caches) clears it while a real repeated stall still fails after the attempts
are exhausted.

Builds on the CliRunner subprocess stabilization in feast-dev#6560.

Signed-off-by: Nikolaus Schuetz <nikolauspschuetz@gmail.com>
@nikolauspschuetz nikolauspschuetz requested a review from a team as a code owner July 9, 2026 07:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant