test: Retry CliRunner apply subprocess on timeout to reduce flakiness#6592
Open
nikolauspschuetz wants to merge 1 commit into
Open
test: Retry CliRunner apply subprocess on timeout to reduce flakiness#6592nikolauspschuetz wants to merge 1 commit into
nikolauspschuetz wants to merge 1 commit into
Conversation
CliRunner.run() (used for 'feast apply' in unit tests) spawns a fresh child
interpreter that must import Feast (pandas/pyarrow/...) from cold. On loaded
CI runners — observed on unit-test-python (3.11, macos-14) — this can exceed
the 120s timeout, failing the test with:
AssertionError: Command timed out after 120s: ['apply']
Unlike the materializing paths guarded by run_with_output(), commands routed
through run() do not materialize and so cannot trigger the Dask atexit hang the
timeout was added for. A timeout here is therefore almost always transient cold
-start slowness, not a genuine deadlock, so retrying once (with warm OS/page
caches) clears it while a real repeated stall still fails after the attempts
are exhausted.
Builds on the CliRunner subprocess stabilization in feast-dev#6560.
Signed-off-by: Nikolaus Schuetz <nikolauspschuetz@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
CliRunner.run()(the helper that runsfeast applyin a fresh child interpreter for unit tests) now retries once onsubprocess.TimeoutExpiredinstead of treating the first timeout as terminal. The per-attempt timeout is unchanged (120s); the timeout message now includes the attempt counter.Why
The unit-test workflow flaked on
unit-test-python (3.11, macos-14)while setting uptest_feature_server.py::test_push_and_get:run()spawns a new Python interpreter (sys.executable feast.cli apply) — deliberately, for feature-repo import isolation — which has to import Feast (pandas/pyarrow/protobuf/...) from cold. On a loaded macOS runner that cold start can occasionally exceed the 120s guard. The same commit passed onmacos-14 / 3.12and on all three Ubuntu legs, which is the signature of an environmental cold-start stall rather than a code fault.Crucially, commands routed through
run()(e.g.apply) do not materialize, so they cannot trigger the Dask atexit hang thatrun_with_output()and the module docstring describe. A timeout here is therefore almost always transient slowness, not a deadlock — so a single retry with warm OS/page caches clears it, while a genuine repeated stall still fails once the attempts are exhausted. Retries are intentionally not added torun_with_output(), whose commands can legitimately hang and whose existing machinery recovers partial output rather than retrying.This builds directly on the CliRunner subprocess stabilization in #6560.
Validation
uv run ruff check sdk/python/tests/utils/cli_repo_creator.py— passeduv run ruff format --check sdk/python/tests/utils/cli_repo_creator.py— already formattedsubprocess.run:returncode == 0)returncode == -1, message... (attempt 2/2): [...], no exception raisedNote: the full unit suite could not be collected on my machine (local venv lacks the
google.cloudtest extra pulled byconftest.py); the change is confined to the retry loop, which is covered by the targeted verification above.Separate from #6591 (which surfaced this flake).