Skip to content

rptest/util: add check_consistently() safety helper#31013

Open
nvartolomei wants to merge 2 commits into
redpanda-data:devfrom
nvartolomei:nv/rptest-consistently
Open

rptest/util: add check_consistently() safety helper#31013
nvartolomei wants to merge 2 commits into
redpanda-data:devfrom
nvartolomei:nv/rptest-consistently

Conversation

@nvartolomei

Copy link
Copy Markdown
Contributor

Adds check_consistently() to rptest.util, the safety counterpart to
wait_until's liveness check. Where wait_until blocks until a condition
becomes true, check_consistently asserts a condition stays true for the whole
duration window and fails fast with AssertionError the moment it breaks.

The docstring calls out the failure mode of safety assertions (too short a
window yields a meaningless pass) and recommends pairing them with a positive
control. Unit tests in util_test.py cover the hold, slow-poll, and
becomes-false paths.

Backports Required

  • none - not a bug fix

Release Notes

  • none

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new safety-style polling helper to the rptest test utilities, intended to assert a condition remains true over a time window (as a complement to wait_until’s eventual-true behavior).

Changes:

  • Add check_consistently() to tests/rptest/util.py for “condition must stay true” assertions over a duration.
  • Add ducktape unit-style tests for the helper in tests/rptest/util_test.py.
  • Minor formatting-only adjustments to a couple of existing lambdas for readability.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
tests/rptest/util.py Introduces check_consistently() and adjusts formatting in a couple of call sites.
tests/rptest/util_test.py Adds ducktape tests intended to validate check_consistently() behavior.

Comment thread tests/rptest/util.py
Comment thread tests/rptest/tests/util_test.py
Comment thread tests/rptest/tests/util_test.py
The inverse of wait_until: assert a condition holds for the whole
duration window instead of waiting for it to become true, failing fast
with AssertionError the moment it breaks.
@nvartolomei nvartolomei force-pushed the nv/rptest-consistently branch from bed5f0f to 7880fc6 Compare July 3, 2026 11:43
@vbotbuildovich

Copy link
Copy Markdown
Collaborator

Retry command for Build#86711

please wait until all jobs are finished before running the slash command

/ci-repeat 1
skip-redpanda-build
skip-units
skip-rebase
tests/rptest/tests/partition_force_reconfiguration_test.py::NodeWiseRecoveryTest.test_node_wise_recovery@{"dead_node_count":2}

@vbotbuildovich

vbotbuildovich commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

CI test results

test results on build#86711
test_status test_class test_method test_arguments test_kind job_url passed reason test_history
FLAKY(PASS) DataMigrationsApiTest test_migrated_topic_data_integrity {"params": {"cancellation": null, "include_groups": true, "transfer_leadership": true, "use_alias": true}} integration https://buildkite.com/redpanda/redpanda/builds/86711#019f27d2-9942-487a-a3e9-ea9e1fe57454 10/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0000, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=DataMigrationsApiTest&test_method=test_migrated_topic_data_integrity
FLAKY(FAIL) NodeWiseRecoveryTest test_node_wise_recovery {"dead_node_count": 2} integration https://buildkite.com/redpanda/redpanda/builds/86711#019f27d2-993f-400f-a587-085dce8293a5 7/11 Test FAILS after retries.Significant increase in flaky rate(baseline=0.0256, p0=0.0018, reject_threshold=0.0100) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=NodeWiseRecoveryTest&test_method=test_node_wise_recovery
test results on build#86757
test_status test_class test_method test_arguments test_kind job_url passed reason test_history
FLAKY(PASS) NodeWiseRecoveryTest test_node_wise_recovery {"dead_node_count": 2} integration https://buildkite.com/redpanda/redpanda/builds/86757#019f371c-a5e1-4e88-9342-b9fb95a16161 10/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0281, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=NodeWiseRecoveryTest&test_method=test_node_wise_recovery
FLAKY(PASS) NodeWiseRecoveryTest test_recovery_local_data_missing {"wait_for_final_manifest_uploads": true} integration https://buildkite.com/redpanda/redpanda/builds/86757#019f371c-a5e2-4dd4-92fe-210b9b47d12a 10/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0523, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1487, p1=0.1999, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=NodeWiseRecoveryTest&test_method=test_recovery_local_data_missing

@nvartolomei

Copy link
Copy Markdown
Contributor Author

/ci-repeat 1
skip-redpanda-build
skip-units
skip-rebase
tests/rptest/tests/partition_force_reconfiguration_test.py::NodeWiseRecoveryTest.test_node_wise_recovery@{"dead_node_count":2}

oleiman
oleiman previously approved these changes Jul 3, 2026

@oleiman oleiman left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

Comment thread tests/rptest/util.py Outdated
Raise ConditionNotHeldError, a subclass of AssertionError, so callers
can catch the specific failure while still treating it as an assertion.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants