Reboot crossing watchdog timeout#2939
Open
pasinskim wants to merge 3 commits into
Open
Conversation
…ent watchdog A deployment that crosses a reboot only reaches its terminal status after the client's automatic system-reboot watchdog: mender-update waits a hardcoded chrono::minutes(10) (update_module_call.cpp, AsyncSystemReboot) for the `reboot` to take effect before giving up and reporting a terminal status. When that watchdog is hit, the deployment reaches 'failure' at ~601s. check_expected_statistics defaulted to a 600s poll window -- an exact dead heat with that 600s watchdog -- so reboot-crossing failure cases flaked with "Never found <status>". Bumping to an independent literal (e.g. 900s) only relocates the race (901 vs 900). Introduce a single source of truth, CLIENT_AUTOMATIC_REBOOT_WATCHDOG, and a DERIVED deadline REBOOT_CROSSING_DEPLOYMENT_TIMEOUT (= watchdog * 2). Deriving it (multiplicatively) means a change to either value can never create a dead heat. Polling returns the instant the status is observed, so the larger deadline never slows a passing test; it only bounds a genuinely-stuck deployment. Also make the timeout self-diagnosing: dump the per-device deployment log and the observed status progression, and warn explicitly when max_wait <= the watchdog. The failure now shows the reboot / "Version file error" / watchdog gap directly, without pulling artifacts. Changelog: None Signed-off-by: pasinskim <marcin.pasinski@northern.tech> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…crossing failures The reboot-crossing failure cases (Failure_in_ArtifactCommit_* and the Corrupted_script_version_* sets) drive the client through an automatic system reboot; the deployment only reaches 'failure' after the client's 10-minute reboot watchdog. The default 600s check_expected_statistics window was an exact dead heat with that watchdog, so these cases flaked intermittently with "Never found failure ... after 600 seconds". Use REBOOT_CROSSING_DEPLOYMENT_TIMEOUT (derived from the client watchdog) instead of the default. This supersedes the bare max_wait=15*60 approach: the value is a function of the watchdog, so it cannot drift back into a dead heat, and on timeout the failure now dumps the device deployment log (the reboot / "Version file error" / watchdog gap) directly. Changelog: None Signed-off-by: pasinskim <marcin.pasinski@northern.tech> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t waits verify_reboot_performed() and the post-reboot check_expected_statistics() calls defaulted to a 600s window -- an exact dead heat with the client's 600s automatic-reboot watchdog. When a reboot is slow to take effect the device recovers at ~601s, one second after the wait gives up, producing flaky "Device never rebooted" / "Never found <status>" failures. Use REBOOT_CROSSING_DEPLOYMENT_TIMEOUT (derived from the client watchdog) for the reboot-crossing waits. verify_reboot_not_performed() (the intentional short "confirm no reboot" checks) are left untouched. Changelog: None Signed-off-by: pasinskim <marcin.pasinski@northern.tech> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
@pasinskim, start a full integration test pipeline with:
my commands and optionsYou can prevent me from automatically starting CI pipelines:
You can trigger a client pipeline on multiple prs with:
You can trigger a client pipeline for a specific Mender Client release with:
You can trigger GitHub->GitLab branch sync with:
You can print PR statistics for a repository with:
You can cherry pick to a given branch or branches with:
|
Member
Author
|
@mender-test-bot start integration pipeline |
Contributor
|
Hello 😺 I created a pipeline for you here: Pipeline-2631163881 Build Configuration Matrix
|
danielskinstad
approved these changes
Jun 26, 2026
| deploy.check_expected_statistics(deployment_id, "failure", 1) | ||
| reboot.verify_reboot_performed(max_wait=REBOOT_CROSSING_DEPLOYMENT_TIMEOUT) # since the network is broken, two reboots will be performed, and the last one will be detected | ||
| deploy.check_expected_statistics( | ||
| deployment_id, "failure", 1, max_wait=REBOOT_CROSSING_DEPLOYMENT_TIMEOUT |
Contributor
There was a problem hiding this comment.
won't we as a result wait 2*REBOOT_CROSSING_DEPLOYMENT_TIMEOUT? Once in verify_reboot_performed and a second time in check_expected_statistics?
I think there's no reason to wait for statistics this long.
| logger.info("Network stabilized") | ||
| reboot.verify_reboot_performed() | ||
| deploy.check_expected_statistics(deployment_id, "success", 1) | ||
| reboot.verify_reboot_performed(max_wait=REBOOT_CROSSING_DEPLOYMENT_TIMEOUT) |
Contributor
There was a problem hiding this comment.
I'm not sure I understand the reasoning here. If we wait here for the deployment to finish (successfully), then why would we need to wait for the client watchdog to kill mender-update? If it happens, then there would be no successfully deployment AFAIK.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix the race condition with client reboots inside the integration tests.