Add retry logic to sacctmgr ping in start-services.sh

[azreenz]

Start-services.sh would exit with error code when pinging sacctmgr initially but then exits successfully after second pass of jetpack converge if database is in a different region and there is high latency. Added retry logic to prevent cluster from going red initially during converge

[Copilot]

Pull request overview

This PR updates start-services.sh to reduce false startup failures when slurmdbd is slow to become responsive (e.g., due to cross-region DB latency) by retrying sacctmgr ping before failing converge.

Changes:

• Add a retry loop around sacctmgr ping after starting slurmdbd via systemctl.
• Improve the failure message to indicate retries were attempted.

Criterion	Max Points	Points Awarded	Notes
PR Description Accuracy	20	20	Description matches the change (retrying initial sacctmgr ping).
PR Atomicity	20	20	Single focused change.
Logical Implementation	10	10	Retry approach is straightforward.
Regression Risk	10	0	Current $? usage after the loop is fragile/misleading and could regress diagnosability/behavior if the loop body changes.
Exception Handling	10	0	The final failure condition does not reliably reference the sacctmgr ping exit status.
Code Comments	10	10	Comment accurately describes intent.
Repetitive Code	10	10	No meaningful duplication introduced.
Spelling	5	5	No spelling issues found in the diff.
Logging Quality	5	5	Messages are clear enough for operators.

FINAL SCORE: 80/100

RECOMMENDATION: MERGE WITH FOLLOW-UPS
RATIONALE: The retry behavior aligns with the PR goal, but the exit-status handling should be tightened to ensure the code is checking and reporting the real sacctmgr ping result.
BLOCKERS:

• Fix the post-loop exit-status handling so it is based on the sacctmgr ping result (not the exit code of a subsequent [ ... ] test).

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.