feat(reliability): single-leader lease + startup jitter for reply-SLA sweeps#930
Open
seonghobae wants to merge 1 commit into
Open
feat(reliability): single-leader lease + startup jitter for reply-SLA sweeps#930seonghobae wants to merge 1 commit into
seonghobae wants to merge 1 commit into
Conversation
… sweeps With multiple backend replicas, every instance ran the reply-SLA sweep loop concurrently: duplicate escalation scans per tenant on every interval, and a thundering herd at deploy time. - _sync now takes a non-blocking pg_try_advisory_lock lease (same advisory- lock pattern as email import quota); non-holders skip the cycle instead of duplicating the sweep. Non-PostgreSQL sessions (dev/test, single process) bypass coordination entirely, so existing behavior there is unchanged. - The lease is always released in finally — session-level advisory locks outlive pooled connections, so an unreleased lock would leak through the pool and block every future sweep. - _run_loop adds bounded startup jitter (interval/10, capped 60s) so replicas started by one deploy do not contend at the same instant. Tests: lease-held-elsewhere skips sweep without unlock; acquired lease sweeps then unlocks; unlock happens even when the sweep raises; existing scheduler tests unchanged (6 passed) + reply-tracking/main regressions (23 passed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RkKdtHRLG4wSLh6PVsp8J
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+112
to
+113
| random.uniform( | ||
| 0, min(self.interval_seconds / 10, MAX_STARTUP_JITTER_SECONDS) |
Contributor
|
PR governance metadata gate is not ready for
|
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| self.executed = True | ||
|
|
||
| class _Result: | ||
| def scalars(self_inner): |
| class _Result: | ||
| def scalars(self_inner): | ||
| class _Scalars: | ||
| def all(self_all): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gap
With multiple backend replicas,
ReplySlaSchedulerran on every instance: duplicate escalation scans per tenant each interval + a thundering herd at deploy (reliability audit #9).What
pg_try_advisory_lockin_sync(reuses the repo's existing advisory-lock pattern from email import quota). Non-holders skip the cycle. Non-PostgreSQL sessions (dev/test) bypass coordination — zero change there.finally— session-level advisory locks outlive pooled connections; an unreleased lock would leak through the pool and block all future sweeps permanently. Covered by a dedicated test.interval/10, cap 60s) de-synchronizes replicas at deploy.Verification (local — CI frozen org-wide)
test_reply_tracking_service+test_main→ 23 passed.🤖 Generated with Claude Code