feat: resilient background job manager with retry & monitoring#382
Open
robotica4us-collab wants to merge 1 commit intorohitdash08:mainfrom
Open
feat: resilient background job manager with retry & monitoring#382robotica4us-collab wants to merge 1 commit intorohitdash08:mainfrom
robotica4us-collab wants to merge 1 commit intorohitdash08:mainfrom
Conversation
Implements a production-ready background job system that wraps APScheduler with automatic retry, dead-letter tracking, and full observability. Features: - Configurable exponential backoff retry (per-job RetryPolicy) - Dead-letter queue for permanently failed jobs with admin reset - Per-job execution history with timing and error details - Prometheus metrics (executions, retries, dead-letters, duration) - Redis-backed state persistence for crash recovery - Admin API endpoints (/jobs/status, /jobs/dead-letters, /jobs/<id>/reset) - Unauthenticated health check (/jobs/health) for monitoring systems - Automatic Flask app context injection for job functions The reminder processing job is wired up as the first managed job, replacing the manual /reminders/run endpoint with automatic processing. Includes 17 tests covering: - RetryPolicy exponential backoff and delay capping - JobState history capping - Job registration and status reporting - Dead-letter detection and reset - Flask integration (health endpoint, admin auth) - Successful execution resets attempts - Failing jobs retry then dead-letter - Transient failures recover on retry - Redis persistence and crash recovery Closes rohitdash08#130
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements a production-ready background job system that wraps APScheduler with automatic retry, dead-letter tracking, and full observability.
Features:
The reminder processing job is wired up as the first managed job, replacing the manual /reminders/run endpoint with automatic processing.
Includes 17 tests covering:
Closes #130
Summary
Validation
cd app && npm run lintcd app && npm test -- --runInBand./scripts/test-backend.ps1Security and Ownership
main)Checklist