Skip to content

Add retry monitoring for reminder jobs#1030

Open
darshan-Jahagirdar wants to merge 1 commit into
rohitdash08:mainfrom
darshan-Jahagirdar:codex/reminder-job-retry-monitoring
Open

Add retry monitoring for reminder jobs#1030
darshan-Jahagirdar wants to merge 1 commit into
rohitdash08:mainfrom
darshan-Jahagirdar:codex/reminder-job-retry-monitoring

Conversation

@darshan-Jahagirdar
Copy link
Copy Markdown

@darshan-Jahagirdar darshan-Jahagirdar commented May 13, 2026

Summary

  • Add bounded retry handling for due reminder delivery jobs
  • Keep failed reminders unsent, track cumulative delivery attempts, and store last_error
  • Add Prometheus background job counters for attempts, retries, successes, and failures
  • Return processed, sent, failed, and retries from /reminders/run
  • Document retry configuration and monitoring behavior

Closes #130

Validation

  • py -3.11 -m venv .venv-backend-test
  • .\.venv-backend-test\Scripts\python.exe -m pip install -r packages\backend\requirements.txt
  • From packages/backend: ..\..\.venv-backend-test\Scripts\python.exe -m pytest tests\test_reminders.py -q -k run_due -> 2 passed, 2 deselected
  • From packages/backend: ..\..\.venv-backend-test\Scripts\python.exe -m black --check app\config.py app\__init__.py app\models.py app\observability.py app\services\jobs.py app\routes\reminders.py tests\test_reminders.py
  • From packages/backend: ..\..\.venv-backend-test\Scripts\python.exe -m flake8 app\config.py app\__init__.py app\models.py app\observability.py app\services\jobs.py app\routes\reminders.py tests\test_reminders.py
  • python -m py_compile packages\backend\app\config.py packages\backend\app\__init__.py packages\backend\app\models.py packages\backend\app\observability.py packages\backend\app\services\jobs.py packages\backend\app\routes\reminders.py packages\backend\tests\test_reminders.py
  • git diff --check

Note: running the entire existing tests/test_reminders.py module locally requires Redis for the pre-existing auth_header fixture; the new retry tests avoid Redis by minting JWTs directly and passed in the local environment.

/claim #130

@darshan-Jahagirdar
Copy link
Copy Markdown
Author

Validation update for this PR:

python -m compileall -q packages\backend\app
git diff --check

Result: both passed locally.

I also tried the full backend pytest suite in a temporary Python 3.11 virtualenv with packages/backend/requirements.txt installed. That run is blocked in this local environment because auth-dependent tests try to connect to Redis at redis:6379, which is not resolvable here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Resilient background job retry & monitoring

1 participant