Skip to content

feat: auto-disable recurring task after 4 consecutive failures#221

Merged
gagantrivedi merged 5 commits into
mainfrom
feat/disable-recurring-task
May 1, 2026
Merged

feat: auto-disable recurring task after 4 consecutive failures#221
gagantrivedi merged 5 commits into
mainfrom
feat/disable-recurring-task

Conversation

@gagantrivedi

@gagantrivedi gagantrivedi commented Apr 29, 2026

Copy link
Copy Markdown
Member

Summary

  • Adds is_disabled and num_consecutive_failures to RecurringTask. mark_failure increments the counter and flips is_disabled once it hits 4; mark_success resets it.
  • Updates the pickup SQL function so disabled rows are skipped at the source, no worker wakeup needed.
  • Surfaces the state in the Django admin (column, filter, "Re-enable selected tasks" action).

Test plan

  • Covered by unit tests

@codecov-commenter

codecov-commenter commented Apr 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.38%. Comparing base (7746773) to head (3dff037).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #221      +/-   ##
==========================================
+ Coverage   97.32%   97.38%   +0.05%     
==========================================
  Files         104      105       +1     
  Lines        4525     4586      +61     
==========================================
+ Hits         4404     4466      +62     
+ Misses        121      120       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

gagantrivedi and others added 4 commits April 30, 2026 14:42
Add is_disabled and num_consecutive_failures to RecurringTask. mark_failure
increments the counter and flips is_disabled when it reaches the threshold;
mark_success resets it. Pickup SQL is updated to skip disabled rows so dead
tasks stop wasting worker cycles. Surfaced in the Django admin with a
"Re-enable selected tasks" action.
@gagantrivedi gagantrivedi force-pushed the feat/disable-recurring-task branch from fbc56bd to f358de0 Compare April 30, 2026 09:13
@gagantrivedi gagantrivedi changed the base branch from refactor/simplify-run-recurring-tasks to main April 30, 2026 09:13
@gagantrivedi gagantrivedi reopened this Apr 30, 2026
@gagantrivedi gagantrivedi marked this pull request as ready for review April 30, 2026 11:10
@gagantrivedi gagantrivedi requested a review from a team as a code owner April 30, 2026 11:10
@gagantrivedi gagantrivedi requested review from a team, emyller and khvn26 and removed request for a team and emyller April 30, 2026 11:10

@khvn26 khvn26 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The approach seems sound to me — I've only called out lack of logging.

Comment thread src/task_processor/models.py
Comment thread tests/unit/task_processor/test_unit_task_processor_processor.py Outdated
- Emit an error log when the consecutive-failure threshold flips a
  recurring task to disabled, so ops can see flapping tasks getting
  quarantined.
- Use a one-line `...` stub for the never-executed dummy task in the
  disabled-task test so coverage doesn't flag the unreachable body.
@gagantrivedi gagantrivedi requested a review from khvn26 May 1, 2026 03:43

@khvn26 khvn26 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gagantrivedi gagantrivedi merged commit 8fe9ea3 into main May 1, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants