feat: Support task backoff

[khvn26]

This PR adds support for task backoff via raising a TaskBackoffError.

The task runner now reschedules a task if the above exception class is raised inside a regular (non-recurring) task. The exception class supports a delay_until argument to specify which datetime the next task invocation should be scheduled for. If not specified, settings.TASK_BACKOFF_DEFAULT_DELAY_SECONDS will be used.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 98.70130% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 94.42%. Comparing base (b6a250b) to head (af18bca).
⚠️ Report is 79 commits behind head on main.

Files with missing lines	Patch %	Lines
...sk_processor/test_unit_task_processor_processor.py	97.77%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #81      +/-   ##
==========================================
+ Coverage   94.17%   94.42%   +0.25%     
==========================================
  Files          74       74              
  Lines        2147     2208      +61     
==========================================
+ Hits         2022     2085      +63     
+ Misses        125      123       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[matthewelwell]

My general concern with this is that we could end up with a task infinitely failing and being rescheduled, right? Also, I don't see any logic here that 'nullifies' the original retry logic that will still retry the task as soon as possible, to a maximum of 3 times. Shouldn't we also intercept that logic somehow?

[matthewelwell]

Why not use registered_task.task_type here?

[khvn26]

The assertion also reassures Mypy that the task_handler attribute is not None.

[khvn26]

Clarified/cleaned up in b289839.

[khvn26]

@matthewelwell

we could end up with a task infinitely failing and being rescheduled, right?

This is a valid concern. We could do one of the following to address it:

1. Add a num_failures argument to the .delay method to carry over the number of failures when scheduling the task.
2. Do not use delay at all to re-create the task and modify the scheduled_for field in-place for the existing one.

Note that both of the options will cap the backoff attempts at the currently hard-coded value of 3 attempts.

Shouldn't we also intercept that [retry] logic somehow?

Good catch, done in b905d54 — but that will need to be reverted should we choose the option (2) above.

[khvn26]

Implemented option 2 in 9de8df5, which also resulted in a leaner overall diff 👍