Feat/requeue stale requests

[peshence]

Description

Sometimes a request stays active (processing or queued) on the request store, but is not actually on the queue (and is therefore likely not being worked on). This is likely due to issues during redeployment.

Now, when the queue is empty, the broker checks if there are any active requests in the request store and sets them back to "waiting" so they can be requeued.

Contributor Declaration

By opening this pull request, I affirm the following:

• All authors agree to the Contributor License Agreement.
• The code follows the project's coding standards.
• I have performed self-review and added comments where needed.
• I have added or updated tests to verify that my changes are effective and functional.
• I have run all existing tests and confirmed they pass.

[Copilot]

Pull request overview

This PR adds functionality to detect and requeue stale requests that appear stuck in QUEUED or PROCESSING status but are not actually on the queue (likely due to deployment issues). The broker now checks for such stuck requests when the queue is empty and resets them to WAITING status for reprocessing.

Key changes:

• Added get_active_requests() method to request store interface to fetch requests in QUEUED or PROCESSING status
• Implemented detection logic in the broker to identify and requeue stuck requests when the queue is empty
• Added warning when the number of active requests exceeds the configured queue size, suggesting potential stuck requests

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
polytope_server/common/request_store/request_store.py	Added abstract method get_active_requests() to the base request store interface
polytope_server/common/request_store/mongodb_request_store.py	Implemented get_active_requests() using MongoDB query with $in operator
polytope_server/common/request_store/dynamodb_request_store.py	Implemented get_active_requests() using DynamoDB scan with FilterExpression
polytope_server/broker/broker.py	Added stuck request detection and requeue logic when queue is empty; added warning for excessive active requests
tests/unit/test_request_store.py	Added test helper function _test_get_active_requests() to verify the method returns only QUEUED and PROCESSING requests
tests/unit/test_mongodb_request_store.py	Added test case calling _test_get_active_requests() for MongoDB store
tests/unit/test_dynamodb.py	Added test case calling _test_get_active_requests() for DynamoDB store

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[sametd]

LGTM