Bug: Redis Pool Starvation and Deadlock After `deadpool` 0.12.3 Upgrade

[ahmadbky]

Summary

After upgrading the deadpool dependency from 0.12.2 to 0.12.3, the API is experiencing increased request timeouts, especially for endpoints that require Redis DB access.

Root Cause

• The default Redis pool buffer size in deadpool changed from <num_cpus> * 4 to <num_cpus> * 2 in version 0.12.3.
• In production, this dropped the available pool connections from 8 to 4, starting with API release 1.4.0.
• Normally, if 4 concurrent requests occupy all 4 connections, any additional request will queue and wait for a connection to be released (not block indefinitely).
• However, some handlers acquire two Redis connections sequentially within a single process (example below). If 4 requests to such a handler occur simultaneously:
  • Each request takes its first Redis connection (the pool is now fully occupied).
  • When each handler tries to acquire its second Redis connection, they all end up waiting for a connection to be released, but no handler can proceed to release theirs (since they’re blocked, holding their first connection), causing a deadlock.
• As a result, all pool connections are stuck, and any API call requiring Redis will be blocked until a timeout.

Example

A problematic pattern appears in, for instance:
player_finished.rs#L273

Consequences

• 50% reduction in pool connections increased the risk of pool starvation and deadlocks for handlers needing multiple connections.
• These requests can indefinitely block each other, and by extension, block unrelated API calls needing Redis access.

Solutions

To address this:

1. Configure the Redis pool to timeout when retrieving a connection
   Prevent indefinite blocking by failing gracefully if a connection is not available after a short wait.
2. Increase and set a fixed Redis pool size (e.g., 8 or 16 connections)
   Avoid upstream default changes from silently causing operational failures.
3. Refactor request handlers to only ever acquire one Redis connection per request
   Remove or consolidate code that needs two connections in a single request/handler.