Replace 100ms worker polling with long polling to avoid "polling storms" [gcp de feedback]

[jamescmartinez]

Workers poll the backend every 100ms. At scale, this creates a self-induced DDoS (e.g., 10k workers generate 100k reads per second even when idle).

Most small/medium deployments will have a single- or double-digit number of workers, so this only affects larger scale deployments.

[nathancahill]

Unfortunately the more I use this the more I'm thinking this issue is a fatal blocker for scaling beyond small/toy use without a secondary service for pushing events.

[jamescmartinez]

Can you share some more specifics about the problems you're having?

Are you running hundreds of workers that are hammering your DB? or something else?