API denial of service in master branch due to CpuBoundWork held while writing responses to clients

[julianbrost]

The following describes a problem that only affects the master branch so far as it was introduced in #10516 which made it to no released version so far.

CpuBoundWork was previously held while processing full HTTP requests and released before writing the response to the client. With the changes to support generating responses on the fly while writing them to the client, this now means that HTTP handlers themselves may trigger writes to the client. The CpuBoundWork in question is acquired here:

icinga2/lib/remote/httpserverconnection.cpp

Lines 427 to 430 in 3335340

	CpuBoundWork handlingRequest (yc);
	cpuBoundWorkTime = std::chrono::steady_clock::now() - start;
	HttpHandler::ProcessRequest(waitGroup, request, response, yc);

This can easily be triggered by the following shell command (adapt 24 to ncpu*3/2). It creates that many curl instances that request /v1/objects/services and reads it very slowly (10B/s):

for i in {1..24}; do
  curl -Ssku root:icinga https://localhost:5665/v1/objects/services -X GET --json '{"pretty":true}' --limit-rate 10 &
done

There are already two PRs that should address this issue (it could also be fixed in isolation without touching CpuBoundWork), however there was no issue so far for tracking this problem properly:

• CpuBoundWork#CpuBoundWork(): don't spin on atomic int to acquire slot #9990
• Replace CpuBoundWork #10572