Bug Report
What did you do?
With audit or rate-limit middleware enabled, send HTTP requests to a PD
follower so that the follower forwards them to the PD leader via
customReverseProxies (e.g. GET /pd/api/v1/cluster).
When the forwarding http.Client.Do needs to retry on a new connection —
typically due to a stale keep-alive connection or HTTP/2 GOAWAY — the retry
fails.
What did you expect to see?
The forwarded request is transparently retried on a new connection and
succeeds.
What did you see instead?
The forward fails with:
[ERROR] [apiutil.go:465] ["request failed"] [error="[PD:http:ErrSendRequest]send HTTP request failed: Get \"https://.../pd/api/v1/cluster\": net/http: cannot rewind body after connection loss"]
Root cause: when audit/rate-limit middleware is enabled,
requestutil.getBodyParam replaces r.Body with
io.NopCloser(bytes.NewBuffer(buf)) without setting r.GetBody. This turns
what was originally http.NoBody into a non-nil body reader with no rewind
function. On the forwarding path, net/http Transport probes the body
(didRead=true), and on connection loss cannot rewind because GetBody is
nil, so the retry path errors out with the message above.
What version of PD are you using (pd-server -V)?
master (reproduces on current master and on the release branches that carry
the same audit middleware code path).
Bug Report
What did you do?
With audit or rate-limit middleware enabled, send HTTP requests to a PD
follower so that the follower forwards them to the PD leader via
customReverseProxies(e.g.GET /pd/api/v1/cluster).When the forwarding
http.Client.Doneeds to retry on a new connection —typically due to a stale keep-alive connection or HTTP/2 GOAWAY — the retry
fails.
What did you expect to see?
The forwarded request is transparently retried on a new connection and
succeeds.
What did you see instead?
The forward fails with:
Root cause: when audit/rate-limit middleware is enabled,
requestutil.getBodyParamreplacesr.Bodywithio.NopCloser(bytes.NewBuffer(buf))without settingr.GetBody. This turnswhat was originally
http.NoBodyinto a non-nil body reader with no rewindfunction. On the forwarding path,
net/httpTransport probes the body(
didRead=true), and on connection loss cannot rewind becauseGetBodyisnil, so the retry path errors out with the message above.
What version of PD are you using (
pd-server -V)?master (reproduces on current master and on the release branches that carry
the same audit middleware code path).