embed: enforce HTTP body size limit on /v3/ gRPC-gateway endpoints#22004
embed: enforce HTTP body size limit on /v3/ gRPC-gateway endpoints#22004crawfordxx wants to merge 1 commit into
Conversation
Without a body-size guard, an unauthenticated HTTP/JSON request to any /v3/* gRPC-gateway endpoint can force etcd to read and decode an arbitrarily large request body before authentication or MaxRequestBytes enforcement, causing GiB-scale RSS growth from a single 256 MiB request. Fix: in accessController.ServeHTTP, wrap req.Body with http.MaxBytesReader (limit = MaxRequestBytes*4) for all /v3/ paths before delegating to the mux. The 4x multiplier accounts for base64 encoding overhead (~4/3) and JSON field-name framing, ensuring that a legitimate maximum-size payload encoded as JSON is never rejected. Requests that declare an oversized Content-Length are rejected with 413 before any body bytes are read. WebSocket upgrade requests have an empty body, so the MaxBytesReader wrapper is a no-op for them. Fixes etcd-io#21555 Signed-off-by: crawfordxx <crawfordxx@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: crawfordxx The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @crawfordxx. Thanks for your PR. I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Problem
Without a request-body size guard on the gRPC-gateway HTTP path, an
unauthenticated attacker can force etcd to allocate GiB-scale memory from a
single 256 MiB HTTP/JSON POST to any
/v3/*endpoint (e.g./v3/kv/range).The allocation happens inside JSON decoding, before authentication runs
or
--max-request-bytesis enforced at the gRPC layer. Observed impact:a 256 MiB JSON request body causes ~1.75 GiB RSS growth, ~7× amplification
(base64 decode + JSON parse + protobuf allocation).
Fix
In
accessController.ServeHTTP, wrapreq.Bodywithhttp.MaxBytesReaderfor all
/v3/requests before delegating to the mux:The 4× factor accounts for:
This ensures a legitimate maximum-size payload (up to
MaxRequestBytes)encoded as a JSON/base64 body is never rejected, while an oversized
body is either:
Content-Lengthdeclares it oversized, orMaxBytesReadermid-read ifContent-Lengthis absent.WebSocket upgrade requests carry no body, so the wrapper is a no-op for them.
Testing
Manual testing with a
256 MiBbase64 key in a/v3/kv/rangerequest:Fixes #21555