Skip to content

embed: enforce HTTP body size limit on /v3/ gRPC-gateway endpoints#22004

Open
crawfordxx wants to merge 1 commit into
etcd-io:mainfrom
crawfordxx:fix-grpc-gateway-body-limit
Open

embed: enforce HTTP body size limit on /v3/ gRPC-gateway endpoints#22004
crawfordxx wants to merge 1 commit into
etcd-io:mainfrom
crawfordxx:fix-grpc-gateway-body-limit

Conversation

@crawfordxx

Copy link
Copy Markdown

Problem

Without a request-body size guard on the gRPC-gateway HTTP path, an
unauthenticated attacker can force etcd to allocate GiB-scale memory from a
single 256 MiB HTTP/JSON POST to any /v3/* endpoint (e.g. /v3/kv/range).
The allocation happens inside JSON decoding, before authentication runs
or --max-request-bytes is enforced at the gRPC layer. Observed impact:
a 256 MiB JSON request body causes ~1.75 GiB RSS growth, ~7× amplification
(base64 decode + JSON parse + protobuf allocation).

Fix

In accessController.ServeHTTP, wrap req.Body with http.MaxBytesReader
for all /v3/ requests before delegating to the mux:

maxBodyBytes := int64(ac.s.Cfg.MaxRequestBytes) * 4

The 4× factor accounts for:

  • Base64 encoding overhead (binary → text is ~4/3×)
  • JSON field-name framing (~20–30% additional overhead)

This ensures a legitimate maximum-size payload (up to MaxRequestBytes)
encoded as a JSON/base64 body is never rejected, while an oversized
body is either:

  • Rejected immediately with 413 if Content-Length declares it oversized, or
  • Rejected by MaxBytesReader mid-read if Content-Length is absent.

WebSocket upgrade requests carry no body, so the wrapper is a no-op for them.

Testing

Manual testing with a 256 MiB base64 key in a /v3/kv/range request:

  • Before: RSS grows ~1.75 GiB; request eventually returns 429 (rate-limited)
  • After: 413 returned immediately; RSS stays flat

Fixes #21555

Without a body-size guard, an unauthenticated HTTP/JSON request to any
/v3/* gRPC-gateway endpoint can force etcd to read and decode an
arbitrarily large request body before authentication or MaxRequestBytes
enforcement, causing GiB-scale RSS growth from a single 256 MiB request.

Fix: in accessController.ServeHTTP, wrap req.Body with
http.MaxBytesReader (limit = MaxRequestBytes*4) for all /v3/ paths
before delegating to the mux. The 4x multiplier accounts for base64
encoding overhead (~4/3) and JSON field-name framing, ensuring that a
legitimate maximum-size payload encoded as JSON is never rejected.
Requests that declare an oversized Content-Length are rejected with
413 before any body bytes are read.

WebSocket upgrade requests have an empty body, so the MaxBytesReader
wrapper is a no-op for them.

Fixes etcd-io#21555

Signed-off-by: crawfordxx <crawfordxx@users.noreply.github.com>
@kubernetes-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: crawfordxx
Once this PR has been reviewed and has the lgtm label, please assign jmhbnz for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow

Copy link
Copy Markdown

Hi @crawfordxx. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

gRPC-gateway HTTP /v3/* endpoints allow unauthenticated memory exhaustion via oversized JSON/base64 request body amplification

1 participant