HttpClient / ZstandardStream silently truncates multi-frame zstd responses to the first frame (.NET 11 preview 4)

[christosk92]

TL;DR

After upgrading a desktop app from .NET 10 to .NET 11 preview 4, large HTTP responses started getting silently truncated. It looks like the new zstd decompression path treats a single completed frame as the end of the whole stream, so any Content-Encoding: zstd response made of multiple concatenated frames loses everything after the first frame — with no exception.

How I ran into it

I maintain a WinUI 3 client that talks to Spotify's Pathfinder GraphQL endpoint (https://api-partner.spotify.com/pathfinder/v2/query). My HttpClient handler is configured the obvious way:

new SocketsHttpHandler { AutomaticDecompression = DecompressionMethods.All }

This worked fine on .NET 10. The moment I retargeted to .NET 11 preview 4 (11.0.100-preview.4.26230.115), the bigger GraphQL calls (home feed, artist overview) began failing while small ones kept working. The failure surfaced in System.Text.Json as a body that just ends partway through:

System.Text.Json.JsonException: Expected end of string, but instead reached end of data.
Path: $.data.home.sectionContainer.sections.items[3].sectionItems.items[0].content.data
| LineNumber: 0 | BytePositionInLine: 65536.
   ...
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonTypeInfo`1 jsonTypeInfo)
   at <my code>.PathfinderClient.QueryAsync[T](...)

Always at BytePositionInLine: 65536 (the body is single-line minified JSON, hence LineNumber: 0). It truncated identically whether I read via ReadAsStringAsync() + JsonSerializer.Deserialize(string) or ReadAsStreamAsync() + DeserializeAsync(stream), so the body was already 64 KB by the time my code touched it — upstream of my read.

What changed between .NET 10 and 11 for this handler: DecompressionMethods.All is -1 (all bits), and .NET 11 added DecompressionMethods.Zstandard = 8. So on .NET 11 my client started advertising Accept-Encoding: …, zstd and the server switched to zstd. Confirmed with a raw request in Postman:

Request : Accept-Encoding: zstd
Response: content-encoding: zstd
          content-length: 10086      <- the full compressed body (~10 KB) arrives intact

So the compressed body is delivered completely (this is not an HTTP/2 / transport truncation), but it decompresses to well over 64 KB of JSON and gets cut at exactly 65536 bytes — i.e. the server emits the payload as multiple concatenated zstd frames (~64 KB each) and .NET stops after the first one.

Minimal repro (no Spotify, no auth, deterministic)

using System.IO.Compression;

static byte[] Frame(byte[] data)
{
    using var ms = new MemoryStream();
    using (var z = new ZstandardStream(ms, CompressionMode.Compress, leaveOpen: true))
        z.Write(data, 0, data.Length);
    return ms.ToArray();
}

var a = new byte[100_000]; var b = new byte[100_000];
Array.Fill(a, (byte)'A'); Array.Fill(b, (byte)'B');

// Two concatenated zstd frames = one valid zstd stream (RFC 8878 §3).
byte[] body = [.. Frame(a), .. Frame(b)];

using var input = new MemoryStream(body);
using var z = new ZstandardStream(input, CompressionMode.Decompress);
using var outMs = new MemoryStream();
z.CopyTo(outMs);

Console.WriteLine(outMs.Length);   // expected 200000, actual 100000

Expected: 200000 — both frames (A×100000 then B×100000).
Actual: 100000 — first frame only; the second frame is silently dropped.

I also reproduced the exact path my app hits — HttpClient with AutomaticDecompression = DecompressionMethods.Zstandard against a local HttpListener that serves the same two-frame body with Content-Encoding: zstd. Full matrix from the repro:

Runtime      : .NET 11.0.0-preview.4.26230.115
Architecture : OS=Arm64, Process=Arm64

PASS  ZstandardStream  | single frame (200 KB)  expected=  200000  actual=  200000
FAIL  ZstandardStream  | 2 concatenated frames  expected=  200000  actual=  100000
PASS  HttpClient(zstd) | single frame (200 KB)  expected=  200000  actual=  200000
FAIL  HttpClient(zstd) | 2 concatenated frames  expected=  200000  actual=  100000

A single frame larger than 64 KB decompresses correctly, so this is not an output-buffer-size limit — it is strictly the second frame. Runnable repro project: https://github.com/christosk92/zstd-net11-repro

Where it might be

I haven't dug into the internals, but it looks like it's in the new zstd decompression support (the System.IO.Compression.Zstandard stream/decoder used by HttpClient automatic decompression, around #123531) — specifically the decoder appearing to stop at the first frame rather than continuing across concatenated frames, which ZSTD_decompressStream supports natively. Happy to dig further if useful; I'll leave the exact spot to you.

Why this is worth fixing before RTM

• Silent data corruption — no exception, just a short result. In my case it only showed up as a downstream System.Text.Json parse error; a consumer not parsing strict JSON would get truncated data with no signal at all.
• Only large responses are affected (small single-frame bodies work), so it slips through casual testing and bites at production-sized payloads.
• DecompressionMethods.All silently opts everyone in. Code that set AutomaticDecompression = DecompressionMethods.All on .NET ≤10 had no zstd; on .NET 11 the same line negotiates zstd, so existing apps inherit this the moment they retarget — against any server/CDN that emits multi-frame Content-Encoding: zstd (common; many encoders flush a frame per buffer).

Workaround (for anyone who lands here)

Exclude zstd until this is fixed; the server falls back to brotli, which decodes correctly:

AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate | DecompressionMethods.Brotli

Configuration

• .NET SDK 11.0.100-preview.4.26230.115 / runtime 11.0.0-preview.4.26230.115
• Windows 11, ARM64
• Affected: System.IO.Compression.ZstandardStream, HttpClient / SocketsHttpHandler AutomaticDecompression = DecompressionMethods.Zstandard (and therefore DecompressionMethods.All)

Related

• Zstd HTTP automatic decompression added in Add DecompressionMethods.Zstandard for HTTP automatic decompression #123531
• RFC 8878 §3 — concatenated zstandard frames

[dotnet-policy-service]

Tagging subscribers to this area: @karelz, @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.