EW-9735 EW-9736 [o11y] Drop tail stream events when reaching excessive queue size #5637

fhanau · 2025-12-03T21:15:48Z

This will avoid excessive memory overhead in a few edge cases – usually the tail worker will be able to keep up with reporting events but we still need to put a limit to the queue size.

fhanau · 2025-12-03T22:03:17Z

src/workerd/io/tracer.c++

+      tracing::SpanOpen(span.spanId, span.operationName.clone()), span.startTime, spanNameSize);
  // If a span manages to exceed the size limit, truncate it by not providing span attributes.
-  if (span.tags.size() && messageSize <= MAX_TRACE_BYTES) {
+  if (span.tags.size() && spanTagsSize <= MAX_TRACE_BYTES) {


As the watchful reviewer will notice, this is a minor functional change: We're no longer including the size of the span name in this check since it is included in a different tail event, so we're slightly relaxing the size limit for emitting span tags here.

src/workerd/io/trace-stream.c++

fhanau · 2025-12-03T22:06:13Z

src/workerd/io/trace-stream.c++

 // The TailStreamWriterState holds the current client-side state for a collection
 // of streaming tail workers that a worker is reporting events to.
 struct TailStreamWriterState {
+  // The maximum size of the queue, in bytes.


Open to different names for these constants/defining them somewhere else.

Note that testing that these checks take effect at some point will be provided in a downstream PR (to follow). This will also include a discussion of why they are sufficient for almost all use cases.

fhanau · 2025-12-03T22:10:01Z

This is now feature-complete. As noted in a PR comment – tests for this and rationale will be provided in a downstream PR, but I think we can already discuss the merits of the code changes here.

src/workerd/io/trace.h

src/workerd/io/trace-stream.c++

mar-cf · 2025-12-10T13:58:04Z

Add tests

mar-cf

My biggest concern is if it's possible to produce inconsistent traces due to dropping.

src/workerd/io/trace-stream.h

src/workerd/io/tracer.c++

src/workerd/io/trace-stream.c++

mar-cf · 2026-01-02T11:46:25Z

Offline the team discussed some concerns. I'm going to assume this isn't blocked on me until I see those updates. Feel free to ping if I'm needed.

fhanau · 2026-01-08T18:38:40Z

Add tests

Tests for this are available downstream, in #11842 and #12029.

mar-cf · 2026-01-14T16:39:55Z

src/workerd/io/trace-stream.c++

+        auto log = kj::str(
+            "[\"Dropped ", active->droppedEvents, " tail events due to excessive queueing\"]");
+        TailEvent droppedEventsLog(event.spanContext.clone(), event.invocationId, event.timestamp,
+            event.sequence, tracing::Log(event.timestamp, LogLevel::WARN, kj::mv(log)));


If we're injecting a synthetic log just before the outcome, why not just report it as a field in the outcome event? There's no chance for the handler to act on a drop signal at this point anyway.

I discussed this with @zebp at the last or second last meeting – he suggested to just have a log for now, rationale is that we'll likely want something more complex (perhaps a new event type for stream events) but this is not something we need at this stage, having a temporary log is better than changing the Outcome event API.

Matching against this exact string and parsing out the number dropped in the log handler, could be done but doesn't seem very robust. Also this string could change and silently break, and it could be crafted artificially.

I'm going to assume @zebp confirmed, but we should reconsider that decision.

A log is fine for now, but we should extract this out into a separate event before STWs are shipped publicly. As @fhanau mentioned I'd rather not complicate the Outcome event by adding a variant that is specific to STWs and would cause STW outcomes to not match regular outcomes.

@mar-cf does make a good point that this does make it a bit hard to parse, can we instead emit a JSON structured log (with a special value to prevent colisions with customer logs) so we can not rely on brittle string parsing?

{"$":"cloudflare-streaming-tail-workers-internal","type":"dropped","count":0} would be fine.

Wouldn't it take a day to just add an event type? Why are we hacking?

@mar-cf updated the PR based on Zeb's suggestion – this produces a structured log now, the downstream test has been updated to expect that log.

mar-cf

My objections are noted. I won't block on them since this gets us functionally there. Maybe down the line, I'll get around to them.

github-actions · 2026-01-21T04:59:23Z

The generated output of @cloudflare/workers-types matches the snapshot in types/generated-snapshot 🎉

This will avoid excessive memory overhead in a few edge cases.

codecov-commenter · 2026-01-22T22:40:56Z

Codecov Report

❌ Patch coverage is 52.88462% with 49 lines in your changes missing coverage. Please review.
✅ Project coverage is 69.65%. Comparing base (2752663) to head (a5ae1f8).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
src/workerd/io/trace-stream.c++	29.41%	21 Missing and 3 partials ⚠️
src/workerd/io/trace.c++	48.48%	11 Missing and 6 partials ⚠️
src/workerd/io/tracer.c++	76.92%	5 Missing and 1 partial ⚠️
src/workerd/io/trace-test.c++	81.81%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #5637       +/-   ##
===========================================
+ Coverage   68.10%   69.65%    +1.54%     
===========================================
  Files          29      397      +368     
  Lines        2593   106075   +103482     
  Branches       15    17978    +17963     
===========================================
+ Hits         1766    73885    +72119     
- Misses        827    21386    +20559     
- Partials        0    10804    +10804

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

fhanau · 2026-01-22T23:28:18Z

src/workerd/io/worker-interface.capnp

+    # events diagnostic is supported. We do not need to include the diagnostic type here (it is
+    # always the same for a given diagnostic) so droppedEvents can be represented using just the
+    # count variable.
+    diagnostic :union {


capnp does not allow a single member union – is there a better way to make this type easily extensible than having an unused "undefined" value? In practice, this should be fine as undefined should only take up one bit in the serialized message.

fhanau force-pushed the felix/112625-stw-load-shed branch from 3a09407 to 7692d3e Compare December 3, 2025 22:01

fhanau commented Dec 3, 2025

View reviewed changes

src/workerd/io/trace-stream.c++ Outdated Show resolved Hide resolved

fhanau commented Dec 3, 2025

View reviewed changes

fhanau marked this pull request as ready for review December 3, 2025 22:08

fhanau requested review from a team as code owners December 3, 2025 22:08

fhanau added the observability label Dec 3, 2025

fhanau requested a review from mar-cf December 3, 2025 22:08

fhanau commented Dec 3, 2025

View reviewed changes

src/workerd/io/trace.h Outdated Show resolved Hide resolved

fhanau commented Dec 3, 2025

View reviewed changes

src/workerd/io/trace-stream.c++ Outdated Show resolved Hide resolved

fhanau force-pushed the felix/112625-stw-load-shed branch 2 times, most recently from 70cfa7f to 51d612b Compare December 16, 2025 02:25

mar-cf reviewed Dec 17, 2025

View reviewed changes

src/workerd/io/trace-stream.h Show resolved Hide resolved

src/workerd/io/tracer.c++ Show resolved Hide resolved

src/workerd/io/trace-stream.c++ Show resolved Hide resolved

fhanau changed the title ~~[o11y] Drop tail stream events when reaching excessive queue size~~ EW-9735 EW-9736 [o11y] Drop tail stream events when reaching excessive queue size Jan 9, 2026

fhanau force-pushed the felix/112625-stw-load-shed branch 2 times, most recently from a0cc9a9 to db917db Compare January 9, 2026 17:02

This comment was marked as outdated.

Sign in to view

fhanau force-pushed the felix/112625-stw-load-shed branch 3 times, most recently from 1aecc65 to 0074510 Compare January 13, 2026 19:50

mar-cf reviewed Jan 14, 2026

View reviewed changes

mar-cf approved these changes Jan 14, 2026

View reviewed changes

fhanau force-pushed the felix/112625-stw-load-shed branch 2 times, most recently from 3f788be to 9e67080 Compare January 21, 2026 04:52

fhanau requested a review from a team as a code owner January 21, 2026 04:52

fhanau force-pushed the felix/112625-stw-load-shed branch from 9e67080 to 61386e4 Compare January 21, 2026 19:46

fhanau added 2 commits January 22, 2026 16:53

[o11y] Drop tail stream events when reaching excessive queue size

4f983f4

This will avoid excessive memory overhead in a few edge cases.

[o11y] Implement StreamDiagnosticsEvent in STW tail protocol

5751154

fhanau force-pushed the felix/112625-stw-load-shed branch from 61386e4 to 922fcde Compare January 22, 2026 22:17

fhanau force-pushed the felix/112625-stw-load-shed branch from 922fcde to ccf98d3 Compare January 22, 2026 23:26

fhanau commented Jan 22, 2026

View reviewed changes

[STW] Update StreamDiagnosticsEvent API definition

a5ae1f8

fhanau force-pushed the felix/112625-stw-load-shed branch from ccf98d3 to a5ae1f8 Compare January 23, 2026 15:31

EW-9735 EW-9736 [o11y] Drop tail stream events when reaching excessive queue size #5637

Are you sure you want to change the base?

EW-9735 EW-9736 [o11y] Drop tail stream events when reaching excessive queue size #5637

Uh oh!

Conversation

fhanau commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

fhanau Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fhanau commented Dec 3, 2025

Uh oh!

Uh oh!

Uh oh!

mar-cf commented Dec 10, 2025

Uh oh!

mar-cf left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mar-cf commented Jan 2, 2026

Uh oh!

fhanau commented Jan 8, 2026

Uh oh!

This comment was marked as outdated.

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mar-cf left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

fhanau commented Dec 3, 2025 •

edited

Loading

fhanau Dec 3, 2025 •

edited

Loading

github-actions bot commented Jan 21, 2026 •

edited

Loading

codecov-commenter commented Jan 22, 2026 •

edited

Loading