Built-in monitoring & alerting for stuck / unprocessed envelopes

[dalnoki]

Problem Statement

A customer encountered an issue where their application silently stopped sending events to Sentry. After local investigation, they discovered a number of .envelope files accumulating in the SDK’s local Sentry folder including very large files (~2GB) and very old files (dating back to February).

After manually clearing the folder, the Sentry SDK immediately resumed normal operation.

To prevent this kind of silent failure, they are requesting a built-in mechanism in the Sentry SDK that can detect envelope backlogs and surface an alert or warning to the Sentry dashboard (or at least application logs) so teams can act before ingestion silently stops.

Solution Brainstorm

No response

[linear]

JAVA-254

[adinauer]

@dalnoki is this Android, JVM Desktop or something else?

[mhogerheijde]

Hi, original reporter here.

This is on JVM 21, Temurin runtime. Desktop I guess? I mean: we're running it on a server 😁.

We're using Java modules io.sentry:sentry and io.sentry:sentry-logback both version 8.26.0.

On Nov. 6th we switched from 8.23.0 to 8.25.0
On Nov. 18th we switched from 8.25.0 to 8.26.0

The oldest unsent .envelope file in the system was from February, so none of these versions was able to clean up the backlog. We noticed that Sentry stopped sending since we sent a "startup warning" every release in order to verify we didn't break anything. We release every week and upon inspection Sentry SDK stopped sending messages somewhere halfway the release.

Unfortunately, the engineer that worked on the production machine removed the envelopes instead of moving them for later inspection. So I can't tell you what was in the envelopes so you can reproduce.

Please let me know if we can be of any (further) assistance.

[adinauer]

Thanks for the additional details @mhogerheijde.

Just to make sure, you did configure cacheDirPath, correct?

Let me check if there's some cleanup we perform on Android but don't on JVM.