Investigate OTLP integration

[jamescrosswell]

Description

• Developer docs
• Python reference implementation
  • Documentation
  • Pull request

Note

The Exception Interceptor is something we already have in the POTEL implementation for .NET but have problems with. This will be deprecated in the OTEL APIs. The recommendation is to use the OTEL logging APIs for this instead.

Initial head scratching

Currently the meat of our OTEL integration is implemented in the SentrySpanProcessor. Basically it takes OTEL spans and converts these into Sentry spans, stored on the Sentry scope. We have a couple of different scope implementations: global and AsyncLocal... since Activity.Current (Microsoft's implementation of OTEL) is AsyncLocal, this works best in backend applications. In front end apps using a global scope, often spans from background services can end up interleaved with front end operations.

From what I can gather from the docs and the Python implementation, moving to OTLP would flip this on it's head and we'd instead send Sentry instrumented spans to an OTEL exporter which would pipe both the Sentry spans and any spans from the user's code through to a DSN specific OLTP endpoint on the Sentry server/ingest.

Problems so solve

The SentrySpanProcessor currently ensures OTEL and Sentry spans are parented to one another correctly:

sentry-dotnet/src/Sentry.OpenTelemetry/SentrySpanProcessor.cs

Lines 96 to 114 in 54511b5

	if (data.ParentSpanId != default && _map.TryGetValue(data.ParentSpanId, out var mappedParent))
	{
	// Explicit ParentSpanId of another activity that we have already mapped
	CreateChildSpan(data, mappedParent, data.ParentSpanId);
	}
	// Note if the current span on the hub is OTel instrumented and is not the parent of `data` then this may be
	// intentional (see https://opentelemetry.io/docs/languages/net/instrumentation/#creating-new-root-activities)
	// so we explicitly exclude OTel instrumented spans from the following check.
	// of Sentry
	else if (_hub.GetSpan() is IBaseTracer { IsOtelInstrumenter: false } inferredParent)
	{
	// When mixing Sentry and OTel instrumentation and we infer that the currently active span is the parent.
	var inferredParentSpan = (ISpan)inferredParent;
	CreateChildSpan(data, inferredParentSpan, inferredParentSpan.SpanId);
	}
	else
	{
	CreateRootSpan(data);
	}

sentry-dotnet/src/Sentry.OpenTelemetry/SentrySpanProcessor.cs

Line 175 in 54511b5

_hub.ConfigureScope(static (scope, transaction) => scope.Transaction = transaction, transaction);

It also does a bit of parsing to synthesise errors and enrich the spans:

sentry-dotnet/src/Sentry.OpenTelemetry/SentrySpanProcessor.cs

Lines 274 to 280 in 54511b5

	GenerateSentryErrorsFromOtelSpan(data, attributes);
	var status = GetSpanStatus(data.Status, attributes);
	foreach (var enricher in _enrichers)
	{
	enricher.Enrich(span, data, _hub, _options);
	}

So I guess a couple of logical questions arise:

1. How do we ensure Sentry and OTEL spans are interleaved/parented to one another correctly?

• And how does this work with both backend applications like ASP.NET Core and front end applications like MAUI or WinUI?

3. Do we still want/need to do any additional error synthesis or enrichment and, if so, is this also possible when using the OLTP approach?

Why?

Probably the biggest question I have is, "Why would we want to do all of this at all?". For that, ideally we'd have an example of a concrete customer problem that exists when using POTEL (our current approach to OTEL integration) and a clear explanation of how a move to OTLP would solve that... as well as some confidence that moving to OTLP wouldn't trade our current problems for new ones that we don't currently have.

[linear]

NET-567

[sl0thentr0py]

@jamescrosswell

From what I can gather from the docs and the Python implementation, moving to OTLP would flip this on it's head and we'd instead send Sentry instrumented spans to an OTEL exporter which would pipe both the Sentry spans and any spans from the user's code through to a DSN specific OLTP endpoint on the Sentry server/ingest.

No, in the case of OTLP, users will use OTEL entirely for their instrumentation. No more cross interleaving of spans from both Sentry and OTEL. So they will turn tracing off on the Sentry side. We are decoupling and not dealing with the cross-span mess at all now.

[jamescrosswell]

No, in the case of OTLP, users will use OTEL entirely for their instrumentation. No more cross interleaving of spans from both Sentry and OTEL. So they will turn tracing off on the Sentry side. We are decoupling and not dealing with the cross-span mess at all now.

Oh, so none of the tracing from our own integrations would work anymore @sl0thentr0py ?

Are we looking at supporting both instrumentation methods so that SDK users can still choose to enable either Sentry native tracing or OTEL via OTLP? Or are we looking to drop the Sentry instrumentation altogether so that SDK users would need to use OTEL in order to see traces?

I think there are community OTEL replacements for the tracing emitted by our Web and DB integrations. Probably only the stuff we did for MAUI had no OTEL equivalent.

We could change our MAUI integration to emit OTEL instrumentation instead or as well...

[sl0thentr0py]

Are we looking at supporting both instrumentation methods so that SDK users can still choose to enable either Sentry native tracing or OTEL via OTLP?

yes, it's a hard choice between instrumentation providers, either Sentry or OTEL.

Or are we looking to drop the Sentry instrumentation altogether so that SDK users would need to use OTEL in order to see traces?

For the considerable future, we will continue to maintain and build our own Sentry instrumentation that is best in class for certain frameworks but as OTEL matures eventually, we will consider dropping our own. But that's something for the future.

Basically we gave up on the (unjustified) complexity of having span interleaving and are opening up a path for enterprise-y users who prefer the OTEL stack to just send us that data directly and making sure our Trace Connectedness still works for other events (Errors/Logs/Metrics).

[jamescrosswell]

@sl0thentr0py I just started building this for .NET in #4899.

I'm still not really sure why we're doing this... unless it improves things on the ingestion side for Sentry. From what I can tell, there are two main reasons why SDK users would want to use OTLP:

1. They also want to send telemetry to that endpoint from processes that are not instrumented using a Sentry SDK... and they want all the telemetry in the sam place. However, this can be done today without any changes to the SDKS (Sentry already exposes collector endpoints that don't need to be configured in the SDKs)
2. They want all their telemetry to go to a collector so that they can change where that telemetry is sent without having to recompile of reconfigure the appilcations where that telemetry comes from... for example, they could tell the collector to send the data to Prometheus as well as or instead of Sentry, without touching the code for the apps. However they wouldn't be able to do that with Sentry's collector, since Sentry's collector just sends the data to Sentry.

I'm guessing there's something I'm missing here. Why were your customers asking for this?

EDIT: Ah actually I see we have a Routing Connector... so it'd possible to have telemetry sent to multiple projects. If this is OTLP specific, that would be a differentiation between OTLP and POTEL.

[sl0thentr0py]

However, this can be done today without any changes to the SDKS (Sentry already exposes collector endpoints that don't need to be configured in the SDKs)

yes but they wouldn't get Trace Connectedness for other Sentry events like Errors, Logs, Metrics and Replays which is what this integration mainly enables with the external propagation context stuff.

As to the choice of collector vs exporter setup in OTLPIntegration, they can use either but we slightly prefer an out of the box experience as in this integration since it sets up the endpoint from the DSN without them having to search for it in docs/product.

The Routing Connector is a parallel project that is mainly intended for some large enterprises who want routing to different projects. For the 90% use case, we still prefer the one project per service config. Ultimately, Sentry doesn't want to enforce users to run other things on their infra with the exception being large orgs who have special needs.
@dingsdax maybe you want to elaborate here on the business cases for the OTLPIntegration vs the Routing Connector.