RFC: Multiple exporter support

[mheffner]

Intro

At the moment Rotel supports a single exporter, we would like to expand support for multiple exporters. This will also allow you to emit different telemetry types when a given exporter may only support a single type (e.g AWS X-Ray).

Goals

• Allow different telemetry types to be sent to different destinations, e.g traces to XRay, logs to Clickhouse, and metrics to Honeycomb.
• Allow multiple exporters to be configured entirely via environment variables. We realize that eventually we'll need to support more advanced configuration with config files (toml, yaml, json, etc), but we'd like to delay that requirement for now.
• Allow a single telemetry type to be sent to multiple destinations

Non-goals

• Processing, filtering, transforming telemetry per pipeline. We are in the process of evaluating our Python Processor SDK and native Rust processors, so we'd like to decouple multiple exporters from that process.
• Full support for CLI arguments for multiple exporters. While current configuration options can be passed as command line arguments to Rotel, most configuration for pyrotel or the Lambda extension are done via environment variables.

Requirements

• Existing configurations should still work as expected. While we may need to break backwards compatibility in the future, ideally we don’t need to break it now.
• Exporters with different batching requirements should be able to receive telemetry from the same pipeline. This may be require some batch splitting downstream of the core batching.
• Default exporter property values remain the same.

Configuration

Existing configuration

Generally the existing exporter configuration works like the following. We should still support this model of configuration for now.

ROTEL_EXPORTER=<datadog|clickhouse|...>
ROTEL_<EXPORTER_TYPE>_EXPORTER_<EXPORTER PROPERTY>=<custom value>

Expanded Configuration

Say I have the following exporter configuration I’d like to support:

• Two Clickhouse destinations:
  • “debug”: receives traces and logs
  • “production”: receives traces

Define the named exporters and their types:

ROTEL_EXPORTERS=debug:clickhouse,production:clickhouse

Setting exporter properties per exporter:

ROTEL_EXPORTER_DEBUG_ENDPOINT=https://debug.clickhouse.cloud
ROTEL_EXPORTER_PRODUCTION_ENDPOINT=https://prod.clickhouse.cloud
ROTEL_EXPORTER_PRODUCTION_DATABASE="otel_prod"

# General format is:
# ROTEL_EXPORTER_<NAME>_<EXPORTER PROPERTY>=<custom value>

Connecting exporters to pipelines:

ROTEL_EXPORTERS_TRACES=debug,production
ROTEL_EXPORTERS_LOGS=debug

Metrics would not be connected and hence we would not accept metrics at the receiver.

Simplified form

In a simplified case, if there is only one exporter of a given type, you can elide the custom name. In that scenario, the name is the same as the exporter type.

For example, given Clickhouse and Datadog exporter destinations the following configuration would be allowed:

ROTEL_EXPORTERS=clickhouse,datadog
ROTEL_EXPORTER_CLICKHOUSE_ENDPOINT=https://prod.clickhouse.cloud
ROTEL_EXPORTER_CLICKHOUSE_DATABASE="otel_prod"
ROTEL_EXPORTER_DATADOG_REGION=us1

ROTEL_EXPORTERS_TRACES=datadog,clickhouse
ROTEL_EXPORTERS_LOGS=clickhouse

Configuration caveats

• Using ROTEL_EXPORTERS and ROTEL_EXPORTER would not be supported together
• Setting ROTEL_EXPORTERS requires at least one of ROTEL_EXPORTERS_<TRACES|METRICS|LOGS> to be set. All unset telemetry types would default to the “no listeners” setting, thereby dropping traffic.
• Exporter names are case insensitive and use the character set: [a-z0-9].
• It’s an error to assign an exporter to a telemetry type it does not support. For example, assigning an xray exporter to ROTEL_EXPORTERS_LOGS would be an error during initialization.
• The OTLP exporter allows per-telemetry type properties, with this new approach we may be able to relax those and allow the user to simply define more than one named OTLP exporter.

Next steps

1. Share discussion and collect any feedback.
2. Prototype dynamic configuration handling in combination with existing options. May require config libraries outside of the current clap crate.
3. Deliver multiple exporter support with single exporter per telemetry type.
4. Follow up with multiple exporters allowing multiple exporters per telemetry type. This will require some additional thought on back-pressure handling.

[lucky]

It might be worth calling out the behavior when one exporter fails and another succeeds.

Exporters with different batching requirements should be able to receive telemetry from the same pipeline. This may be require some batch splitting downstream of the core batching.

This makes me think "core" batching should be the de facto minimum, whereas downstream exporters can only batch more than this, not less?

[mheffner]

It might be worth calling out the behavior when one exporter fails and another succeeds.

Agreed, I've included that process a bit more in step 4 when we add multiple exporters for a single pipeline. I'd want to compare with the behavior of the existing collector here.

This makes me think "core" batching should be the de facto minimum, whereas downstream exporters can only batch more than this, not less?

So we just merged support that will allow an exporter to split payloads based on the upstream batches. This is mostly for Clickhouse metrics support at the moment, but it could also help with re-splitting/re-batching from an upstream batches. Therefore, I'm a bit torn on what the best upstream batching looks like.

One challenge is that it may not be easy to calculate what the batch size of generic OTLP types should be for any given exporter. An exporter may convert to a different format and the aggregate batch size of that new format may vary based on the size of OTLP attributes. So we can't tell the size to batch ResourceSpans, for example, until we've converted them and byte packed them into an exporters outgoing payload type. At that point we may need to split a payload if it exceeds the exporter's destination limits.

[mugli]

👋

We have a use-case for this goal:

Allow a single telemetry type to be sent to multiple destinations

Where this non-goal isn't a problem:

Processing, filtering, transforming telemetry per pipeline

This is the intended flow:

As an alternative, we could fanout the traces from the other OTel collector in the diagram, but self-service for teams owning different lambdas is more important where they can configure if they want to use the traces for guarded rollout, so we are not considering that approach.

We can give the Rotel lambda layer a try (for performance reason) when multiple destinations support is added. :)

[mheffner]

@mugli Thanks for the clarification and added diagram. We should have multiple exporter destinations ready for testing shortly, this should be the remaining PR to add this support: #211

[mheffner]

Hey @mugli

This is now available in the latest Lambda release: https://github.com/streamfold/rotel-lambda-extension/releases/tag/v36-alpha

Checkout the Multiple Exporters documentation for how to configure multiple exporters and send telemetry to multiple configured exporters. Let us know how it goes!

[mugli]

Awesome! Going to give it a try this week 🥳

[mheffner]

The design here has been implemented and released so far. Closing this discussion for now.