[Change Proposal] Support OTel input packages that collect data with multiple signal types

[jsoriano]

Package policies for input packages are currently designed to collect a single signal type, and send it to a single data stream.

In the case of OTel, a single receiver can collect multiple signal types if it is connected to pipelines of different types, as seen in elastic/integrations#16003

Changes in package-spec

• Define a new field available_types in the manifest that allows the author to specify the signal types available for the package, for instance:

available_types: [logs, metrics, traces]

Changes in Fleet

• Generate a different transform for each signal defined in available_types - example in case of all three signals present:

processors:
  transform/foo
    log_statements: 
      - set(attributes["data_stream.type"], "logs")
      - set(attributes["data_stream.dataset"], "<policy_template_name>")
      - set(attributes["data_stream.namespace"], "<namespace>")
    metric_statements: 
      - set(attributes["data_stream.type"], "metrics")
      - set(attributes["data_stream.dataset"], "<policy_template_name>")
      - set(attributes["data_stream.namespace"], "<namespace>")
    trace_statements: 
      - set(attributes["data_stream.type"], "traces")
      - set(attributes["data_stream.dataset"], "<policy_template_name>")
      - set(attributes["data_stream.namespace"], "<namespace>")

• in case of for OTel input packages, update the UI element in "advanced options" that allows to select the datastream type so that the radio buttons for data stream type conform to the configured available_types.
  e.g. a user can toggle between logs or metrics when available_types: [logs, metrics] is set, but can't select traces

The changes can be tested by compiling the test package defined in elastic/integrations#16003 with the new available_types field.

Original description

We have to design how this should work, as this has some implications, between them: * There are currently UI elements to select the data stream type, should they be disabled for OTel input packages with multiple pipelines? * Should support for multiple signal types in a package detected automatically or indicated by some flag in the manifest? * How would routing work? Using fixed indexes in the exporter, different transforms for different dynamic mappings? Routing on ingestion? Should this be configurable per package? ...

[cmacknz]

I think for elastic/integrations#16003 that we don't want this to behave any differently than it does in a standalone agent.

I also don't think this is the first input that has this problem or that it is specific to otelcol packages, for example a kafka input package can also read documents of any type.

Both OTLP and Kafka have been described as "conduit" inputs where data passes through them, they aren't individually responsible for collection or for routing decisions by default, but processing can be added to control or influence routing.

[cmacknz]

The simplest case of an OTLP receiver is that the data hits the OTLP exporter in the pipeline with no transformations and the data is routed based on it's attributes or scope values.

This is also true of a Beat based Kafka -> Kafka or Kafka -> Elasticsearch gateway.

[kpollich]

Our previous generation solution for the Agent/Kafka use case was routing rules defined by integrations that are collected and written to ingest pipelines in the form of reroute processors elastic/kibana#151898.

This doesn't align with the OTel native model, though, and we need a way to adhere to the routing scheme laid out in https://www.elastic.co/docs/reference/edot-collector/components/elasticsearchexporter#document-routing

There are currently UI elements to select the data stream type, should they be disabled for OTel input packages with multiple pipelines?

Yes. Disabling them when a package specifies multiple signal types is sensible.

Additionally, I think these should be made less prominent and moved to "advanced settings" in general. It shouldn't be the norm to use a "custom logs" integration to collect metrics, and we should only support this use case for power users off the happy path.

Should support for multiple signal types in a package detected automatically or indicated by some flag in the manifest?

Detecting these automatically might be tricky since the available signals are configured in .yml.hbs files which we can't actually parse on the integration side. The simplest way to support might be a manifest field available_signals: [logs, metrics, traces].

How would routing work? Using fixed indexes in the exporter, different transforms for different dynamic mappings? Routing on ingestion? Should this be configurable per package?

Fleet could generate different transforms for each collected signal e.g.

processors:
  transform/foo
    log_statements: 
      - set(attributes["data_stream.type"], "logs")
      - set(attributes["data_stream.dataset"], "<policy_template_name>")
      - set(attributes["data_stream.namespace"], "<namespace>")
    metric_statements: 
      - set(attributes["data_stream.type"], "metrics")
      - set(attributes["data_stream.dataset"], "<policy_template_name>")
      - set(attributes["data_stream.namespace"], "<namespace>")
    trace_statements: 
      - set(attributes["data_stream.type"], "traces")
      - set(attributes["data_stream.dataset"], "<policy_template_name>")
      - set(attributes["data_stream.namespace"], "<namespace>")

[criamico]

Trying to summarize the discussion into actionable items:

Changes in package-spec

• Define a new field available_types in the manifest that allows the author to specify the signal types available for the package, for instance:

available_types: [logs, metrics, traces]

Changes in Fleet

• Generate a different transform for each signal defined in available_types - example in case of all three signals present:

processors:
  transform/foo
    log_statements: 
      - set(attributes["data_stream.type"], "logs")
      - set(attributes["data_stream.dataset"], "<policy_template_name>")
      - set(attributes["data_stream.namespace"], "<namespace>")
    metric_statements: 
      - set(attributes["data_stream.type"], "metrics")
      - set(attributes["data_stream.dataset"], "<policy_template_name>")
      - set(attributes["data_stream.namespace"], "<namespace>")
    trace_statements: 
      - set(attributes["data_stream.type"], "traces")
      - set(attributes["data_stream.dataset"], "<policy_template_name>")
      - set(attributes["data_stream.namespace"], "<namespace>")

• in case of for OTel input packages, update the UI element in "advanced options" that allows to select the datastream type
  so that the radio buttons for data stream type conform to the configured available_types.
  • e.g. a user can toggle between logs or metrics when available_types: [logs, metrics] is set, but can't select traces

The changes can be tested by compiling the test package defined in elastic/integrations#16003 with the new available_types field.

[jsoriano]

Proposal looks good to me, maybe we could use available_types instead of available_signals, as we are using "type" now for the data stream type.

[kpollich]

Should the radio buttons for data stream type conform to the configured available_types as well? e.g. a user can toggle between logs or metrics when available_types: [logs, metrics] is set, but can't select traces?

[cmacknz]

Changes in Fleet
Generate a different transform for each signal defined in available_signals - example in case of all three signals present:

Would this be better suited as something that gets introduced as part of the base package concept in composable integrations? I am a little cautious about generating collector configuration fragments like this in Fleet instead of keeping it in the package (in case we need to change it or the precedence order, or have it be different per agent/collector version, etc).

[criamico]

Should the radio buttons for data stream type conform to the configured available_types as well? e.g. a user can toggle between logs or metrics when available_types: [logs, metrics] is set, but can't select traces?

@kpollich yes that would make sense, instead of totally removing these UI elements we would show only the ones that are present.