Figure out how to support more OTel knobs

[dprotaso]

TIL: there's an almost stable config that will work across OTel implementation here:

go implementation https://pkg.go.dev/go.opentelemetry.io/contrib/otelconf/v0.3.0
spec: open-telemetry/opentelemetry-specification#4374

We could support additional OTel knobs by allowing operators to specify such a config in config-observability

Then we could merge the existing keys with the keys in said config.

Alternatively, given that config is pretty large (probably was pulled out of the collector) we could simply add more options as separate config map keys as people request them.

[dprotaso]

Was thinking about this for #3256. But honestly I don't think this should gate that change

cc @evankanderson @Cali0707 for any input.

[dprotaso]

Also all the knobs can be manipulated with env vars

https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/
eg. https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp#pkg-overview

[dprotaso]

One thing I noticed is that auth for certain backends use an API key. To configure OTel clients we would use WithHeaders option to set an Authorization header.

This API key is sensitive so it shouldn't be in the config map but a secret

[dprotaso]

This API key is sensitive so it shouldn't be in the config map but a secret

Talking to @Cali0707 - I think a pragmatic approach here is to have metrics be sent to a OTel collector in the cluster. Then the collector has all the configuration that's specific to the desired backend.

Some advantages to this is it allows there to be multiple 'processing' pipelines so metrics can be sent to one place and then multiplex to different destinations.

Thus to prioritize which knobs to support I think we should focus on what's people need to send metrics to a collector. For example we can default to emitting cumulative metrics and the collector can change the temporality to 'delta'. Thus we don't need a temporality knob. (ref: #3256)

[Cali0707]

Thus to prioritize which knobs to support I think we should focus on what's people need to send metrics to a collector.

@dprotaso do we have anywhere we are tracking all the knobs + which ones we want to expose? Maybe in the otel gdoc?

[dprotaso]

@Cali0707

do we have anywhere we are tracking all the knobs

No - but there are a ton of knobs. eg. otelhttp knobs are here: https://pkg.go.dev/go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp

We could just recommend folks use env vars as I mentioned here #3257 (comment)

But then we need a way to set env vars on components we deploy - eg. Knative Revisions

[evankanderson]

But then we need a way to set env vars on components we deploy - eg. Knative Revisions

This would specifically be the queue-proxy, right (since they can already set environment variables on the user containers)?

[dprotaso]

This would specifically be the queue-proxy, right (since they can already set environment variables on the user containers)?

Yup.

I believe eventing has scenarios where some infrastructure is dynamic based on the CRDs applied.

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.