OpenTelemetry Writer

[tbauriedel]

I would like to propose the implementation of an OpenTelemetry writer for performance data output in Icinga 2.

Currently the writers are limited to tools like Influx, Graphite, etc. This feature would allow metrics generated by Icinga 2 to be exported natively in OpenTelemetry format, enabling seamless integration with modern observability stacks such as Prometheus, Grafana, Jaeger, or any OpenTelemetry-compliant backend.

Currently, there is no native support in Icinga 2 for exporting metrics via OpenTelemetry. As OpenTelemetry becomes the de-facto standard for such data, having direct support in Icinga 2 would be a significantly improvement.

[martialblog]

Note, this was also mentioned in #9900 (comment)

[mdetrano]

I think this is a good idea, and would add that events could also be sent as Open Telemetry Logs. So your state changes or notification history could also be sent to be ingested by Otel systems.

[yolanjoy]

I see this is labelled as "Feature". Is this something being considered and will it be available at some point in a future release? Thanks

[oxzi]

@yolanjoy: We have talked about this internally just yesterday and it is on our mind. At the moment, we cannot promise anything, neither the "when" nor the "if". However, we are definitely considering adding OTel, both for time series as a remote writer, but also in a broader sense.

The "Feature" type itself is just to distinguish between Bugs and Features. Otherwise, it has no meaning.

[martialblog]

Regarding OTel/OTLP I just want to mention these discussions over in the OpenTelemetry semantic-conventions repo that are related to this issue here.

open-telemetry/semantic-conventions#1106

open-telemetry/semantic-conventions#3148

[martialblog]

There's now a branch with a OTLP HTTP Writer here #10685

Yonas is looking for people to test it and to give feedback.

[mdetrano]

I've started an independent project as an alternative for connecting Icinga to OpenTelemery, and if anyone would like to have a look it is available here:

(https://github.com/Intelicolab/icinga2otel)

This method just sets up a separate service that, in short, connects to the (incredibly useful) Icinga EventStream API on one end, and the OpenTelemetry API on the other. There is support for metrics and logs collection, some filtering by event and object, security and performance options, etc.

I'd consider the project at this point to be in the "alpha" stage (by which I mean it works, but...dude). I'd certainly be open to suggestions for improvements or better preparation of the data for inclusion in an Otel-based observability environment. This may fill in as a way to get started with a monitoring/observability connection while the Icinga OTLP Writer module is developed.

[LilWatson]

Thanks @yhabteab for the great work at #10685! Looking forward to use it :)

May i ask (since the milestone was removed) when its expected to be released?
Sry, just saw the 2.16.0 milestone reference.