Prometheus remote writer

[p4k8]

Is your feature request related to a problem? Please describe.

Problem is storing perfdata in a Prometheus-compatible storage (Victoriametrics, Cortex, Thanos, Mimir etc) for it to be compatible with other monitoring tools while being cost-efficient.

Describe the solution you'd like

Native Prometheus remote protocol writer, similar to InfluxDB and OpenTSDB perfdata writers.

Describe alternatives you've considered

As of now I employ workaround consisting of writing with InfluxDB writer to the vmagent tool of Victoriametrics suite. It works fine most of the time, but data model is different and it might name and store things in somewhat awkward way due to conversion.

Additional context

https://prometheus.io/docs/concepts/remote_write_spec/
https://github.com/google/snappy/tree/main
https://github.com/prometheus/compliance/tree/main/remote_write_sender

[martialblog]

Just some thoughts on this: Instead of a Writer for Prometheus alone it might be even more beneficial to use OpenTelemetry https://opentelemetry.io/

This would allow for sending data to even more metric backends, including Prometheus.

Prometheus will move also move to OTel in the longterm: https://prometheus.io/blog/2024/03/14/commitment-to-opentelemetry/

Edit: Elastic and Influx are on their way to fully support OTel as well. Another argument for an OTel Writer.

Together with tools like the OpenTelemetry Collector or Telegraf this would enable Icinga2 to be extremely flexible for writing Perfdata.

[rmeichsner]

This would be a very nice feature

[oxzi]

For this week's blog post, I wrote a bit about Icinga 2 Event Streams and have built a Prometheus remote writer as a proof of concept. I also took a closer look at OTEL as the idea appealed to me, but then decided against it.

• Blog Post: Icinga 2 Insights With Event Streams
• Proof Of Concept Icinga Prometheus Remote Writer Implementation

[hagen1778]

As of now I employ workaround consisting of writing with InfluxDB writer to the vmagent tool of Victoriametrics suite. It works fine most of the time, but data model is different and it might name and store things in somewhat awkward way due to conversion.

According to VictoriaMetrics data model (and Prometheus too), metrics samples could contain only numeric values. In influxdbcommonwriter some non-numeric values (like unit) are set as fields during transmission and can't be properly parsed by VictoriaMetrics for the obvious reasons.

Ideally, for influx-line-protoctol => VictoriaMetrics connection to work, all non-numeric values should be attached as tags (labels in Prom/VM terminology). Maybe it would be easier to implement OpentelemetryWriter instead, as it naturally imposes this limitation for metrics signal.

Implementing PrometheusWriter now would require supporting Prometheus Remote Write protocol v1 and v2.

[martialblog]

There's now a branch with a OTLP HTTP Writer here #10685

I've been testing it with Prometheus.

Yonas is looking for people to test it and to give feedback.