Help: log_to_metric count does not match Loki log count

[bragesande94]

Hi all,
I’m trying to generate metrics from logs using Vector’s log_to_metric transform so I can query counts much faster in Grafana.

However, the metric counts produced by log_to_metric do not match (its close but not close enough) the counts I see when querying logs directly in Loki (using count_over_time()).

[transforms.log_count_metrics]
type = "log_to_metric"
inputs = ["process_and_flatten_logs"]

#Counter metric definition
[[transforms.log_count_metrics.metrics]]
type = "counter"
name = "log_events_total"
namespace = "vector_custom"
field = "kubernetes.namespace_name"

#Labels used for Grafana grouping and filtering.
[transforms.log_count_metrics.metrics.tags]
kubernetes_namespace_name = "{{ kubernetes.namespace_name }}"
service_name = "{{ kubernetes.labels.app_kubernetes_io_name }}"
product = "{{ kubernetes.labels.app_kubernetes_io_part_of }}"
logtype = "{{ logtype }}"
level = "{{ level }}"

The number of log lines I get from Loki (e.g. using count_over_time({...}[$__range])) is higher than the metric counter produced by Vector.

I want the counter metric (log_events_total) to match the actual number of ingested logs as closely as possible. I would really apreciate if anyone got some ideas on how this can be done

[thomasqueirozb]

The mismatch is caused by log events being silently dropped by log_to_metric before a counter increment is emitted. Those events still reach Loki, which is why the counts diverge.

There are two drop paths in your configuration:

1. Missing field value

log_to_metric.rs L397–405:

let value = match log.get(&field) {
    None => Err(TransformError::PathNotFound { path: field.to_string() }),
    Some(Value::Null) => Err(TransformError::PathNull { path: field.to_string() }),
    Some(value) => Ok(value),
}?;

If kubernetes.namespace_name is absent or null on any event, the entire event is dropped — no counter increment is emitted. In a Kubernetes DaemonSet deployment, this commonly affects system-level logs, node logs, or events that arrive before metadata enrichment is complete.

Even with increment_by_value: false (the default), the field value is never actually used in the counter branch — the increment is always hardcoded to 1.0 (L427). Despite this, the field lookup still happens and still drops the event if the field is absent.

2. "All or none" semantics

log_to_metric.rs L944:

// early return to prevent the partial buffer from being sent
return;

If any of your configured metric entries fails (including the field lookup above), none of the metrics for that log event are emitted. You only have one metric defined, so this compounds the first issue rather than introducing a new one, but it is worth knowing for multi-metric configurations.

Note: Template failures in tag values (e.g. {{ kubernetes.labels.app_kubernetes_io_name }}) do not drop the event — they silently omit the tag and continue (L359–365). So your templated tags are not contributing to the mismatch.

Workaround

Add a remap transform upstream that provides default values for any field referenced in log_to_metric, so no events are dropped:

[transforms.ensure_fields]
type = "remap"
inputs = ["process_and_flatten_logs"]
source = '''
  .kubernetes.namespace_name = .kubernetes.namespace_name ?? "unknown"
'''

[transforms.log_count_metrics]
type = "log_to_metric"
inputs = ["ensure_fields"]
# ... rest of config unchanged

This guarantees the field lookup always succeeds, ensuring every event that flows to Loki also increments the counter. Let me know if this works!