Metrics aggregation for workflows

[Liam-hunter]

Hey 👋

I have self-hosted GitHub actions runners, deployed the github_exporter am trying to collect metrics on two specific points of interest.

1. What is the average queue time of jobs in each runner group. I can get that from this exporter with the following expression: avg by(runner_group_name) (github_workflow_job_started_timestamp-github_workflow_job_created_timestamp{runner_group_name!=""})
2. Which repo's are running the most workflow jobs. I can get that from this exporter with the following expression: topk(10, count by(repo,owner) (github_workflow_job_created_timestamp{runner_group_name="default"}))

However, we are running many workflows/jobs and the cardinality of this data is too much for our prometheus instance. I have tried excluding labels to reduce cardinality however, as the data is reported as individual timestamps I am unable to reduce cardinality enough for prometheus scraping.

I was wondering if it could be possible to do more of this aggregation on the exporter itself?

[Liam-hunter]

The other issue, aside from ingesting too many metrics into prometheus is that when the exporter has been running for a sufficient length of time the number of samples being scraped (pictured)

grows to the point that scraping times out at the default value of 10 seconds (pictured)

Of all the metrics from the github_exporter, almost all are the high-cardinality "workflow" metrics

This is happening after the github_exporter has been running for 8 days.

[tboerger]

You can configure the labels, that could potentially reduce the cardinality.

[Liam-hunter]

Thanks for getting back to me @tboerger, I have tried using the labels to reduce cardinality. I believe the two labels at the root of the cardinality for workflow metrics are the 'run_id' and 'identifier' as those seem to be unique for each workflow job. So, when I remove those like so:

I end up with the following output from the exporter:

But this makes me wonder. If I am removing labels to reduce cardinality of timestamps, how are they being aggregated. As in if the exporter aggregates two timestamps from two workflow jobs what is the output? The average of the two timestamps? The sum of the two timestamps?

Thanks!