[Warning] High Utilisation of GitHub Runners

[lupyuen]

Description / Steps to reproduce the issue

Our usage of GitHub Actions Runners has hit 70 Full-Time Runners over the past 12 hours. Remember that ASF Infra Team has demanded that we keep this Under 25 Full-Time Runners, or we might lose access to GitHub Actions CI.

https://infra-reports.apache.org/#ghactions&hours=12&limit=15&group=name

I'm not sure whether this is due to high volume of PRs in NuttX Repo or NuttX Apps. Or possibly some runaway jobs? Let's continue to monitor this, hope it will drop within the next few days.

The URL above is accessible to NuttX PMC Members (possibly to NuttX Maintainers too). Other folks may use this simpler metric for monitoring: https://lupyuen.github.io/nuttx-metrics/github-fulltime-runners.png

I'm travelling overseas for a Marathon Race, my response will be slow. Thanks!

FYI: This is how we redesigned NuttX CI, 13 months ago: https://lupyuen.org/articles/ci3

---

Update: I have new videos that explain GitHub Runners and why we're using so many of them, also potential solutions...

• [Testers Needed] Building PRs in Our Own Repos #18568
• [Test] Workflow to testing PRs in their own repository on GitHub NuttX/manual-nuttx-ci#1
• [FEATURE] Retry for CI Builds #18562
• CI: Retry build upon failure #18576
• [FEATURE] NuttX Build Monitor #18659
• Apache NuttX RTOS: GitHub Runners and Mystery Spikes
• Apache NuttX RTOS: Too Many GitHub Runners
• Apache NuttX RTOS: Dumping the GitHub Jobs
• Apache NuttX RTOS: Most Expensive PR
• Apache NuttX RTOS: Cost of NuttX PRs vs NuttX Apps
• Apache NuttX RTOS: Analysing the Most Expensive PR
• Apache NuttX RTOS: Retrying a CI Build
• Apache NuttX RTOS: Building PRs in Our Own Repo
• Apache NuttX RTOS: Testing of Retry for CI Builds
• Apache NuttX RTOS: NTFC Download Failure + Second Most Expensive PR
• Apache NuttX RTOS: PR Monitoring + Third Most Expensive PR
• Apache NuttX RTOS: Building the PR Monitor + Most Expensive Apps PR
• Apache NuttX RTOS: Parsing the GitHub Build Logs
• Apache NuttX RTOS: Build Monitor
• Apache NuttX RTOS: More Testing of Retry for CI Builds

GitHub Runners Usage up to 8 Mar 2026:

https://docs.google.com/spreadsheets/d/1JH7BCwTB-U6dwvSsy0tfpNXr534jVX-XYkOYi3FLM8g/edit?gid=0#gid=0

(Generated by history.sh)

Based on the data from above, we have identified the following PRs as the Most Expensive PRs, in terms of GitHub Runner Usage. We need to study further why these PRs are consuming so many resources, and whether improving our NuttX Dev Process will fix this problem...

• sched/hrtimer: introduce scheduler support with hrtimer #17573
• sched/spinlock: Refactor critical section implementation with rspinlock foundation and SMP optimization #18135
• [EXPERIMENTAL]sched/hrtimer: hrtimer state machine improvement for SMP cases #17642
• !boards: Simplify NuttX initialization #18408
• sched: add a reference count to the TCB to prevent it from being deleted. #17468
• sched/sched: Part 2: Add high-resolution timer (hrtimer) and os tick support with hrtimer to NuttX #17489
• netutils/netinit: Support the NETINIT_MACADDR. nuttx-apps#3315
• NuttX GitHub Jobs Spreadsheet
• Most Expensive PRs

Which were generated by...

git clone https://github.com/lupyuen/nuttx-github-jobs
cd nuttx-github-jobs
./dump-github-jobs.sh
cargo run

## Import nuttx-github-jobs.tsv into Google Sheets for analysis:
## NuttX GitHub Jobs Spreadsheet: https://docs.google.com/spreadsheets/d/1HY7fIZzd_fs3QPyA0TX7vsYOjL86m1fNOf1Wls93luI/edit?gid=1187577522#gid=1187577522
## Most Expensive PRs: https://docs.google.com/spreadsheets/d/1HY7fIZzd_fs3QPyA0TX7vsYOjL86m1fNOf1Wls93luI/edit?gid=70515654#gid=70515654

---

Over the Past 12 Months: NuttX Repo consumed 84% of GitHub Runners, while NuttX Apps Repo consumed 12%...

What about the past 4 months? The distribution is the same: 84% vs 12%. Interesting note: Within the past 4 months, we consumed Half of The Entire Year's GitHub Runners.

Conclusion: It's definitely worth the effort to do Dependency Analysis and speed up the CI Builds for NuttX Repo. I thought NuttX Apps might take more GitHub Runners (because every CI Build takes 3.5 hours), but I proved myself wrong today.

---

On which OS does this issue occur?

[OS: Linux]

What is the version of your OS?

Ubuntu on GitHub Actions

NuttX Version

master

Issue Architecture

[Arch: all]

Issue Area

[Area: Other]

Host information

No response

Verification

• I have verified before submitting the report.

[simbit18]

Thank you so much @lupyuen !

@acassis @cederom @linguini1 @raiden00pl @xiaoxiang781216 @Donny9 @fdcavalcanti @tmedicci @wangchdo @xuxin930 @extinguish @CV-Bowen @zhhyu7 @GUIDINGLI @hujun260 @no1wudi @anchao

I believe it is necessary to slow down/block PR for a few days, otherwise we may lose access to GitHub Actions CI..

#14376

[acassis]

@lupyuen @SiM, is there some way we could create distributed CI buildings to relieve (alleviate) the runners pressure?

[raiden00pl]

Is it possible to temporarily disable PRs in GH? I'm afraid that not everyone follows the GH notifications and mailing list, especially since in recent weeks these has been flooded with changes from Xiaomi

[simbit18]

Hi @xiaoxiang781216, Could you kindly inform the Xiaomi developers of the current situation so that they temporarily suspend the addition of new PRs?

[linguini1]

We're going to need to come up with a more permanent solution again soon.

[acassis]

@simbit18 yes, maybe they can focus on fix the existing PRs that are in development and coordenate between them to only do git push after previos PR finish the CI

[simbit18]

Hi @acassis, the problem is that we have exceeded 25 full-time runners and are now at 54, so we should stop.

[acassis]

@linguini1 seems like we are in a stage similar to what Greg was facing when he are doing "CI" in machine. We thought that after moving to Apache Foundation all our problems will be fixed, this is very disappoint.

I think they should allows to use more than 25 runners, because this is the amount allowed to all projects, even projects that are not active anymore. It should be better to redistribute it between active projects (maybe the situation is a little different than what I'm seeing, but who knows?)

Maybe we will need to run local machines to do the CI and just send the result to github, but since our user base is small I don't know if we will have enough computing power this way

[lupyuen]

Maybe this will help: I see our devs pushing updated commits, which re-runs the build, and wastes GitHub Runners. (Pic below)

Could we ask our devs to wait for the PRs to finish the builds in their own NuttX Repos, before submitting the PR? Remember that the builds in their own NuttX Repos, won't count towards our quota for GitHub Runners. Thanks!

[linguini1]

@linguini1 seems like we are in a stage similar to what Greg was facing where he are doing "CI" in machine. We thought that after moving to Apache Foundation all our problems will be fixed, this is very disappoint.

I think they should allows to use more than 25 runners, because this is the amount allowed to all projects, even projects that are not active anymore. It should be better to redistribute it between active projects (maybe the situation is a little different than what I'm seeing, but who knows?)

Maybe we will need to run local machines to do the CI and just send the result to github, but since our user base is small I don't know if we will have enough computing power this way

I agree it would be nice if resources were redistributed, but I think we can still get a little more clever with our CI. I remember Lup made some really big improvements before. For instance, now we don't run a full build on every documentation change, just a docs rebuild.

I wonder if we can try to do something similar again. Ex: changes to documents under arm/rp2040 only perform builds of rp2040 boards. It's going to be hard to do that for everything but it might be a better option than self hosting?

Also, I wonder if we can cut down on "duplicate" boards in the regular CI. As in, it's not necessary to test a rebuild of every SMT32H7 board when there's a change, just one should be fine in most cases.

[acassis]

@lupyuen is there some way to force Github to post pone (ignore momentarily) the new PR build? I think it will difficult to user wait the right moment to push their PRs.

[acassis]

@linguini1 seems like we are in a stage similar to what Greg was facing where he are doing "CI" in machine. We thought that after moving to Apache Foundation all our problems will be fixed, this is very disappoint.
I think they should allows to use more than 25 runners, because this is the amount allowed to all projects, even projects that are not active anymore. It should be better to redistribute it between active projects (maybe the situation is a little different than what I'm seeing, but who knows?)
Maybe we will need to run local machines to do the CI and just send the result to github, but since our user base is small I don't know if we will have enough computing power this way

I agree it would be nice if resources were redistributed, but I think we can still get a little more clever with our CI. I remember Lup made some really big improvements before. For instance, now we don't run a full build on every documentation change, just a docs rebuild.

I wonder if we can try to do something similar again. Ex: changes to documents under arm/rp2040 only perform builds of rp2040 boards. It's going to be hard to do that for everything but it might be a better option than self hosting?

Also, I wonder if we can cut down on "duplicate" boards in the regular CI. As in, it's not necessary to test a rebuild of every SMT32H7 board when there's a change, just one should be fine in most cases.

Yes, we discussed this idea with @lupyuen in the past: if a PR is only modifying an specific board it doesn't make sense to test all other boards of the same arch. And if only a chip from a specific arch is modified it doesn't make sense to test all ARM boards.

But I think these rules are difficult to implement on our CI. Maybe we need to rethink our CI from scratch