GitHub Actions Auto-Scaling Runners: ARC Implementation and Workflow Integration. #2
Description
🚀 Project Kick-off: Auto-Scaling GitHub Actions Runners (ARC on K8s)
This document serves as the single source of truth for the ARC implementation project, covering the strategic plan, technical phases, and explaining how to use Actions Runner Controller (ARC) and Runner Scale Sets to create self-scaling GitHub Actions runners, and how to integrate them into your GitHub Actions workflow.
1. Detailed Project Purpose & How It Works:
The core objective is to establish a cost-efficient, high-performance, and secure self-hosted CI/CD runner platform by using the Actions Runner Controller (ARC) on a Kubernetes (K8s) cluster. This builds an auto-scaling and managed system for runners that execute GitHub Actions tasks. This is achieved by integrating the following key components:
| Component | Role in the Project | Key Benefit |
|---|---|---|
| GitHub Actions | The CI/CD system that hosts workflow definitions and manages the queue of jobs waiting to be run. It initiates the request for a runner. | Automation Experience: Provides a native, integrated platform for defining build, test, and deployment workflows directly alongside the code. |
| Actions Runner Controller (ARC) | The "brain" that monitors the GitHub Actions job queue. When a job starts, ARC automatically creates a K8s Pod (the runner); when the job is done, it deletes the Pod. | Auto-Scaling to Zero: Ensures compute resources are only consumed when a job is actively running, optimizing costs. |
| Kubernetes (K8s) | The underlying infrastructure that hosts the controller and runs the runners as temporary, isolated Pods. | High Control & Performance: Allows the use of custom-built runner images with all required dependencies pre-installed to accelerate build times. |
| Runner Scale Sets | A group of runners managed by ARC, identified by a unique name (e.g., arc-runner-set). |
Simple Targeting: Developers can easily specify these runners in their workflows using the runs-on: tag. |
2. Project Phases & Action Checklists:
Phase 1: Planning & Design (Strategy and Requirements)
This phase establishes all architectural decisions and defines key requirements.
| Checklist | Task ID | Task Description | Technical Discussion Point | Owner |
|---|---|---|---|---|
| P-1 | K8s Cluster Architecture Finalized: Define the roles of the 3 K8s clusters (Dev, Staging, Prod) and select the Infrastructure as Code (IaC) tool (e.g., Terraform). | Which Cluster Autoscaler tool will we use (e.g., Karpenter vs. native scaler)? | Infra | |
| P-2 | Custom Runner Image Specification: Finalize the required base OS and list all necessary pre-installed software (SDKs, CLIs, etc.). | What are the top 3 tools that must be included in the image to maximize build speed? | Dev/DevOps | |
| P-3 | GitHub App Authentication Setup: Create the official GitHub App and securely store the App ID and Private Key (e.g., in Azure Key Vault). | What is the secure storage mechanism for the Private Key before deployment? | Lead/Security | |
| P-4 | Security Blueprint (OIDC): Plan the use of OpenID Connect (OIDC) to eliminate long-lived cloud secrets inside the runners. | How will we configure K8s Service Accounts to assume Azure Roles via OIDC? | Security/DevOps |
Phase 2: Infrastructure & Setup (Building the Foundation)
This phase builds the foundational environment for ARC.
| Checklist | Task ID | Task Description | Owner |
|---|---|---|---|
| I-1 | Provision K8s Clusters: Deploy the 3 AKS Clusters and all required networking components using IaC. | Infra | |
| I-2 | Network & Firewall Rules: Verify outbound HTTPS access from the K8s cluster to the GitHub API endpoints. | Infra | |
| I-3 |
Deploy ARC Controller: Install the Actions Runner Controller using Helm into the dedicated arc-systems namespace. |
DevOps | |
| I-4 |
Configure K8s Secret: Create the Kubernetes Secret (e.g., gh-app-creds) containing the GitHub App credentials for ARC to use. |
DevOps |
Phase 3: Implementation & Testing (Deployment and Validation)
This phase validates the auto-scaling and performance of the ARC setup.
| Checklist | Task ID | Task Description | Owner |
|---|---|---|---|
| D-1 | Build & Publish Runner Image: Finalize the Dockerfile, build the custom runner image, and push it to the container registry (ACR). | DevOps | |
| D-2 |
Deploy Runner Scale Sets: Deploy the RunnerScaleSet Custom Resources for each environment (Dev, Staging, Prod). |
DevOps | |
| D-3 |
Functionality Test: Run a simple GitHub Actions workflow (YAML) that targets the new runner set (runs-on: <name>) to verify scale-up and job execution. |
Dev/QA | |
| D-4 |
Load & Scale-Down Test: Run multiple parallel jobs to stress the system to maxRunners and verify it scales back down to zero when idle. |
QA/DevOps | |
| D-5 | Monitoring Implementation: Configure alerts for ARC health, runner capacity, and job latency in Prometheus/Grafana. | DevOps |
3. Actions Runner Controller (ARC) and Runner Scale Sets Setup Process: Code Explanation
This section details the essential steps for setting up Actions Runner Controller (ARC) and Runner Scale Sets on Kubernetes, with explanations for each code block. The goal is to provide self-scaling GitHub Actions runners.
3.1. Prerequisites
Before starting the installation, the following prerequisites must be met:
- A working K8s cluster: You must have a Kubernetes cluster ready for operation GitHub Docs: Deploy runner scale sets..
- Outbound HTTPS connection: The Kubernetes cluster must be able to connect externally to download images and communicate with the GitHub API GitHub Docs: Deploy runner scale sets..
- GitHub Authentication (GitHub App/PAT): ARC needs a method to authenticate with GitHub to monitor task queues. Using a GitHub App is recommended for security reasons, or a Personal Access Token (PAT) can be used GitHub Docs: Deploy runner scale sets..
3.2. Install the Controller (ARC)
The ARC controller is installed in a dedicated Namespace within Kubernetes, typically arc-systems. This is done using Helm, a package manager for Kubernetes GitHub Docs: Deploy runner scale sets..
First, create the Kubernetes namespace for the controller:
# Create a dedicated namespace for the ARC controller components
kubectl create namespace arc-systemsExplanation: This kubectl command creates a new namespace called arc-systems. Namespaces help organize resources within a Kubernetes cluster and provide a scope for names.
Next, install the ARC controller using Helm:
# Install the Actions Runner Controller using its OCI Helm chart
helm install arc \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller \
-n arc-systemsExplanation: This helm install command deploys the ARC controller.
* helm install arc: Installs the chart and names the release arc.
* oci://...: Specifies the OCI (Open Container Initiative) registry path to the Helm chart for the gha-runner-scale-set-controller.
* -n arc-systems: Specifies that the controller should be installed into the arc-systems namespace.
3.3. Create GitHub Auth as a K8s Secret
GitHub authentication data (such as App ID, Installation ID, and the private_key.pem for a GitHub App) must be stored as a Kubernetes Secret. This secret allows ARC to securely communicate with the GitHub API GitHub Docs: Deploy runner scale sets..
First, create a dedicated namespace for the runners:
# Create a dedicated namespace for the GitHub Actions runners
kubectl create namespace arc-runnersExplanation: Similar to arc-systems, this creates a namespace arc-runners to logically separate the runner pods from the controller.
Then, create the Kubernetes Secret containing your GitHub App credentials:
# Create a generic secret named 'gh-app-creds' in the 'arc-runners' namespace
kubectl -n arc-runners create secret generic gh-app-creds \
--from-literal=github_app_id=123456 \
--from-literal=github_app_installation_id=654321 \
--from-file=github_app_private_key=private-key.pemExplanation: This kubectl create secret command creates a Kubernetes Secret.
* -n arc-runners: Specifies the namespace where the secret will reside.
* generic gh-app-creds: Creates a generic secret named gh-app-creds.
* --from-literal=github_app_id=123456: Adds the GitHub App ID as a literal value.
* --from-literal=github_app_installation_id=654321: Adds the GitHub App Installation ID as a literal value.
* --from-file=github_app_private_key=private-key.pem: Reads the private key from a file named private-key.pem and includes it in the secret.
Note: The values 123456, 654321, and private-key.pem must be replaced with the actual values for your GitHub App. Ensure private-key.pem is present in the directory where you run this command.
3.4. Install a Runner Scale Set
After installing the controller and creating the authentication secret, the actual Runner Scale Set is installed. This set will create runners as needed. The installation is also done using Helm, with specific values defined in a values.yaml file GitHub Docs: Deploy runner scale sets..
Example values.yaml file for configuring the Runner Scale Set:
githubConfigUrl: "https://github.com/<your-org>" # or https://github.com/<org>/<repo>
githubConfigSecret: gh-app-creds
runnerScaleSetName: "arc-runner-set"
minRunners: 0
maxRunners: 10
runnerGroup: "default"Explanation: This YAML file defines the configuration for your Runner Scale Set.
* githubConfigUrl: Specifies the GitHub organization or repository URL that this runner scale set will serve.
* githubConfigSecret: References the Kubernetes Secret (gh-app-creds) created in the previous step, which holds the GitHub App credentials.
* runnerScaleSetName: A unique name for this specific runner scale set, used by GitHub Actions workflows.
* minRunners: The minimum number of idle runners to maintain. Setting to 0 means runners will scale down completely when not in use.
* maxRunners: The maximum number of runners that ARC can create for this scale set.
* runnerGroup: (Optional) Specifies an existing runner group in GitHub to associate with this scale set.
Then, install the Runner Scale Set using Helm and the values.yaml file:
helm install arc-runnerset \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
-n arc-runners \
-f values.yamlExplanation: This helm install command deploys the Runner Scale Set.
* helm install arc-runnerset: Installs the chart and names the release arc-runnerset.
* oci://...: Specifies the OCI registry path to the Helm chart for the gha-runner-scale-set.
* -n arc-runners: Specifies that the runner scale set should be installed into the arc-runners namespace.
* -f values.yaml: Uses the values.yaml file to override default chart values with your specific configuration.
4. How to Use ARC Runners in GitHub Actions Workflow: Code Explanation
After successfully setting up Actions Runner Controller (ARC) and Runner Scale Sets in your Kubernetes environment, you can now integrate these self-scaling runners into your GitHub Actions Workflow to execute your tasks. This step is crucial to leverage the auto-scaling and flexibility capabilities provided by ARC GitHub Docs: Deploy runner scale sets., GitHub Docs: Quickstart for Actions Runner Controller..
4.1. Specify the Runner Scale Set Name
During the Runner Scale Set setup process (Section 3.4), you defined a name for your runner group (e.g., arc-runner-set in the values.yaml file). This name is the identifier you will use in your GitHub Actions workflow files GitHub Docs: Deploy runner scale sets..
4.2. Modify the GitHub Actions Workflow File (Workflow YAML)
To make a specific job in your GitHub Actions workflow use ARC runners, you must modify the workflow's YAML file and add the line runs-on: <runner-scale-set-name> within the job definition. This tells GitHub Actions that this job should be executed on runners from the arc-runner-set group instead of GitHub-hosted runners GitHub Docs: Use ARC in a workflow..
Example GitHub Actions workflow file demonstrating runs-on usage:
name: Build and Deploy
on:
push:
branches:
- main
jobs:
build:
# Specify the name of the runner group to be used
runs-on: arc-runner-set
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Say hello from ARC runner
run: echo "Hello from the ARC runner!"
- name: Run tests
run: |
echo "Running tests..."
# You can add your test commands here
deploy:
needs: build
runs-on: arc-runner-set
steps:
- name: Deploy application
run: echo "Deploying application..."Explanation: This YAML snippet defines a typical GitHub Actions workflow.
* name: Build and Deploy: The name of your workflow.
* on: push: branches: - main: Triggers the workflow on pushes to the main branch.
* jobs: build:: Defines a job named build.
* runs-on: arc-runner-set: This is the critical line. It instructs GitHub Actions to use a runner from the arc-runner-set scale set (which is managed by ARC on Kubernetes) to execute this job.
* steps:: Defines the sequence of actions for the job.
* uses: actions/checkout@v4: A common action to check out your repository code.
* run: echo "Hello from the ARC runner!": A simple shell command executed on the ARC runner.
* deploy:: Defines another job named deploy that depends on the build job.
* runs-on: arc-runner-set: Similarly, this job also uses a runner from the arc-runner-set.
In this example:
- The
buildanddeployjobs will use runners managed by ARC from thearc-runner-setgroup. - When this workflow runs, ARC will automatically create Pods in Kubernetes to execute these jobs, and then delete them upon completion, ensuring efficient resource utilization GitHub Docs: Deploy runner scale sets...
4.3. Notes and Considerations
- Unique Runner Group Names: Runner group names must be unique for each runner group. You cannot add additional labels to
runs-onother than the group name GitHub Docs: Deploy runner scale sets.. - No Ingress/Nginx Required: ARC does not require Ingress or Nginx. Runners connect outbound to GitHub, and nothing needs to be exposed GitHub Docs: Deploy runner scale sets..
- Separate Namespaces: It is recommended to separate the controller (ARC) and runners in separate namespaces (e.g.,
arc-systemsandarc-runners) as a security best practice GitHub Docs: Deploy runner scale sets.. - Deployment Scope: You can deploy runner groups at the organization, repository, or enterprise level by changing
githubConfigUrlin thevalues.yamlfile GitHub Docs: Deploy runner scale sets..
By following these steps, you can successfully integrate self-scaling ARC runners into your GitHub Actions workflow, providing you with greater flexibility and efficiency in your CI/CD operations.