Skip to content

README.md

Latest commit

 

History

History
375 lines (274 loc) · 23.5 KB

README.md

File metadata and controls

375 lines (274 loc) · 23.5 KB

Google Cloud Set-Up

This directory contains the Terraform Infrastructure as Code (IaC) configuration and instructions to automatically create the Google Cloud Platform (GCP) infrastructure for the GitHub Actions Runners.

Project

A separate Google Cloud project is recommended for the GitHub Actions Runners.

Create a Google Cloud project with a attached billing account.

IAM Roles

The Owner role (roles/owner) is the easiest option for deploying this project. If the Owner role is not possible (e.g., in enterprise environments with restricted permissions), the following specific roles must be assigned to your Google account on project level:

Role ID Role Name Purpose
roles/artifactregistry.admin Artifact Registry Administrator Manage container images in Artifact Registry
roles/cloudbuild.builds.editor Cloud Build Editor Trigger and manage Cloud Build jobs
roles/compute.admin Compute Admin Manage Compute Engine resources (VMs, templates, images)
roles/iam.roleViewer Role Viewer Provides read access to all custom roles in the project.
roles/iam.serviceAccountAdmin Service Account Admin Create and manage service accounts.
roles/iam.serviceAccountUser Service Account User Run operations as the service account.
roles/logging.admin Logging Admin Access to all logging permissions, and dependent permissions.
roles/monitoring.admin Monitoring Admin All monitoring permissions.
roles/orgpolicy.policyViewer Organization Policy Viewer View organization policies
roles/resourcemanager.projectIamAdmin Project IAM Admin Access and administer a project IAM policies.
roles/run.admin Cloud Run Admin Deploy and manage Cloud Run services
roles/secretmanager.admin Secret Manager Admin Create and manage secrets
roles/serviceusage.serviceUsageAdmin Service Usage Admin Enable and disable Google Cloud APIs
roles/storage.admin Storage Admin Manage Cloud Storage buckets

Tip: Use the assign-iam-roles.sh script in the /tools folder to automatically assign these roles to a user:

./tools/assign-iam-roles.sh PROJECT_ID user@example.com

1. Login

Authenticate with Google Cloud and set the quota project:

gcloud auth login --no-launch-browser
gcloud auth application-default login --no-launch-browser

Set the quota project and project where the resources will be created:

gcloud projects list
export GOOGLE_CLOUD_PROJECT="your-project-id"
gcloud config set project "$GOOGLE_CLOUD_PROJECT"
gcloud auth application-default set-quota-project "$GOOGLE_CLOUD_PROJECT"

2. Organization Policy

Make sure that the organization policy "Allowed ingress settings (Cloud Run)" run.allowedIngress is configured and set to all.

run.allowedIngress
spec:
  rules:
  - values:
      allowedValues:
      - all

That is the default setting. However, in some Google Cloud organizations, mainly those of large companies, these values may have been changed.

gcloud services enable "orgpolicy.googleapis.com" --project="$GOOGLE_CLOUD_PROJECT"
gcloud org-policies describe "run.allowedIngress" --effective --project="$GOOGLE_CLOUD_PROJECT"

If you are using the Fabric FAST Project Factory, add the following to your your-project-id.yaml:

org_policies:
  run.allowedIngress:
    rules:
      - allow:
          values:
            - all

See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information.

Architecture

graph TD
    %% Service Accounts
    subgraph IAM_SA[IAM Service Accounts]
        sa_manager(🆔 sa-github-runners-manager)
        sa_vm(🆔 sa-github-runners)
        sa_build(🆔 sa-cloud-build-github-runners)
    end

    %% Cloud Build
    subgraph Cloud_Build[Cloud Build]
        cb_job(🏗️ Container Build Job)
    end

    %% Cloud Run
    subgraph Cloud_Run[Cloud Run]
        cr_service(🚀 cloud-run-github-runners-manager)
    end

    %% Compute Engine
    subgraph Compute_Engine[Compute Engine]
        vm_templates(🖥️ github-runners-vm-templates)

        subgraph Custom_Images[Custom Images]
            img_amd64(💿 ubuntu-2404-lts-amd64)
            img_arm64(💿 ubuntu-2404-lts-arm64)
        end
    end

    %% Secrets
    subgraph Secrets_Manager[Secret Manager]
        secret_mgr(🗝️ secret-manager)
        secret_ver(🗝️ secret-version-default)
    end

    %% Artifact Registry
    subgraph AR[Artifact Registry]
        ar_repo(📦 container-github-runners-manager)
    end

    %% Storage
    subgraph GCS[Cloud Storage]
        gcs_iac(🪣 gh-iac-bucket)
        gcs_build(🪣 build-bucket)
        gcs_startup(🪣 gh-start-bucket)
    end

    %% Network
    subgraph Network[VPC Network]
        vpc(☁️ vpc-github-runners)
        subnet(🕸️ subnet-github-runners)
        firewall(🛡️ firewall-github-runners)
        nat(🌍 nat-github-runners)
    end

    %% Logic / IAM Flows
    cr_service -->|Run As| sa_manager
    vm_templates -->|Run As| sa_vm
    cb_job -->|Run As| sa_build

    sa_manager -->|roles/secretmanager.admin| secret_mgr
    sa_manager -->|roles/compute.admin| vm_templates
    sa_manager -->|roles/iam.serviceAccountUser| sa_vm

    sa_vm -->|roles/storage.objectViewer| gcs_startup

    sa_build -->|roles/artifactregistry.writer| ar_repo
    sa_build -->|roles/storage.objectAdmin| gcs_build

    %% Dependencies/Relations
    cr_service -->|Image| ar_repo
    
    cb_job -->|Reads Source| gcs_build
    cb_job -->|Pushes| ar_repo

    vm_templates -->|Image| img_amd64
    vm_templates -->|Image| img_arm64

    vm_templates -->|Network| subnet
    subnet -->|Part of| vpc
    firewall -->|Applies to| vpc
    nat -->|Applies to| vpc
    
    secret_ver -->|Version of| secret_mgr

    %% Styling
    class sa_manager,sa_vm,sa_build sa;
Loading

Setup via Terraform

The majority of the required services and resources are configured via Terraform Infrastructure as Code (IaC).

1. Configure

Navigate to the gcp directory (the directory of this README) and create the variables file:

cd gcp

Create a terraform.tfvars file with your configuration.

Google Cloud project ID:

printf 'project_id = "%s"\n' "$GOOGLE_CLOUD_PROJECT" > terraform.tfvars

(Optional) Google Cloud region:

echo "region = \"us-central1\"" >> terraform.tfvars

(Optional) Google Cloud zone:

echo "zone = \"b\"" >> terraform.tfvars

For all available variables, see variables.tf.

2. Apply

Initialize Terraform:

terraform init

Apply the configuration:

terraform apply
  • Review the plan when prompted.
  • Type yes and press Enter to confirm.

(Optional) Migrate Terraform State to GCS

By default, Terraform stores state locally. For production environments or team usage, it's highly recommended to migrate this state to a remote Google Cloud Storage (GCS) backend.

Steps:

  1. Apply Terraform locally first.
  2. Copy providers.tf.gcs to providers.tf (configured for GCS backend).
  3. Run terraform init -migrate-state to copy your local state to the bucket.

Rebuild Container Image

To rebuild the container image, run:

./build-container.sh

Apply the new container image to the Cloud Run service:

terraform apply

The used Dockerfile is located in the upper directory.

Rebuild Google Compute Engine Custom Images

To rebuild the Google Compute Engine custom images, run:

./build-image-[IMAGE_NAME].sh

Example:

./build-image-ubuntu-2404-lts-amd64.sh

The script will create a custom image based on the specified image name. During startup, the script startup/install.sh will be executed.

Default Machine Sizes for Runners

The following table shows the default machine sizes for runners. You can customize the machine sizes in the terraform.tfvars file via the variable github_runners_types.

Note: In the label name (runs-on:), . is automatically replaced with -. This means runs-on: gcp-ubuntu-24.04 in your GitHub Actions workflow YAML file works and uses the gcp-ubuntu-24-04 template.

AMD64

Name Instance Type vCPU Memory (GB) Disk Size (GB) Image Name
gcp-ubuntu-slim e2-medium 2 4 15 ubuntu-2404-lts-amd64
gcp-ubuntu-latest e2-standard-4 4 16 25 ubuntu-2404-lts-amd64
gcp-ubuntu-24-04 e2-standard-4 4 16 25 ubuntu-2404-lts-amd64
gcp-ubuntu-24-04-2core e2-standard-2 2 8 75 ubuntu-2404-lts-amd64
gcp-ubuntu-24-04-4core e2-standard-4 4 16 150 ubuntu-2404-lts-amd64
gcp-ubuntu-24-04-8core e2-standard-8 8 32 300 ubuntu-2404-lts-amd64
gcp-ubuntu-24-04-16core e2-standard-16 16 64 600 ubuntu-2404-lts-amd64
gcp-ubuntu-24-04-32core e2-standard-32 32 128 1200 ubuntu-2404-lts-amd64
gcp-ubuntu-24-04-64core e2-standard-64 64 256 2040 ubuntu-2404-lts-amd64
gcp-ubuntu-24-04-96core n2d-standard-96 96 384 2040 ubuntu-2404-lts-amd64
gcp-ubuntu-24-04-128core n2d-standard-128 128 512 2040 ubuntu-2404-lts-amd64

ARM64

Name Instance Type vCPU Memory (GB) Disk Size (GB) Image Name
gcp-ubuntu-slim-arm c4a-standard-1 1 4 15 ubuntu-2404-lts-arm64
gcp-ubuntu-latest-arm c4a-standard-4 4 16 25 ubuntu-2404-lts-arm64
gcp-ubuntu-24-04-arm c4a-standard-4 4 16 25 ubuntu-2404-lts-arm64
gcp-ubuntu-24-04-2core-arm c4a-standard-2 2 8 75 ubuntu-2404-lts-arm64
gcp-ubuntu-24-04-4core-arm c4a-standard-4 4 16 150 ubuntu-2404-lts-arm64
gcp-ubuntu-24-04-8core-arm c4a-standard-8 8 32 300 ubuntu-2404-lts-arm64
gcp-ubuntu-24-04-16core-arm c4a-standard-16 16 64 600 ubuntu-2404-lts-arm64
gcp-ubuntu-24-04-32core-arm c4a-standard-32 32 128 1200 ubuntu-2404-lts-arm64
gcp-ubuntu-24-04-64core-arm c4a-standard-64 64 256 2040 ubuntu-2404-lts-arm64
gcp-ubuntu-24-04-72core-arm c4a-standard-72 72 288 2040 ubuntu-2404-lts-arm64

Cleanup / Destroy

To destroy all resources created by Terraform, run:

terraform destroy

In addition, you must manuelly delete the GitHub App and its installation.

(Development & Debugging) CLI Script

A CLI script gce.py is available in the tools directory to manually create and delete runner instances for testing purposes.

Usage

Ensure you have the virtual environment activated and the GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_ZONE environment variables set.

export GOOGLE_CLOUD_PROJECT=your-project-id
export GOOGLE_CLOUD_ZONE=your-region-with-zone-suffix

Create Instance

python tools/gce.py create --token [TOKEN] --url [URL] --template [TEMPLATE_NAME]

Delete Instance

python tools/gce.py delete --instance [INSTANCE_NAME]

Providers

Name Version
google 7.19.0
local 2.6.2
null 3.2.4
time 0.13.1

Inputs

Name Description Type Default Required
apis List of Google Cloud APIs to be enable list(string) 
[
  "artifactregistry.googleapis.com",
  "cloudbuild.googleapis.com",
  "cloudresourcemanager.googleapis.com",
  "compute.googleapis.com",
  "iam.googleapis.com",
  "logging.googleapis.com",
  "orgpolicy.googleapis.com",
  "run.googleapis.com",
  "secretmanager.googleapis.com",
  "storage.googleapis.com"
]
 no
github_runners_default_image Default GitHub Actions Runners images (family images) for different CPU architectures map(string) 
{
  "ubuntu-2204-lts-amd64": "projects/ubuntu-os-cloud/global/images/family/ubuntu-2204-lts",
  "ubuntu-2204-lts-arm64": "projects/ubuntu-os-cloud/global/images/family/ubuntu-2204-lts-arm64",
  "ubuntu-2404-lts-amd64": "projects/ubuntu-os-cloud/global/images/family/ubuntu-2404-lts-amd64",
  "ubuntu-2404-lts-arm64": "projects/ubuntu-os-cloud/global/images/family/ubuntu-2404-lts-arm64"
}
 no
github_runners_default_type Default GitHub Actions Runners instance types for different CPU architectures 
object({
    amd64 = object({
      instance_type               = string
      disk_type                   = string
      disk_size                   = number
      disk_provisioned_iops       = number
      disk_provisioned_throughput = number
    })
    arm64 = object({
      instance_type               = string
      disk_type                   = string
      disk_size                   = number
      disk_provisioned_iops       = number
      disk_provisioned_throughput = number
    })
  })
 
{
  "amd64": {
    "disk_provisioned_iops": 0,
    "disk_provisioned_throughput": 0,
    "disk_size": 10,
    "disk_type": "pd-ssd",
    "instance_type": "e2-standard-4"
  },
  "arm64": {
    "disk_provisioned_iops": 3060,
    "disk_provisioned_throughput": 155,
    "disk_size": 10,
    "disk_type": "hyperdisk-balanced",
    "instance_type": "c4a-standard-4"
  }
}
 no
github_runners_manager_max_instance_count GitHub Actions Runners manager app maximum instance count (Max. number of Cloud Run instances) number 1 no
github_runners_manager_min_instance_count GitHub Actions Runners manager app min. instance count (a minimum of one Cloud Run instance is required to avoid GitHub webhook timeout!) number 1 no
github_runners_types GitHub Actions Runners instance types for different CPU architectures 
list(object({
    name                        = string
    instance_type               = string
    vcpu                        = number
    memory                      = number
    disk_type                   = string
    disk_size                   = number
    disk_provisioned_iops       = number
    disk_provisioned_throughput = number
    image                       = string
    arch                        = string
  }))
 
[
  {
    "arch": "amd64",
    "disk_provisioned_iops": 0,
    "disk_provisioned_throughput": 0,
    "disk_size": 15,
    "disk_type": "pd-ssd",
    "image": "ubuntu-2404-lts-amd64",
    "instance_type": "e2-medium",
    "memory": 4,
    "name": "gcp-ubuntu-slim",
    "vcpu": 2
  },
  {
    "arch": "amd64",
    "disk_provisioned_iops": 0,
    "disk_provisioned_throughput": 0,
    "disk_size": 25,
    "disk_type": "pd-ssd",
    "image": "ubuntu-2404-lts-amd64",
    "instance_type": "e2-standard-4",
    "memory": 16,
    "name": "gcp-ubuntu-latest",
    "vcpu": 4
  },
  {
    "arch": "amd64",
    "disk_provisioned_iops": 0,
    "disk_provisioned_throughput": 0,
    "disk_size": 25,
    "disk_type": "pd-ssd",
    "image": "ubuntu-2404-lts-amd64",
    "instance_type": "e2-standard-4",
    "memory": 16,
    "name": "gcp-ubuntu-24-04",
    "vcpu": 4
  },
  {
    "arch": "amd64",
    "disk_provisioned_iops": 0,
    "disk_provisioned_throughput": 0,
    "disk_size": 75,
    "disk_type": "pd-ssd",
    "image": "ubuntu-2404-lts-amd64",
    "instance_type": "e2-standard-2",
    "memory": 8,
    "name": "gcp-ubuntu-24-04-2core",
    "vcpu": 2
  },
  {
    "arch": "amd64",
    "disk_provisioned_iops": 0,
    "disk_provisioned_throughput": 0,
    "disk_size": 150,
    "disk_type": "pd-ssd",
    "image": "ubuntu-2404-lts-amd64",
    "instance_type": "e2-standard-4",
    "memory": 16,
    "name": "gcp-ubuntu-24-04-4core",
    "vcpu": 4
  },
  {
    "arch": "amd64",
    "disk_provisioned_iops": 0,
    "disk_provisioned_throughput": 0,
    "disk_size": 300,
    "disk_type": "pd-ssd",
    "image": "ubuntu-2404-lts-amd64",
    "instance_type": "e2-standard-8",
    "memory": 32,
    "name": "gcp-ubuntu-24-04-8core",
    "vcpu": 8
  },
  {
    "arch": "amd64",
    "disk_provisioned_iops": 0,
    "disk_provisioned_throughput": 0,
    "disk_size": 600,
    "disk_type": "pd-ssd",
    "image": "ubuntu-2404-lts-amd64",
    "instance_type": "e2-standard-16",
    "memory": 64,
    "name": "gcp-ubuntu-24-04-16core",
    "vcpu": 16
  },
  {
    "arch": "amd64",
    "disk_provisioned_iops": 0,
    "disk_provisioned_throughput": 0,
    "disk_size": 1200,
    "disk_type": "pd-ssd",
    "image": "ubuntu-2404-lts-amd64",
    "instance_type": "e2-standard-32",
    "memory": 128,
    "name": "gcp-ubuntu-24-04-32core",
    "vcpu": 32
  },
  {
    "arch": "amd64",
    "disk_provisioned_iops": 0,
    "disk_provisioned_throughput": 0,
    "disk_size": 2040,
    "disk_type": "pd-ssd",
    "image": "ubuntu-2404-lts-amd64",
    "instance_type": "e2-standard-64",
    "memory": 256,
    "name": "gcp-ubuntu-24-04-64core",
    "vcpu": 64
  },
  {
    "arch": "amd64",
    "disk_provisioned_iops": 0,
    "disk_provisioned_throughput": 0,
    "disk_size": 2040,
    "disk_type": "pd-ssd",
    "image": "ubuntu-2404-lts-amd64",
    "instance_type": "n2d-standard-96",
    "memory": 384,
    "name": "gcp-ubuntu-24-04-96core",
    "vcpu": 96
  },
  {
    "arch": "amd64",
    "disk_provisioned_iops": 0,
    "disk_provisioned_throughput": 0,
    "disk_size": 2040,
    "disk_type": "pd-ssd",
    "image": "ubuntu-2404-lts-amd64",
    "instance_type": "n2d-standard-128",
    "memory": 512,
    "name": "gcp-ubuntu-24-04-128core",
    "vcpu": 128
  },
  {
    "arch": "arm64",
    "disk_provisioned_iops": 3090,
    "disk_provisioned_throughput": 162,
    "disk_size": 15,
    "disk_type": "hyperdisk-balanced",
    "image": "ubuntu-2404-lts-arm64",
    "instance_type": "c4a-standard-1",
    "memory": 4,
    "name": "gcp-ubuntu-slim-arm",
    "vcpu": 1
  },
  {
    "arch": "arm64",
    "disk_provisioned_iops": 3150,
    "disk_provisioned_throughput": 177,
    "disk_size": 25,
    "disk_type": "hyperdisk-balanced",
    "image": "ubuntu-2404-lts-arm64",
    "instance_type": "c4a-standard-4",
    "memory": 16,
    "name": "gcp-ubuntu-latest-arm",
    "vcpu": 4
  },
  {
    "arch": "arm64",
    "disk_provisioned_iops": 3150,
    "disk_provisioned_throughput": 177,
    "disk_size": 25,
    "disk_type": "hyperdisk-balanced",
    "image": "ubuntu-2404-lts-arm64",
    "instance_type": "c4a-standard-4",
    "memory": 16,
    "name": "gcp-ubuntu-24-04-arm",
    "vcpu": 4
  },
  {
    "arch": "arm64",
    "disk_provisioned_iops": 3450,
    "disk_provisioned_throughput": 252,
    "disk_size": 75,
    "disk_type": "hyperdisk-balanced",
    "image": "ubuntu-2404-lts-arm64",
    "instance_type": "c4a-standard-2",
    "memory": 8,
    "name": "gcp-ubuntu-24-04-2core-arm",
    "vcpu": 2
  },
  {
    "arch": "arm64",
    "disk_provisioned_iops": 3900,
    "disk_provisioned_throughput": 365,
    "disk_size": 150,
    "disk_type": "hyperdisk-balanced",
    "image": "ubuntu-2404-lts-arm64",
    "instance_type": "c4a-standard-4",
    "memory": 16,
    "name": "gcp-ubuntu-24-04-4core-arm",
    "vcpu": 4
  },
  {
    "arch": "arm64",
    "disk_provisioned_iops": 4800,
    "disk_provisioned_throughput": 590,
    "disk_size": 300,
    "disk_type": "hyperdisk-balanced",
    "image": "ubuntu-2404-lts-arm64",
    "instance_type": "c4a-standard-8",
    "memory": 32,
    "name": "gcp-ubuntu-24-04-8core-arm",
    "vcpu": 8
  },
  {
    "arch": "arm64",
    "disk_provisioned_iops": 6600,
    "disk_provisioned_throughput": 1040,
    "disk_size": 600,
    "disk_type": "hyperdisk-balanced",
    "image": "ubuntu-2404-lts-arm64",
    "instance_type": "c4a-standard-16",
    "memory": 64,
    "name": "gcp-ubuntu-24-04-16core-arm",
    "vcpu": 16
  },
  {
    "arch": "arm64",
    "disk_provisioned_iops": 10200,
    "disk_provisioned_throughput": 1940,
    "disk_size": 1200,
    "disk_type": "hyperdisk-balanced",
    "image": "ubuntu-2404-lts-arm64",
    "instance_type": "c4a-standard-32",
    "memory": 128,
    "name": "gcp-ubuntu-24-04-32core-arm",
    "vcpu": 32
  },
  {
    "arch": "arm64",
    "disk_provisioned_iops": 15240,
    "disk_provisioned_throughput": 2400,
    "disk_size": 2040,
    "disk_type": "hyperdisk-balanced",
    "image": "ubuntu-2404-lts-arm64",
    "instance_type": "c4a-standard-64",
    "memory": 256,
    "name": "gcp-ubuntu-24-04-64core-arm",
    "vcpu": 64
  },
  {
    "arch": "arm64",
    "disk_provisioned_iops": 15240,
    "disk_provisioned_throughput": 2400,
    "disk_size": 2040,
    "disk_type": "hyperdisk-balanced",
    "image": "ubuntu-2404-lts-arm64",
    "instance_type": "c4a-standard-72",
    "memory": 288,
    "name": "gcp-ubuntu-24-04-72core-arm",
    "vcpu": 72
  }
]
 no
project_id Existing Google Cloud project ID string n/a yes
region Google Cloud region name string "us-central1" no
zone Google Cloud zone suffix string "b" no

Outputs

Name Description
github_runners_manager_url Service URL of the GitHub Actions Runners manager (Cloud Run) https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/blob/v53.0.0/modules/cloud-run-v2/README.md#outputs