Quattor role

.github/workflows/os_builder.yaml

@@ -20,11 +20,11 @@ jobs:
             sudo apt-get update
             sudo apt-get install -y ansible
             cd os_builders
-            ansible-playbook -i inventory/localhost.yml playbooks/prep_builder.yml
+            ansible-playbook prep_builder.yml
 
         - name: Validate packer files
           run: |
-            cd os_builders/packfiles &&
+            cd os_builders &&
             packer init . &&
             packer validate -syntax-only .
 
@@ -45,11 +45,12 @@ jobs:
             pip3 install --upgrade pip
             pip3 install ansible python-debian
         - name: Run pre-prep playbook
-          # Patch the inventory to run on this machine
           run: |
-           . venv/bin/activate
-           cd os_builders && sed -i 's/all/default/g' inventory/localhost.yml
-           ansible-playbook -i inventory/localhost.yml playbooks/prepare_user_image.yml --extra-vars provision_this_machine=True
+            . venv/bin/activate
+            cd os_builders
+            ansible-playbook vm_baseline.yml
+            ansible-playbook image_fixes.yml
+            ansible-playbook tidy_image.yml
 
   test_image_build_rocky:
     strategy:
@@ -62,8 +63,11 @@ jobs:
       - name: Install Ansible
         run: |
           dnf install epel-release -y
+          # These have to be separate steps as ansible is provided by epel-release
           dnf install ansible -y
       - name: Run pre-prep playbook
         run: |
-          cd os_builders && sed -i 's/all/default/g' inventory/localhost.yml
-          ansible-playbook -i inventory/localhost.yml playbooks/prepare_user_image.yml --extra-vars provision_this_machine=True
+          cd os_builders
+          ansible-playbook vm_baseline.yml
+          ansible-playbook image_fixes.yml
+          ansible-playbook tidy_image.yml

os_builders/README.md

@@ -1,131 +1,157 @@
-Image Building
-==============
+# Image Building
 
-This directory contains the Ansible playbooks and Packer templates for building the VM images.
-It sets a baseline for all VMs to comply with our security policies, in an OS agnostic way.
+## Contents:
 
-Pipeline
---------
+- [Pipeline](#pipeline)
+- [Setting Up the Environment](#setting-up-the-environment)
+- [Building Images for Release](#building-images-for-release)
+- [Testing Changes to Images (Troubleshoot or Bug Fixing)](#testing-changes-to-images-troubleshoot-or-bug-fixing)
+- [Project Layout](#project-layout)
 
-The pipeline consists of the following steps:
-
-- Packer builds the VM image, with a packer user and password set to `packer`
-- The VM is booted and Ansible is run to configure the VM using the `image_prep.yml` playbook
-- The image is tidied, the packer user is deleted and the VM shuts down. 
-- Packer converts this to a qcow2 image ready for upload.
-
-*Note: This script does not upload or test the image, this is done separately currently*
-
-Building Locally
-=================
-
-The easiest way is to run the Ansible playbooks which will prepare the current machine, and
-also handle the multi-stage builds.
-
-Preparing a builder
--------------------
-
-To build locally, you need to have the following installed:
-- ansible
-
-First, install required ansible collections:
-```
-ansible-galaxy install -r requirements.yml
-```
-
-Then run the following command to install Qemu and setup the user's groups. You will need to log out and back in again for the groups to take effect.
-
-
-```
-ansible-playbook -i inventory/localhost playbooks/prep_builder.yml --ask-become-pass
-```
-
-Running the build
---------------------
-
-Images can be built with the following command
-```
-ansible-playbook -i inventory/localhost playbooks/builder.yml
-````
-
-This will build all images (it implies the `all` tag). Individual tags can be selected as follows
-
-```
-ansible-playbook -i inventory/localhost playbooks/builder.yml -t <tag>
-````
-
-The following tags are available:
-- `all` - Build all images (default)
-- `ubuntu` - Build all Ubuntu variants
-- `ubuntu_2204` - Build Ubuntu 22.04
-
-
-Running builds on a VM
-----------------------
-
-By default we show a VNC window for the packer build, this is useful for debugging but will not work on a headless VM.
-
-To run a headless build, you need to set the `headless` variable to true. This can be done by passing the variable on the command line:
+## Pipeline
 
-```
-ansible-playbook -i inventory/localhost playbooks/builder.yml --extra-vars packer_headless=true
-```
-
-
-Development and Testing
-========================
-
-Packer uses a multi stage build. Stage 1 will build the VM image using auto-install. Stage 2 will provision the image with any customisations we want.
-
-This allows us to rapidly iterate on our customisations without having to wait for the OS install to complete.
-
-Directory Layout
-----------------
-
-- `packfiles/` - Packer templates for building the VM images
-- `packfiles/ubuntu_sources.pkr.hcl` - Contains image definitions for Ubuntu
-- `packfiles/rocky_sources.pkr.hcl` - Contains image definitions for Rocky (TODO)
-- `packfiles/build.pkr.hcl` - Contains the build steps for the VM image. This uses a two stage build described below
-
-CI/CD Files:
-
-- `packfiles/headless.pkrvars.hcl` - Contains the variables for doing a headless build
-
-
-Testing New OS Variants
---------------------------
-It's recommended you run this locally, so that you can see the VNC window and debug any issues:
-
-- Ensure the builder is configured, as above
-- Add your new build to the sources file, you need to add a base step and a provisioning step. See the Ubuntu file for an example.
-- Run your new/modified stage 1 build through the auto-install step: `cd packerfiles && packer build --only=stage1*<name>*` 
-
-For example:
-`cd packerfiles && packer build --only=stage1*ubuntu_2204 .`
-
-- Test the provisioning step on your new image:
-`cd packerfiles && packer build --only=stage2*<name>*`
-
-
-Prototyping new Ansible changes on a VM
-----------------------------------------
-It's recommended you use an existing VM for this testing, as it will be quicker than running an OS install and uploading :
-
-- Add your hosts to a testing inventory file, e.g. `cat inventory/testing.yml`:
-
-```
-all:
-    hosts:
-        host-172-16-255-255.nubes.stfc.ac.uk:
-            ansible_user: ubuntu  # or rocky
-```
-
-**Ensure you are on a VM!**
-
-The `provision_this_machine` variable acts as a guard from trashing your own machine. 
+The pipeline consists of the following steps:
 
-- Run the playbook
-```
-ansible-playbook -i inventory/testing.yml playbooks/provision_image.yml --extra-vars provision_this_machine=true
+- Packer pulls the latest generic image from the OS mirror into OpenStack
+- Packer generates a SSH key and uploads it to OpenStack
+- The VM is booted with the public key and uses the app creds user
+- Ansible is run to configure the VM using the [prep_user_image](prep_user_image.yml) playbook
+- Packer snapshots the VM and deletes the VM, SSH key and generic image
+
+## Setting up the environment
+
+1. Install pip, Ansible, Packer and OpenStack CLI
+  ```shell
+  # Ubuntu
+  sudo apt install python3-pip python3-venv -y
+
+  # Rocky
+  sudo dnf install python3-pip python3-venv -y
+
+  python3 -m venv image_builders
+  source image_builder/bin/activate
+  pip install -r requirements.txt
+
+  ansible-playbook prep_builder.yml
+  ```
+2. Create an applications credential (admin is only required to make images public)
+  ```shell
+  # Either place app creds in directory
+
+  mkdir -p ~/.config/openstack
+  mv clouds.yaml ~/.config/openstack/clouds.yaml
+
+  # Or
+
+  # Export credentials
+  export OS_AUTH_URL=https://<openstack_url>:5000/v3
+  export OS_APPLICATION_CREDENTIAL_ID=<app_cred_id>
+  export OS_APPLICATION_CREDENTIAL_SECRET=<app_cred_secret>
+  ```
+3. Git clone repository
+  ```shell
+  git clone https://github.com/stfc/cloud-image-builders.git
+  cd cloud-image-builders/os_builders
+  ```
+
+## Building Images for Release
+
+1. Activate virtual environment if not already
+  ```shell
+  source image_builder/bin/activate  # As made in the set up steps
+  ```
+
+2. Edit OpenStack External Network ID
+  ```shell
+  # Contents of: build.pkr.hcl
+  18 networks          = [""]  # OpenStack External Network ID
+  ```
+3. Run Packer
+  ```shell
+  packer build .
+  # Or to build only certain images
+  packer build -only openstack.<builder-name>,openstack.<builder-name> .
+  ```
+4. Rename the current images to warehoused and new images to current name
+  ```shell
+  # REQUIRES ADMIN
+  # For each image you are releasing
+  current_image_name="<current-image-name>"
+  new_image_id="<new-image-name>"
+  timestamp=$(date +%F)
+  openstack image set --deactivate --name "warehoused-${current_image_name}-${timestamp}" $current_image_name
+  openstack image set --public --name "${current_image_name}" $new_image_id
+
+  # For example, this may look like the below
+  current_image_name="ubuntu-noble-24.04-nogui"
+  new_image_id="ubuntu-noble-24.04-nogui-2025-11-20-abcde"
+  timestamp=$(date +%F)
+  openstack image set --deactivate --name "warehoused-${current_image_name}-${timestamp}" $current_image_name
+  openstack image set --public --name "${current_image_name}" $new_image_id
+
+  # ubuntu-noble-24.04-nogui is:
+  #  - deactivated
+  #  - renamed to warehoused-ubuntu-noble-24.04-nogui-2025-22-20
+  # ubuntu-noble-24.04-nogui-2025-11-20-abcde is:
+  #  - set to public
+  #  - renamed to ubuntu-noble-24.04-nogui
+  ```
+
+## Testing Changes to Images (Troubleshoot or Bug Fixing)
+
+1. Activate virtual environment if not already
+  ```shell
+  source image_builder/bin/activate  # As made in the set up steps
+  ```
+2. Generate a temporary SSH key to use on the VMs and add to OpenStack
+  ```shell
+  passphrase=$(pwgen 10 1)
+  fed_id=<fed-id>
+  ssh-keygen -t rsa -f image_testing_key -N $passphrase
+  openstack keypair create --public-key=./image_testing_key.pub "image-testing-key-${fed_id}"
+  ```
+3. Create a VM using the current latest image for the OS you are fixing
+  ```shell
+  openstack server create <server-name> --image <os-image> --key-name "image-testing-key-${fed_id}" \
+  --flavor l3.nano --network Internal  --wait
+  ```
+4. Edit `inventory.yml` and add your hosts IP
+  ```shell
+  # Get IP of VM
+  openstack server show <server-name> -f json | jq .addresses.Internal | jq first
+
+  # Contents of: inventory.yml
+  5 ansible_host: "172.16.255.255"  # Your hosts IP
+  ```
+5. Run the baseline against the VM
+  ```shell
+  ansible-playbook -i inventory vm_baseline.yml
+  ```
+6. Run any other custom playbooks against the VM which you want to test
+  ```shell
+  ansible-playbook -i inventory image_fixes.yml
+  ansible-playbook -i inventory tidy_image.yml
+  ansible-playbook -i inventory <playbook.yml>
+  ```
+
+7. Repeat step 5/6 making changes to the playbooks and commit and PR any changes that are working.
+
+## Project Layout
+```shell
+os_builders
+├── README.md
+├── build.pkr.hcl  # Packer build file
+├── galaxy.yml  # Ansible Galaxy collection metadata
+├── prep_builder.yml  # Playbook to install Packer
+├── vm_baseline.yml  # Playbook to configure the images
+├── image_fixes.yml  # Playbook to apply fixes to the images
+├── quattor.yml  # Playbook to install quattor onto the image
+├── tidy_image.yml  # Playbook to tidy the image before snapshotting
+├── requirements.txt  # Specifies Ansible version
+└── roles  # Roles to configure the image
+    ├── container_registry/
+    ├── nubes_bootcontext/
+    ├── prep_builder/
+    ├── tidy_image/
+    └── vm_baseline/
 ```
-