diff --git a/cluster-api/README.md b/cluster-api/README.md index 764d48fa..dcfac0d9 100644 --- a/cluster-api/README.md +++ b/cluster-api/README.md @@ -1,42 +1,32 @@ -Requirements -============ +# cluster-api -- Ansible (can be installed using apt or dnf) +This project contains the current versions of Kubernetes that the Cloud Group will support and builds images for. -Setup ------ +## Setup -- Install Ansible, if the version of Ansible core is too old it can be upgraded with: +1. Install Ansible and any other pip requirements: + ```shell + sudo apt-get install python3-venv unzip -y + # or + sudo dnf install python3-venv unzip -y -```shell -sudo apt-get install python3.9-venv unzip -y -python3.9 -m venv venv -source venv/bin/activate -pip install "ansible" "ansible-core" --upgrade -``` + python3 -m venv venv + source venv/bin/activate + pip install -r os_builders/requirements.txt + ``` -- cd to the `os_builders` directory -- Configure the local machine / VM to be a builder: +2. Install Packer and dependencies: + ```shell + cd os_builders + # If sudo is passwordless: + ansible-playbook prep_builder.yml + # If password is required for sudo: + ansible-playbook prep_builder.yml --ask-become-pass + ``` -```shell -cd os_builders +3. Update the date in the [common_vars.json](./common_vars.json#L12) file for `image_name` -# If sudo is passwordless: -ansible-playbook -i inventory/localhost.yml playbooks/prep_builder.yml - -# If password is required for sudo: -ansible-playbook -i inventory/localhost.yml playbooks/prep_builder.yml --ask-become-pass -``` -- Log out and back in again to ensure the groups are applied -```shell -groups | grep -i kvm # no output -exit -# ssh @ -groups | grep -i kvm # output: kvm -``` - -Rate Limiting -------------- +## Rate Limiting You may run into GitHub rate limiting issues when building images. To avoid this, you can set the following environment variable: @@ -44,59 +34,98 @@ You may run into GitHub rate limiting issues when building images. To avoid this The token can be generated from your GitHub settings, under developer access, and only needs the `public_repo` scope (i.e. the default). -Building the latest image -========================= - -- Grab the latest version of the K8s Image Builder submodule: - -```shell -cd .. # back to repo root -git submodule update --init --recursive --remote - -# Point to our custom roles: -export ANSIBLE_ROLES_PATH="$(pwd)/os_builders/roles:$(pwd)/cluster-api/roles" -export PACKER_VAR_FILES="$(pwd)/cluster-api/ansible_stfc_roles.json" - -# Run build -make -C k8s-image-builder/images/capi build-qemu-ubuntu-2204 -``` - -Building a custom version -========================= - -To build a custom version of the image, you can specify the version of the image builder to use additional variables to override the default role definition file: - -```shell -# E.g. to build 1.25.x -cd .. # back to repo root -export ANSIBLE_ROLES_PATH="$(pwd)/os_builders/roles:$(pwd)/cluster-api/roles" -export K8S_VERSION="cluster-api/versions/v1_25.json" -export ROLE_DEFINITION="cluster-api/ansible_stfc_roles.json" - -export PACKER_VAR_FILES="$(pwd)/${K8S_VERSION} $(pwd)/${ROLE_DEFINITION}" - -make -C k8s-image-builder/images/capi build-qemu-ubuntu-2204 -``` - -Adding a new version -==================== -- Update the image builder : - -```shell -cd k8s-image-builder/ -git fetch -git reset --h origin/main -cd .. -``` - -- Navigate to https://kubernetes.io/releases/ -- Find the version you want to add or update -- Update the semver in the relevant JSON file. There should be a 1:1 mapping of -JSON files to major versions of Kubernetes. E.g. a file for 1.24, 1.25, etc. - - -Developer Notes ---------------- +## OpenStack authentication + +You need to set up credentials for OpenStack authentication as we are using a remote builder. Create a clouds.yaml application credential and place it into `~/.config/openstack/clouds.yaml`. See [here](https://stfc.atlassian.net/wiki/spaces/CLOUDKB/pages/211484774/Application+Credentials) for help. + +## Build a specific version + +1. Grab the latest version of the K8s Image Builder submodule: + ```shell + cd .. # back to repo root + git submodule update --init --recursive --remote + ``` +2. Set up variable paths + ```shell + # Point to our custom roles: + export ANSIBLE_ROLES_PATH="$(pwd)/os_builders/roles:$(pwd)/cluster-api/roles" + # Choose K8s version from "versions" directory + export K8S_VERSION="cluster-api/versions/v1_33.json" + # Choose which environment to build in + export PACKER_BUILD_ENV="" + # Tell Packer where the vars files are + export PACKER_VAR_FILES="$(pwd)/${K8S_VERSION} $(pwd)/cluster-api/${PACKER_BUILD_ENV}_vars.json $(pwd)/cluster-api/common_vars.json" + ``` +3. Build the image + ```shell + # Run build + make -C k8s-image-builder/images/capi build-openstack-ubuntu-2204 + # It will be released with the following properties: + # - name: capi-ubuntu-22.04-kube-- + # - visibility: private + ``` +4. Follow steps to update image for release [here](#update-an-image-for-release) + +## Adding a new version +1. Update the image builder: + ```shell + cd .. # back to repo root + git submodule update --init --recursive --remote + ``` +2. Add new version file: + 1. Navigate to https://kubernetes.io/releases/ + 2. Find the version you want to add or update + 3. Create/Update the semver in the relevant JSON file. There should be a 1:1 mapping of JSON files to major versions of Kubernetes. E.g. a file for 1.24, 1.25, etc. + +3. Follow steps to build a specific version [here](#build-a-specific-version) +4. Follow steps to update image for release [here](#update-an-image-for-release) + +## Rebuild all images +1. Update the image builder: + ```shell + cd .. # back to repo root + git submodule update --init --recursive --remote + ``` +2. Update patch versions in version files + ``` + # Contents of ./versions/v1_33.json + { + "kubernetes_series": "v1.33", + "kubernetes_semver": "v1.33.3", -> "v1.33.4" + "kubernetes_deb_version": "1.33.3-1.1" -> "v1.33.4-1.1" + } + ``` +3. Run build-all.sh + ```shell + cd scripts + ./build-all.sh # dev or prod + ``` +4. Update image properties for each image following [Update and image for release](#update-an-image-for-release) + +## Update an image for release +If you need to release an individual image or need to update an existing image you must follow these steps +1. Check the image before you make any changes + ```shell + openstack image show + ``` +2. Update image properties + ```shell + openstack image set \ + --property hw_machine_type=q35 \ + --property hw_disk_bus=scsi \ + --property hw_firmware_type=uefi \ + --property hw_qemu_guest_agent=yes \ + --property hw_scsi_model=virtio-scsi \ + --property hw_vif_multiqueue_enabled=true \ + --property os_require_quiesce=yes \ + + ``` +3. Set image to public + ```shell + openstack image set --public + ``` + +### Developer Notes Since we cannot add comments to JSON files I've documented some points here: - Currently we need to manually update minor versions of Kubernetes. We need to investigate how to update this long-term. diff --git a/cluster-api/ansible_stfc_roles.json b/cluster-api/ansible_stfc_roles.json deleted file mode 100644 index 35dbb9e3..00000000 --- a/cluster-api/ansible_stfc_roles.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "extra_debs": "nfs-common open-iscsi", - "firstboot_custom_roles_post": "vm_baseline containerd" -} \ No newline at end of file diff --git a/cluster-api/common_vars.json b/cluster-api/common_vars.json new file mode 100644 index 00000000..386b5c3a --- /dev/null +++ b/cluster-api/common_vars.json @@ -0,0 +1,14 @@ +{ + "extra_debs": "nfs-common open-iscsi", + "node_custom_roles_post": "vm_baseline containerd", + "source_image": "", + "source_image_filter_name": "ubuntu-jammy-22.04-nogui", + "flavor": "l3.nano", + "volume_size": "50", + "use_blockstorage_volume": "false", + "image_disk_format": "", + "floating_ip_network": "", + "goss_inspect_mode": "true", + "image_name": "capi-{{user `build_name`}}-kube-{{user `kubernetes_semver`}}-YYYY-MM-DD", + "image_visibility": "private" +} \ No newline at end of file diff --git a/cluster-api/dev_vars.json b/cluster-api/dev_vars.json new file mode 100644 index 00000000..6b64a3dd --- /dev/null +++ b/cluster-api/dev_vars.json @@ -0,0 +1,5 @@ +{ + "volume_type": "f7afc62e-999d-4812-9bf3-7f4f93680bad", + "_volume_type_comment": "UUID of volume type cinder # openstack volume type list", + "networks": "fa2f5ebe-d0e0-4465-9637-e9461de443f1" +} \ No newline at end of file diff --git a/cluster-api/prod_vars.json b/cluster-api/prod_vars.json new file mode 100644 index 00000000..09b9e028 --- /dev/null +++ b/cluster-api/prod_vars.json @@ -0,0 +1,5 @@ +{ + "volume_type": "446c92c3-9f5e-43eb-8afd-10726a1b3f9c", + "_volume_type_comment": "UUID of volume type cinder # openstack volume type list", + "networks": "5be315b7-7ebd-4254-97fe-18c1df501538" +} \ No newline at end of file diff --git a/scripts/README.md b/scripts/README.md index eea40f2f..6d4cedf9 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,25 +1,18 @@ -Helper scripts for building multiple images +# Helper scripts for building multiple images - build-all.sh - Build all CAPI images based on the JSON files in the `versions` directory -- upload_all.py - Upload all CAPI images to OpenStack based on the os-cloud provided - -Upload All -========== - -This automatically globs all the output files in the k8s-image-builder directory and uploads them to OpenStack with the -name format including today's date. - -Preparation ------------ - -- Create a venv -- Install the requirements from `requirements.txt` -- Build your images with `build-all.sh` or manually following the readme in the `cluster-api` directory - -Usage ------ - -`upload_all.py` requires the name of the OS cloud to upload to. It also has the following optional arguments: -- `--dry-run` - Do not upload the images, just print the commands -- `--public` - If specified the images will be public, otherwise they will be private to the project +## Preparation +1. Create a venv with the dependencies + ```shell + sudo apt-get install python3-venv + python3 -m venv venv + source venv/bin/activate + pip install -r os_builders/requirements.txt + ``` +2. Build your images with `build-all.sh` or manually following the readme in the `cluster-api` directory + ```shell + # Specify dev or prod OpenStack + ./scripts/build-all.sh + ``` +3. This will build all new images and upload them to OpenStack in the form `ubuntu-2204-kube- diff --git a/scripts/build-all.sh b/scripts/build-all.sh index c78c5f5e..4d18cd75 100755 --- a/scripts/build-all.sh +++ b/scripts/build-all.sh @@ -2,11 +2,24 @@ set -euxo pipefail shift $((OPTIND-1)) +# Enter dev or prod when running this script. I.e ./build-all.sh dev +env=$1 # Get root of repo based on the location of this script REPO_ROOT="$(dirname "$(dirname "$(readlink -fm "$0")")")" # Store the location to the custom roles which are shared from our OS builder... -CUSTOM_ROLE_PATH="${REPO_ROOT}/cluster-api/ansible_stfc_roles.json" + +CUSTOM_ROLE_PATH="${REPO_ROOT}/cluster-api/${env}_vars.json" + +# Update the image name in vars file to include date +COMMON_VARS_PATH="${REPO_ROOT}/cluster-api/common_vars.json" +if grep -E "YYYY-MM-DD" "${COMMON_VARS_PATH}"; then + sed -i -E "s/YYYY-MM-DD/$(date +%F)/" "${COMMON_VARS_PATH}" +else + echo "Date not found in common_vars.json" + exit 1 +fi + # ... and make sure Ansible knows where to find on this machine export ANSIBLE_ROLES_PATH="${REPO_ROOT}/os_builders/roles:${REPO_ROOT}/cluster-api/roles" @@ -16,6 +29,6 @@ VERSIONS=( "${REPO_ROOT}"/cluster-api/versions/*.json ) for version_path in "${VERSIONS[@]}"; do echo "Building image for version: ${version_path}..." && \ - export PACKER_VAR_FILES="${CUSTOM_ROLE_PATH} ${version_path}" && \ - make -C "${REPO_ROOT}/k8s-image-builder/images/capi" build-qemu-ubuntu-2204 & + export PACKER_VAR_FILES="${CUSTOM_ROLE_PATH} ${COMMON_VARS_PATH} ${version_path}" && \ + make -C "${REPO_ROOT}/k8s-image-builder/images/capi" build-openstack-ubuntu-2204 & done