diff --git a/.gitignore b/.gitignore index c03ced65..32042ae0 100644 --- a/.gitignore +++ b/.gitignore @@ -3,6 +3,7 @@ pre-commit rosetta .local +.DS_Store share hooks/__pycache__/ diff --git a/content/cluster-installation/disconnected/assets/disco-cluster-operators.png b/content/cluster-installation/disconnected/assets/disco-cluster-operators.png new file mode 100644 index 00000000..416c5f78 Binary files /dev/null and b/content/cluster-installation/disconnected/assets/disco-cluster-operators.png differ diff --git a/content/cluster-installation/disconnected/assets/openshift-mirror-registry-diagramm.png b/content/cluster-installation/disconnected/assets/openshift-mirror-registry-diagramm.png new file mode 100644 index 00000000..1dcdc3fc Binary files /dev/null and b/content/cluster-installation/disconnected/assets/openshift-mirror-registry-diagramm.png differ diff --git a/content/cluster-installation/disconnected/assets/quay-mirror-registry.png b/content/cluster-installation/disconnected/assets/quay-mirror-registry.png new file mode 100644 index 00000000..6f1bfb2f Binary files /dev/null and b/content/cluster-installation/disconnected/assets/quay-mirror-registry.png differ diff --git a/content/cluster-installation/disconnected/disconnected-installation.md b/content/cluster-installation/disconnected/disconnected-installation.md new file mode 100644 index 00000000..79d31726 --- /dev/null +++ b/content/cluster-installation/disconnected/disconnected-installation.md @@ -0,0 +1,1678 @@ +# Disconnected Agent-Based Cluster Installation + +A disconnected environment is an environment that does not have full access to the internet. + +OpenShift Container Platform is designed to perform many automatic functions that depend on an internet connection, such as retrieving release images from a registry or retrieving update paths and recommendations for the cluster. Without a direct internet connection, you must perform additional setup and configuration for your cluster to maintain full functionality in the disconnected environment. + +Source: [Understanding disconnected installation mirroring](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/installing_an_on-premise_cluster_with_the_agent-based_installer/understanding-disconnected-installation-mirroring) + +## How it works + +You can use a mirror registry for disconnected installations and to ensure that your clusters only use container images that satisfy your organization’s controls on external content. Before you install a cluster on infrastructure that you provision in a disconnected environment, you must mirror the required container images into that environment. To mirror container images, you must have a registry for mirroring. + +## Connected Mirroring vs Disconnected Mirroring + +**Connected Mirroring** is if you have a host that can access both the internet and your mirror registry, but not your cluster nodes, you can directly mirror the content from that machine. + +**Disconnected Mirroring** is if you have no such host, you must mirror the images to a file system and then bring that host or removable media into your restricted environment. + +## Lab Environment Setup + +At a high-level, the testing setup has a bastion host which is dual-homed configured. One NIC is connected to the "public" network and the other NIC to the "disconnected" network. Most customer setups don't have a bastion connected to the public (internet) network. Anyways! We're going to "download" the images and storing all in a tar ball. This tar ball will be copied over to the mirror-registry host. + +![network-diagram](assets/openshift-mirror-registry-diagramm.png) + +## Bastion Host Preperation + +### Hostname + +Make sure that your `hostname` is set correctly! + +```code +hostnamectl +``` + +The hostname must be a fqdn! + +```code +sudo hostnamectl set-hostname rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com +sudo reboot +``` + +### RHEL Subscription Manager + +Configure RHEL Subscription Manager: + +`sudo subscription-manager register --username --password '' --auto-attach` + +### Networking Bastion-Host + +Setup a Bastion Host with two nics. One is connected to the "internet-zone" and the other one to the disconnected network. + +Configure the interfaces accordingly: + +Configure the interface which has internet connection: + +```code +nmcli con show + +NAME UUID TYPE DEVICE +System eth0 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 ethernet eth0 +lo dd314177-6d3f-4ad3-a1af-2875d094c193 loopback lo +Wired connection 1 c8d40ef7-3d02-3ba5-b047-dacd6d013b24 ethernet -- +``` + +```bash +nmcli con mod "System eth0" \ +ipv4.addresses 10.32.96.145/20 \ +ipv4.gateway 10.32.111.254 \ +ipv4.dns "10.32.96.1,10.32.96.31" \ +ipv4.method manual +``` + +Configure the interface which is connected to the disconnected network: + +```bash +nmcli con mod "Wired connection 1" \ +ipv4.addresses 192.168.69.208/24 \ +ipv4.method manual +``` + +Bring both interfaces up: + +```code +nmcli dev reapply eth0 && nmcli dev reapply eth1 + +Connection successfully reapplied to device 'eth0'. +Connection successfully reapplied to device 'eth1'. +``` + +Alternatively: + +```code +nmcli con up "System eth0" && nmcli con up "Wired connection 1" + +Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/25) +Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/26) +``` + +Check the new config: + +```code +ip -br a +lo UNKNOWN 127.0.0.1/8 ::1/128 +eth0 UP 10.32.96.145/20 2620:52:0:2060:d8:6dff:fe0f:3ed3/64 fe80::d8:6dff:fe0f:3ed3/64 +eth1 UP 192.168.69.208/24 fe80::a0eb:9896:d4e1:3fbb/64 +``` + +The configuration is stored under: + +```code +nmcli -f NAME,UUID,FILENAME con show +NAME UUID FILENAME +System eth0 5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 /etc/sysconfig/network-scripts/ifcfg-eth0 +Wired connection 1 c8d40ef7-3d02-3ba5-b047-dacd6d013b24 /etc/NetworkManager/system-connections/Wired connection 1.nmconnection +lo dd314177-6d3f-4ad3-a1af-2875d094c193 /run/NetworkManager/system-connections/lo.nmconnection +``` + +Readable configuration: + +```bash +less /etc/sysconfig/network-scripts/ifcfg-eth0 + +# Created by cloud-init automatically, do not edit. +# +AUTOCONNECT_PRIORITY=120 +BOOTPROTO=none +DEVICE=eth0 +HWADDR=02:D8:6D:0F:3E:D3 +IPV6INIT=yes +ONBOOT=yes +TYPE=Ethernet +USERCTL=no +PROXY_METHOD=none +BROWSER_ONLY=no +IPADDR=10.32.96.145 +PREFIX=20 +GATEWAY=10.32.111.254 +DNS1=10.32.96.1 +DNS2=10.32.96.31 +DEFROUTE=yes +IPV4_FAILURE_FATAL=no +IPV6_AUTOCONF=yes +IPV6_DEFROUTE=yes +IPV6_FAILURE_FATAL=no +NAME="System eth0" +UUID=5fb06bd0-0bb0-7ffb-45f1-d6edd65f3e03 +``` + +And the other interface: + +```bash +less /etc/NetworkManager/system-connections/Wired\ connection\ 1.nmconnection + +[connection] +id=Wired connection 1 +uuid=c8d40ef7-3d02-3ba5-b047-dacd6d013b24 +type=ethernet +autoconnect-priority=-999 +interface-name=eth1 + +[ethernet] + +[ipv4] +address1=192.168.69.208/24 +method=manual + +[ipv6] +addr-gen-mode=default +method=auto + +[proxy] +``` + +Enable IP Forwarding: `sysctl -w net.ipv4.ip_forward=1` + +This allows the VM to forward packets between interfaces. + +Make it permanently: + +```bash +vi /etc/sysctl.conf +net.ipv4.ip_forward = 1 +``` + +Apply the changes: `sysctl -p` + +On Red Hat Enterprise Linux, `firewall-cmd` is provided by the firewalld package. It is not guaranteed to be installed in minimal VM images. + +Check whether firewalld is installed: + +```code +rpm -q firewalld +``` + +If it is missing, install and enable it: + +```code +sudo dnf install -y firewalld +sudo systemctl enable --now firewalld +``` + +Allow forwarding: + +`firewall-cmd --permanent --add-forward-port=port=22:proto=tcp:toport=22` + +Add trusted zones for both interfaces: + +```bash +firewall-cmd --permanent --zone=public --add-interface=eth0 +firewall-cmd --permanent --zone=trusted --add-interface=eth1 +``` + +Check the configs: + +```code +firewall-cmd --get-active-zones +public + interfaces: eth0 +trusted + interfaces: eth1 +``` + +Also: + +```code +firewall-cmd --zone=public --list-all +public (active) + target: default + icmp-block-inversion: no + interfaces: eth0 + sources: + services: cockpit dhcpv6-client ssh + ports: + protocols: + forward: yes + masquerade: no + forward-ports: + source-ports: + icmp-blocks: + rich rules: +``` + +```code +firewall-cmd --zone=trusted --list-all +trusted (active) + target: ACCEPT + icmp-block-inversion: no + interfaces: eth1 + sources: + services: + ports: + protocols: + forward: yes + masquerade: no + forward-ports: + source-ports: + icmp-blocks: + rich rules: +``` + +We will need to open up port 8443 for Quay: + +```code +firewall-cmd --add-port 8443/tcp --permanent +``` + +Reload firewall rules: `firewall-cmd --reload` + +Configure Routing (if needed). + +If devices in 192.168.69.0/24 need internet access via the VM, you need NAT: + +```bash +sudo firewall-cmd --permanent --add-masquerade +sudo firewall-cmd --reload +``` + +Check IP forwarding: `cat /proc/sys/net/ipv4/ip_forward` + +### SSH + +Copy your ssh key to the bastion host (client --> bastion) + +`ssh-copy-id -f ~/.ssh/id_ed25519 rguske@rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com` + +Generate a SSH key pair on your bastion host. The ssh key can be used for troubleshooting purposes with the OpenShift nodes. + +`ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519` + +### Command Line Interfaces (CLIs) + +On the bastion host, download the necessary cli's from [Homepage](https://console.redhat.com/openshift/downloads): + +You could use `curl -LO ` for it: + +OpenShift Installer: `curl -LO https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.21.11/openshift-install-rhel9-amd64.tar.gz` + +OpenShift Client: `curl -LO https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.21.11/openshift-client-linux-amd64-rhel9-4.21.11.tar.gz` + +The "mirror" plugin for the OpenShift CLI client (oc) controls the process of mirroring all relevant container image for a full disconnected OpenShift installation in a central, declarative tool. Learn more(new window or tab) + +RHEL 9 is FIPS compatible; RHEL 8 is non-FIPS compatible. + +OpenShift Mirror CLI: `curl -LO https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/latest/oc-mirror.rhel9.tar.gz` + +Download and install a local, minimal single instance deployment of Red Hat Quay to aid bootstrapping the first disconnected cluster. + +OpenShift Tiny Quay Registry: `curl -LO https://mirror.openshift.com/pub/cgw/mirror-registry/latest/mirror-registry-amd64.tar.gz` + +This helps untaring the packages: + +```code +alias untar='tar -zxvf' +``` + +This is how it should like if you've downloaded all: + +```code +tree +. +├── clis +│   ├── oc-mirror.rhel9.tar.gz +│   ├── openshift-client-linux-amd64-rhel9-4.21.11.tar.gz +│   └── openshift-install-rhel9-amd64.tar.gz +└── mirror-registry + └── mirror-registry-amd64.tar.gz +``` + +Unpack the `.gz` files, except execution-environment.tar, image-archive.tar and sqlite3.tar of the folder mirror-registry and move them into `/usr/local/bin`: + +```code +sudp mv {kubectl,oc,oc-mirror,openshift-install-fips} /use/local/bin +``` + +Apply rights: + +```code +sudo chown -R $USER /usr/local/bin/{kubectl,oc,oc-mirror,openshift-install-fips} +sudo chmod +x /usr/local/bin/{kubectl,oc,oc-mirror,openshift-install-fips} +``` + +```code +tree /usr/local/bin +/usr/local/bin +├── execution-environment.tar +├── firstboot-network-firewall.sh +├── image-archive.tar +├── kubectl +├── mirror-registry +├── oc +├── oc-mirror +├── openshift-install-fips +└── sqlite3.tar + +0 directories, 9 file +``` + +If /usr/local/bin isn't included in the `$PATH`, run +`export PATH=/usr/local/bin:$PATH` + +### Install Podman and Nmstate + +In order to run the mirror registry, the bastion host needs a container-runtime installed. + +Podman is the runtime of choice and is included in the `container-tools` package. + +Install it via `sudo dnf install container-tools -y`. + +The installer also uses `nmstatectl` for the creation of the agent.iso. Install it via `sudo dnf install nmstate -y`. Otherwise, you'll get the error: + +```code +FATAL * failed to validate network yaml for host 0, install nmstate package, exec: "nmstatectl": executable file not found in $PATH +``` + +### Installing Podman Offline + +- Mount a RHEL installation iso file to the system (VM or BM via Board Management Controller). + +Check e.g. with `lsblk` for the disconnected "cdrom" (iso) device: + +```bash +[root@mirror-rguske ~]# lsblk +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS +sda 8:0 0 120G 0 disk +├─sda1 8:1 0 600M 0 part /boot/efi +├─sda2 8:2 0 1G 0 part /boot +└─sda3 8:3 0 118.4G 0 part + ├─rhel-root 253:0 0 70G 0 lvm / + ├─rhel-swap 253:1 0 7.9G 0 lvm [SWAP] + └─rhel-home 253:2 0 40.5G 0 lvm /home +sr0 11:0 1 11G 0 rom +``` + +- Create a folder in which the iso content will be accessible: + +`mkdir -p /mnt/rhel-iso` + +- Mount the connected iso accordingly: + +`mount -o loop /path/to/rhel.iso /mnt/rhel-iso` + +Example in my case with a VM: `mount -o loop /dev/sr0 /mnt/rhel-iso` + +- Create a Local Repository: + +```code +cat < The OpenShift image registry cannot be used as the target registry because it does not support pushing without a tag, which is required during the mirroring process. + +Install the mirror registry: + +At this point it is important to understand that my bastion is dual-homed. + +```code +dig +short rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com +10.32.96.145 +dig +short rguske-rhel9-disco-bastion.disco.local +192.168.69.208 +``` + +In this scenario it is important to use the DNS record which points to the IP in the disco subnet. + +```code +mirror-registry install \ + --quayHostname rguske-rhel9-disco-bastion.disco.local \ + --quayRoot "$HOME/quay-install/root" \ + --quayStorage $HOME/quay-install \ + --sqliteStorage $HOME/quay-install \ + --initPassword 'r3dh4t1!' \ + --verbose + + __ __ + / \ / \ ______ _ _ __ __ __ + / /\ / /\ \ / __ \ | | | | / \ \ \ / / +/ / / / \ \ | | | | | | | | / /\ \ \ / +\ \ \ \ / / | |__| | | |__| | / ____ \ | | + \ \/ \ \/ / \_ ___/ \____/ /_/ \_\ |_| + \__/ \__/ \ \__ + \___\ by Red Hat + Build, Store, and Distribute your Containers + +INFO[2026-04-22 05:16:03] Install has begun +DEBU[2026-04-22 05:16:03] Ansible Execution Environment Image: quay.io/quay/mirror-registry-ee:latest +DEBU[2026-04-22 05:16:03] Pause Image: registry.access.redhat.com/ubi8/pause:8.10-5 +DEBU[2026-04-22 05:16:03] Quay Image: registry.redhat.io/quay/quay-rhel8:v3.12.14 +DEBU[2026-04-22 05:16:03] Redis Image: registry.redhat.io/rhel8/redis-6:1-1766406130 +INFO[2026-04-22 05:16:03] Found execution environment at /usr/local/bin/execution-environment.tar +INFO[2026-04-22 05:16:03] Loading execution environment from execution-environment.tar +DEBU[2026-04-22 05:16:03] Importing execution environment with command: /bin/bash -c /usr/bin/podman image import \ + --change 'ENV PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' \ + --change 'ENV HOME=/home/runner' \ + --change 'ENV container=oci' \ + --change 'ENTRYPOINT=["entrypoint"]' \ + --change 'WORKDIR=/runner' \ + --change 'EXPOSE=6379' \ + --change 'VOLUME=/runner' \ + --change 'CMD ["ansible-runner", "run", "/runner"]' \ + - quay.io/quay/mirror-registry-ee:latest < /usr/local/bin/execution-environment.tar +Getting image source signatures +Copying blob 159de7f3f142 done | +Copying config 3055d6ebc0 done | +Writing manifest to image destination +sha256:3055d6ebc0dd81d1b676e94a7c1c06eef2c94ddc5abe2f197147b013184afd81 +INFO[2026-04-22 05:16:13] Detected an installation to localhost +INFO[2026-04-22 05:16:13] Did not find SSH key in default location. Attempting to set up SSH keys. +INFO[2026-04-22 05:16:13] Generating SSH Key +Generating public/private rsa key pair. +Your identification has been saved in /root/.ssh/quay_installer +Your public key has been saved in /root/.ssh/quay_installer.pub + +[...] + +TASK [mirror_appliance : Create init user] *********************************************************************************************************************************************************************************************************************************************************************** +included: /runner/project/roles/mirror_appliance/tasks/create-init-user.yaml for rguske@rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com + +TASK [mirror_appliance : Creating init user at endpoint https://rguske-rhel9-disco-bastion.disco.local:8443/api/v1/user/initialize] ****************************************************************************************************************************************************************************** +ok: [rguske@rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com] + +TASK [mirror_appliance : Enable lingering for systemd user processes] ******************************************************************************************************************************************************************************************************************************************** +changed: [rguske@rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com] + +PLAY RECAP ******************************************************************************************************************************************************************************************************************************************************************************************************* +rguske@rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com : ok=49 changed=29 unreachable=0 failed=0 skipped=15 rescued=0 ignored=0 + +INFO[2026-04-24 10:18:34] Quay installed successfully, config data is stored in /home/$USER/downloads/mirror-registry/root +INFO[2026-04-24 10:18:34] Quay is available at https://rguske-rhel9-disco-bastion.disco.local:8443 with credentials (init, r3dh4t1!) +``` + +Alternative customizations examples: + +1.: `/mirror-registry install --quayHostname mirror.example.org --sslKey tls.key --targetHostname internal.mirror --quayRoot /var/mirror-registry --initPassword changeme` + +Source: [Mirror registry for Red Hat OpenShift flags](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/disconnected_environments/installing-mirroring-creating-registry). + +![mirror-registry](assets/quay-mirror-registry.png) + +### Validating the installation + +Validating the endpoint using `curl`: + +```code +curl -k https://rguske-rhel9-disco-bastion.disco.local:8443/health/instance +{"data":{"services":{"auth":true,"database":true,"disk_space":true,"registry_gunicorn":true,"service_key":true,"web_gunicorn":true}},"status_code":200} +``` + +Checking the certificate: + +```code +echo | openssl s_client -connect rguske-rhel9-disco-bastion.disco.local:8443 -showcerts +Connecting to 192.168.69.208 +CONNECTED(00000003) +depth=1 C=US, ST=VA, L=New York, O=Quay, OU=Division, CN=rguske-rhel9-disco-bastion.disco.local +verify error:num=19:self-signed certificate in certificate chain +verify return:1 +depth=1 C=US, ST=VA, L=New York, O=Quay, OU=Division, CN=rguske-rhel9-disco-bastion.disco.local +verify return:1 +depth=0 CN=quay-enterprise +verify return:1 +--- +Certificate chain + 0 s:CN=quay-enterprise + i:C=US, ST=VA, L=New York, O=Quay, OU=Division, CN=rguske-rhel9-disco-bastion.disco.local + a:PKEY: RSA, 2048 (bit); sigalg: sha256WithRSAEncryption + v:NotBefore: Apr 24 14:17:18 2026 GMT; NotAfter: Apr 15 14:17:18 2027 GMT +-----BEGIN CERTIFICATE----- + +[...] +``` + +Also, validate the certificate which we are going trust on our bastion host. + +```code +openssl x509 -in ~/downloads/mirror-registry/root/quay-config/ssl.cert -text -noout +Certificate: + Data: + Version: 3 (0x2) + Serial Number: + 16:06:dd:b0:be:99:47:81:76:52:85:9c:15:1e:76:0d:ab:99:35:ea + Signature Algorithm: sha256WithRSAEncryption + Issuer: C=US, ST=VA, L=New York, O=Quay, OU=Division, CN=rguske-rhel9-disco-bastion.disco.local + Validity + Not Before: Apr 24 14:17:18 2026 GMT + Not After : Apr 15 14:17:18 2027 GMT +[...] +``` + +Systemd auto-start is also configured: + +```code +sudo systemctl list-units --type service | grep quay + quay-app.service loaded active running Quay Container + quay-pod.service loaded active exited Infra Container for Quay + quay-redis.service loaded active running Redis Podman Container for Quay +``` + +### Login into the Mirror Registry + +```code +podman login -u init -p 'r3dh4t1!' https://rguske-rhel9-disco-bastion.disco.local:8443 --tls-verify=false +``` + +It is also possible without using the option `--tls-verify=false` by trusting the newly created certificates which are stored in `root` / `quay-config`: + +```code +tree +. +├── pause.tar +├── quay.tar +├── redis.tar +└── root + ├── quay-config + │   ├── config.yaml + │   ├── openssl.cnf + │   ├── ssl.cert + │   ├── ssl.csr + │   └── ssl.key + └── quay-rootCA + ├── rootCA.key + ├── rootCA.pem + └── rootCA.srl +``` + +Copy the certs: + +`sudo cp ~/downloads/mirror-registry/root/quay-config/ssl.cert /etc/pki/ca-trust/source/anchors/` + +```code +tree /etc/pki/ca-trust/source/anchors/ +└── ssl.cert + +0 directories, 1 file +``` + +Update the trust: + +`update-ca-trust` + +Logout: + +```code +podman logout https://rguske-rhel9-disco-bastion.disco.local:8443 + +Removed login credentials for rguske-rhel9-disco-bastion.disco.local:8443 +``` + +Login again: + +```code +podman login -u init -p 'r3dh4t1!' 'https://rguske-rhel9-disco-bastion.disco.local:8443' + +Login Succeeded! +``` + +### Uninstalling the Mirror Registry + +`mirror-registry uninstall` + +## Mirroring Images + +> You must have access to the internet to obtain the necessary container images. In this procedure, you place your mirror registry on a mirror host that has access to both your network and the internet. If you do not have access to a mirror host, use the [Mirroring Operator catalogs](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html-single/disconnected_environments/index#olm-mirror-catalog_installing-mirroring-installation-images) for use with disconnected clusters procedure to copy images to a device you can move across network boundaries with. + +Procedure and prerequisites: + +- You configured a mirror registry to use in your disconnected environment. +- You identified an image repository location on your mirror registry to mirror images into. +- You provisioned a mirror registry account that allows images to be uploaded to that image repository. +- You have write access to the mirror registry. + +Obtain your [pull secret from Red Hat OpenShift Cluster Manager](https://console.redhat.com/openshift/install/pull-secret) and paste the json content into $XDG_RUNTIME_DIR/containers/auth.json. Be careful! It must be in `json`. + +Make a copy of your pull secret in JSON format by running the following command: + +Paste the content in the file pull-scret. Then: + +`cat ./pull-secret | jq . > $(pwd)/pull-secret.json` + +Replace the existing `auth.json` file in $XDG_RUNTIME_DIR/containers/ + +```code +sudo mv pull-secret.json $XDG_RUNTIME_DIR/containers/auth.json +``` + +Next up is to generate the base64-encoded user name and password or token for your mirror registry by running the following command: + +`echo -n ':' | base64 -w0` + +For and , specify the user name and password that you configured for your registry. + +Example: + +`echo -n 'init:r3dh4t1!' | base64 -w0` + +Edit the JSON file and add a section that describes your registry to it: + +```json + "auths": { + "rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com:8443": { + "auth": "aW5pdDpyM2RoNHQxIQ==", + "email": "rguske@redhat.com" + }, + "cloud.openshift.com": { + +[...] +``` + +## Creating the image set configuration + +Create an `ImageSetConfiguration` YAML file and modify it to include your required images. + +List the available Operator using e.g.: + +```code +oc mirror list operators --catalogs --version=4.21 --v1 +``` + +```yaml +tee imagesetconfiguration.yaml > /dev/null <<'EOF' +kind: ImageSetConfiguration +apiVersion: mirror.openshift.io/v2alpha1 +mirror: + platform: + channels: + - name: stable-4.21 + type: ocp + shortestPath: true + minVersion: 4.21.10 + maxVersion: 4.21.11 + graph: true + operators: + - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.21 + packages: +## cincinnati-operator:v1 + - name: cincinnati-operator + channels: + - name: v1 + minVersion: '5.0.3' + # maxVersion: '5.0.3' +## kubernetes-nmstate-operator:stable + - name: kubernetes-nmstate-operator + channels: + - name: 'stable' + minVersion: '4.21.0-202604080925' + # maxVersion: '4.17.0-202502120148' +## kubevirt-hyperconverged:stable + - name: kubevirt-hyperconverged + channels: + - name: stable + minVersion: '4.21.3' + # maxVersion: '4.17.4' +## metallb-operator:stable + - name: metallb-operator + channels: + - name: stable + minVersion: '4.21.0-202604140043' + # maxVersion: 'v4.17.0' +## web-terminal:fast + - name: web-terminal + channels: + - name: fast + minVersion: '1.16.0' + # maxVersion: 'v1.15.0' +## web-terminal relies on devworkspace-operator +# - name: devworkspace-operator +# channels: +# - name: fast +# minVersion: '0.40-1776457293' +## OpenShift Data Foundation + - name: odf-operator + channels: + - name: stable-4.21 + minVersion: '4.21.2-rhodf' +## OpenShift Local Storage Operator + - name: local-storage-operator + channels: + - name: stable + minVersion: '4.21.0-202604200440' + additionalImages: + - name: registry.redhat.io/ubi8/ubi:latest + - name: registry.redhat.io/rhel9/rhel-guest-image:latest + - name: quay.io/rhn_support_sreber/curl:latest + # Important for KMM & GPFS Build + - name: registry.redhat.io/ubi9/ubi-minimal:latest + - name: registry.redhat.io/ubi9/ubi@sha256:20f695d2a91352d4eaa25107535126727b5945bff38ed36a3e59590f495046f0 + - name: quay.io/rguske/vddk@sha256:26d07e11f7f8dcca263e83a1d942fe9274c90418c5bfc17fad88b61ddabf95ed + - name: quay.io/rguske/simple-web-app@sha256:f1c474d0b214975d2fb95d14967b620daa0cdbef094ee509fec1659d55c3a6de + # Virtualization Images + - name: quay.io/containerdisks/centos-stream:9 + - name: quay.io/containerdisks/fedora:latest +EOF +``` + +Ensure that clis are in your $PATH. Otherwise `export PATH=/usr/local/bin:$PATH`. + +The oc-mirror plugin v2 automatically generates the following custom resources: + +- `ImageDigestMirrorSet` (IDMS): Handles registry mirror rules when using image digest pull specifications. Generated if at least one image of the image set is mirrored by digest. +- `ImageTagMirrorSet` (ITMS): Handles registry mirror rules when using image tag pull specifications. Generated if at least one image from the image set is mirrored by tag. +- `CatalogSource`: Retrieves information about the available Operators in the mirror registry. Used by Operator Lifecycle Manager (OLM) Classic. +- `ClusterCatalog`: Retrieves information about the available cluster extensions (which includes Operators) in the mirror registry. Used by OLM v1. +- `UpdateService`: Provides update graph data to the disconnected environment. Used by the OpenShift Update Service. + +Mirror the images from the specified image set configuration to the disk by running the following command: + +```code +oc mirror -c $(pwd)/openshift/imagesetconfiguration.yaml file://$(pwd) --v2 +``` + +```code +oc-mirror -c $(pwd)/openshift/imagesetconfiguration.yaml file:///home/rguske/openshift/mirror --v2 +``` + +Result: + +```code +2026/04/23 12:55:55 [INFO] : === Results === +2026/04/23 12:55:55 [INFO] : ✓ 193 / 193 release images mirrored successfully +2026/04/23 12:55:55 [INFO] : ✓ 95 / 95 operator images mirrored successfully +2026/04/23 12:55:55 [INFO] : ✓ 7 / 7 additional images mirrored successfully +2026/04/23 12:55:55 [INFO] : 📦 Preparing the tarball archive... +2026/04/23 12:58:48 [INFO] : mirror time : 19m53.229903012s +2026/04/23 12:58:48 [INFO] : 👋 Goodbye, thank you for using oc-mirror +``` + +```code +ls -ltr +total 61212944 +drwxr-xr-x. 12 rguske rguske 4096 Apr 23 12:39 working-dir +-rw-r--r--. 1 rguske rguske 62682043392 Apr 23 12:58 mirror_000001.tar +``` + +Upload to your mirror registry: + +```code +oc mirror -c $(pwd)/imagesetconfiguration.yaml \ +--from file://$(pwd)/mirror/ \ +docker://rguske-rhel9-disco-bastion.disco.local:8443/disco \ +--v2 +``` + +Results: + +```code +2026/04/24 11:34:43 [INFO] : === Results === +2026/04/24 11:34:43 [INFO] : ✓ 193 / 193 release images mirrored successfully +2026/04/24 11:34:43 [INFO] : ✓ 95 / 95 operator images mirrored successfully +2026/04/24 11:34:43 [INFO] : ✓ 7 / 7 additional images mirrored successfully +2026/04/24 11:34:43 [INFO] : 📄 Generating IDMS file... +2026/04/24 11:34:43 [INFO] : /home/rguske/openshift/mirror/working-dir/cluster-resources/idms-oc-mirror.yaml file created +2026/04/24 11:34:43 [INFO] : 📄 Generating ITMS file... +2026/04/24 11:34:43 [INFO] : /home/rguske/openshift/mirror/working-dir/cluster-resources/itms-oc-mirror.yaml file created +2026/04/24 11:34:43 [INFO] : 📄 Generating CatalogSource file... +2026/04/24 11:34:43 [INFO] : /home/rguske/openshift/mirror/working-dir/cluster-resources/cs-redhat-operator-index-v4-21.yaml file created +2026/04/24 11:34:43 [INFO] : 📄 Generating ClusterCatalog file... +2026/04/24 11:34:43 [INFO] : /home/rguske/openshift/mirror/working-dir/cluster-resources/cc-redhat-operator-index-v4-21.yaml file created +2026/04/24 11:34:43 [INFO] : 📄 Generating Signature Configmap... +2026/04/24 11:34:43 [INFO] : /home/rguske/openshift/mirror/working-dir/cluster-resources/signature-configmap.json file created +2026/04/24 11:34:43 [INFO] : /home/rguske/openshift/mirror/working-dir/cluster-resources/signature-configmap.yaml file created +2026/04/24 11:34:43 [INFO] : 📄 Generating UpdateService file... +2026/04/24 11:34:43 [INFO] : /home/rguske/openshift/mirror/working-dir/cluster-resources/updateService.yaml file created +2026/04/24 11:34:43 [INFO] : mirror time : 50m26.676667165s +2026/04/24 11:34:43 [INFO] : 👋 Goodbye, thank you for using oc-mirror +``` + +NOTE! The necessary manifests for your `install-config.yaml` are generated by the `oc-mirror` tool: + +> Generating IDMS file... +2026/04/24 11:34:43 [INFO] : /home/rguske/openshift/mirror/working-dir/cluster-resources/ + +### Unexpected status code 413 Request Entity Too Large + +If this hits you, try using the following options: + +```code +oc mirror -c $(pwd)/imagesetconfiguration.yaml --from file://$(pwd)/mirror/ docker://rguske-rhel9-disco-bastion. +disco.local:8443/disco --image-timeout 2h --parallel-images=10 --parallel-layers=10 --retry-times=5 --retry-delay=10s --v2 +``` + +Documented in [oc-mirror v2 fails with context deadline exceeded when mirroring large images to a local registry in RHOCP 4](https://access.redhat.com/solutions/7130341). + +## Installing a disconnected Cluster using the Agent Based Installer + +> When you use a disconnected mirror registry, you must add the certificate file that you created previously for your mirror registry to the additionalTrustBundle field of the install-config.yaml file. + +Workflow: + +- Create mirror registry content (oc mirror) ✅ +- Create installation assets (install-config.yaml) +- Create the cluster +- Apply mirror configuration to the new cluster + +### Cluster Preperations + +Network 192.168.69.0/24 +DNS: 192.168.69.6 +GW: 192.168.69.254 +VLAN ID 69 + +Collecting the necessary nic information: + +| name | nic | mac | ipv4 | comment | +|---|---|---|---|---| +| rguske-ocp42-disco-1 | enp1s0 | 02:d8:6d:0f:3e:dc | 192.168.69.202 | Node 1 | +| rguske-ocp42-disco-2 | enp1s0 | 02:d8:6d:0f:3e:dd | 192.168.69.203 | Node 2 | +| rguske-ocp42-disco-3 | enp1s0 | 02:d8:6d:0f:3e:de | 192.168.69.204 | Node 3 | + +BaseDomain: disco.local + +## Configurations + +Create the `agent-config.yaml as well as the install-config.yaml` + +```code +tree rguske-ocp42-disco/ +rguske-ocp42-disco/ +└── conf + ├── agent-config.yaml + └── install-config.yaml +``` + +```yaml +cat > agent-config.yaml << EOF +apiVersion: v1beta1 +kind: AgentConfig +metadata: + name: rguske-ocp42-disco +rendezvousIP: 192.168.69.202 +hosts: + - hostname: rguske-ocp42-disco-1.disco.local + role: master + interfaces: + - name: enp1s0 + macAddress: 02:d8:6d:0f:3e:dc + networkConfig: + interfaces: + - name: enp1s0 + type: ethernet + state: up + mac-address: 02:d8:6d:0f:3e:dc + ipv4: + enabled: true + address: + - ip: 192.168.69.202 + prefix-length: 24 + dhcp: false + dns-resolver: + config: + server: + - 192.168.69.6 + routes: + config: + - destination: 0.0.0.0/0 + next-hop-address: 192.168.69.254 + next-hop-interface: enp1s0 + table-id: 254 + - hostname: rguske-ocp42-disco-2.disco.local + role: master + interfaces: + - name: enp1s0 + macAddress: 02:d8:6d:0f:3e:dd + networkConfig: + interfaces: + - name: enp1s0 + type: ethernet + state: up + mac-address: 02:d8:6d:0f:3e:dd + ipv4: + enabled: true + address: + - ip: 192.168.69.203 + prefix-length: 24 + dhcp: false + dns-resolver: + config: + server: + - 192.168.69.6 + routes: + config: + - destination: 0.0.0.0/0 + next-hop-address: 192.168.69.254 + next-hop-interface: enp1s0 + table-id: 254 + - hostname: rguske-ocp42-disco-3.disco.local + role: master + interfaces: + - name: enp1s0 + macAddress: 02:d8:6d:0f:3e:de + networkConfig: + interfaces: + - name: enp1s0 + type: ethernet + state: up + mac-address: 02:d8:6d:0f:3e:de + ipv4: + enabled: true + address: + - ip: 192.168.69.204 + prefix-length: 24 + dhcp: false + dns-resolver: + config: + server: + - 192.168.69.6 + routes: + config: + - destination: 0.0.0.0/0 + next-hop-address: 192.168.69.254 + next-hop-interface: enp1s0 + table-id: 254 +EOF +``` + +The ssl certificate of the mirror-registry which will be used in the `install-config.yaml` can be found within the `mirror-registry/root/quay-rootCA` folder. Example: `/home/rguske/downloads/mirror-registry/root/quay-rootCA` + +```json +{ + "auths": { + "rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com:8443": { + "auth": "aW5pdDpyM2RoNHQxIQ==", + "email": "rguske@redhat.com" + } + } +} +``` + +Remember! The IDS config is stored here: + +```code +Generating IDMS file... +2026/04/24 11:34:43 [INFO] : /home/rguske/openshift/mirror/working-dir/cluster-resources/ +``` + +```yaml +cat > install-config.yaml << EOF +apiVersion: v1 +baseDomain: disco.local +ImageDigestSources: +- mirrors: + - rguske-rhel9-disco-bastion.disco.local:8443/disco/openshift/release + source: quay.io/openshift-release-dev/ocp-v4.0-art-dev +- mirrors: + - rguske-rhel9-disco-bastion.disco.local:8443/disco/openshift/release-images + source: quay.io/openshift-release-dev/ocp-release +additionalTrustBundle: | + -----BEGIN CERTIFICATE----- + MIIEHDCCAwSgAwIBAgIUFY/Z+WmgJ+8SIREIa3Cl3FRj9jYwDQYJKoZIhvcNAQEL + BQAwgYAxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJWQTERMA8GA1UEBwwITmV3IFlv + cmsxDTALBgNVBAoMBFF1YXkxETAPBgNVBAsMCERpdmlzaW9uMS8wLQYDVQQDDCZy + Z3Vza2UtcmhlbDktZGlzY28tYmFzdGlvbi5kaXNjby5sb2NhbDAeFw0yNjA0MjQx + NDE3MTZaFw0yOTAyMTExNDE3MTZaMIGAMQswCQYDVQQGEwJVUzELMAkGA1UECAwC +... + ZyYMLJyR83M5sD7sVbbuSOkYgNt20ZdIcqigIkyABRkqcahC7kOypXXJbhkj3fYL + kyGukgbJRF96hCB9oO8bW3evact/P40arsjHT6qKRIZf0kKm7CYUVRjI4+jlz+oV + n7iB3Rs8P16UvuFB2LfWmyNfuu21InZhXLmJ+rZJc0qnpq6Rm8iXAq0n8L5ycCHc + gPt4JJQZJ8JP6bSREgAhqfNSngfLj73O1+S2fuN7i3mCQEv0UajEhQgHQtcZ6r1C + -----END CERTIFICATE----- +compute: +- name: worker + replicas: 0 +controlPlane: + name: master + replicas: 3 +metadata: + name: rguske-ocp42-disco +networking: + clusterNetwork: + - cidr: 10.128.0.0/14 + hostPrefix: 23 + machineNetwork: + - cidr: 192.168.69.0/24 + serviceNetwork: + - 172.30.0.0/16 + networkType: OVNKubernetes +platform: + baremetal: + apiVIPs: + - 192.168.69.200 + ingressVIPs: + - 192.168.69.201 +fips: false +pullSecret: '{ + "auths": { + "rguske-rhel9-disco-bastion.disco.local:8443": { + "auth": "aW5pdDpyM2RoNHQxIQ==", + "email": "rguske@redhat.com" + } + } +}' +sshKey: 'ssh-ed25519 AAAAC3NzaC1lZ...VzGQ/Ur5Ek0v9gF rguske@rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com' +EOF +``` + +Create a new `openshift-install-fips` binary which only points to your mirror-registry. + +```code +export LOCAL_SECRET_JSON='/home/rguske/openshift/new-pull-secret.json' \ +export LOCAL_REGISTRY='rguske-rhel9-disco-bastion.disco.local:8443' \ +export LOCAL_REPOSITORY='disco/openshift/release-images' \ +export OCP_RELEASE='4.21.10' \ +export ARCHITECTURE='x86_64' +``` + +```code +oc adm release extract -a ${LOCAL_SECRET_JSON} --idms-file=/home/rguske/openshift/mirror/working-dir/cluster-resources/idms-oc-mirror.yaml --command=openshift-install-fips "${LOCAL_REGISTRY}/${LOCAL_REPOSITORY}:${OCP_RELEASE}-${ARCHITECTURE}" +``` + +```code +tree -L 1 +. +├── downloads +├── oc-mirror-web-app +├── openshift +└── openshift-install-fips +``` + +Validate it: + +```code +./openshift-install-fips version +./openshift-install-fips 4.21.10 +built from commit 6285755d199e7aa7bf29db5fe6964ce7f3684ed9 +release image rguske-rhel9-disco-bastion.disco.local:8443/disco/openshift/release-images@sha256:5d591a70c92a6dfa3b6b948ffe5e5eac7ab339c49005744006aa0dd9d6d98898 +Release Image Architecture is unknown +release architecture unknown +default architecture amd64 +``` + +Note the `release image` reference! This binary will be used for the creation of the agent.iso. + +## Create Agent iso + +Create the install-config.yaml and agent-install.yaml file. + +Run `./openshift-install-fips agent create image --dir /home/rguske/openshift/rguske-ocp42-disco/conf/` + +Example output : + +```code +INFO Configuration has 3 master replicas, 0 arbiter replicas, and 0 worker replicas +WARNING The imageDigestSources configuration in install-config.yaml should have at least one source field matching the releaseImage value rguske-rhel9-disco-bastion.disco.local:8443/disco/openshift/release-images@sha256:5d591a70c92a6dfa3b6b948ffe5e5eac7ab339c49005744006aa0dd9d6d98898 +INFO The rendezvous host IP (node0 IP) is 192.168.69.202 +INFO Extracting base ISO from release payload +INFO Base ISO obtained from release and cached at [/home/rguske/.cache/agent/image_cache/coreos-x86_64.iso] +INFO Consuming Install Config from target directory +INFO Consuming Agent Config from target directory +INFO Generated ISO at /home/rguske/openshift/rguske-ocp42-disco/conf/agent.x86_64.iso. +``` + +If you still have problemes with the release-image reference use: + +```code +export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE='rguske-rhel9-disco-bastion.rguske.coe.muc.redhat.com:8443/disco/openshift/release-images@sha256:5d591a70c92a6dfa3b6b948ffe5e5eac7ab339c49005744006aa0dd9d6d98898' +``` + +Mount the `agent.x86_64.iso` on the machines (BM or VM). + +--- + +If you're running the bastion host on e.g. OpenShift Virtualization, you can use the following command in order to upload the ise to a pvc: + +```code +VERSION=$(curl -s https://api.github.com/repos/kubevirt/kubevirt/releases/latest | grep tag_name | cut -d '"' -f 4) +curl -L -o virtctl https://github.com/kubevirt/kubevirt/releases/download/$VERSION/virtctl-$VERSION-linux-amd64 +chmod +x virtctl +sudo mv virtctl /usr/local/bin/ +``` + +Login into your existing OpenShift cluster, change into your project and upload the iso: + +```code +virtctl image-upload pvc rguskeagentiso-disco --size 2Gi --storage-class coe-netapp-nas --access-mode ReadWriteMany --image-path ~/openshift/rguske-ocp42-disco/conf/agent.x86_64.iso --insecure +``` + +--- + +Boot the machines and wait until the installation is done. + +Validate the installer progress using `./openshift-install-fips wait-for install-complete --dir /home/rguske/openshift/rguske-ocp42-disco/conf/ --log-level=debug` + +## Run a `httpd` webserver on the bastion to share the iso + +Depending on your environment, providing the created iso can be cumbersome. + +One quick and easy way could be by making it downloadable via a webserver. + +Install `httpd` on the bastion host. + +```bash +dnf install httpd +sudo systemctl enable --now httpd +sudo firewall-cmd --permanent --add-service=http +sudo firewall-cmd --reload +``` + +Validate the service is running: + +```bash +sudo ss -tuln | grep :80 +curl -I http://localhost +sudo tail -f /var/log/httpd/error_log +``` + +Copy (`scp agent.x86_64.iso root@192.168.69.208:/root/download/`) the agent.iso from the mirror registry to the bastion host which has access to the target (ESXi server for example). + +Copy the created iso into `/var/www/html/` on the bastion host. + +Download the iso by using e.g. `wget http:///agent.x86_64.iso`. + +Example: + +```code +wget http://bastion-rguske.rguske.coe.muc.redhat.com/agent.x86_64.iso +Connecting to bastion-rguske.rguske.coe.muc.redhat.com (10.32.96.138:80) +saving to 'agent.x86_64.iso' +agent.x86_64.iso 21% ******************************************** | 261M 0:00:10 ETA +``` + +## Post Cluster Installation + +After the cluster installation is done, apply the by `oc-mirror ... --from file://` created manifests to the cluster: + +```code +oc apply -f /home/rguske/openshift/mirror/working-dir/cluster-resources/ . +``` + +## Using Operator Lifecycle Manager in disconnected environments + +Docs: [Using Operator Lifecycle Manager in disconnected environments](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/disconnected_environments/olm-restricted-networks) + +Disable the `CatalogSources` for the default catalogs by adding `disableAllDefaultSources`: true to the OperatorHub object: + +```code +oc patch OperatorHub cluster --type json \ + -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]' +``` + +The new `CatalogSource` manifest was create by `oc-mirror` in the working-dir: (/home/rguske/openshift/mirror/working-dir/cluster-resources/) + +Example: + +```yaml +oc create -f - <

kubevirt-hyperconverged

stable

replace

replace

replace

replace

kubevirt-hyperconverged-operator.v4.17.4

kubevirt-hyperconverged-operator.v4.17.3

kubevirt-hyperconverged-operator.v4.17.5

kubevirt-hyperconverged-operator.v4.17.7

kubevirt-hyperconverged-operator.v4.17.11

 \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 688dc6c3..369ab03b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -119,7 +119,9 @@ extra_css: nav: - Cluster installation: - cluster-installation/index.md - - Air-gapped: cluster-installation/air-gapped.md + - Air-gapped/Disconnected: + - "Cluster Installation": cluster-installation/disconnected/disconnected-installation.md + - "Operator Installation": cluster-installation/disconnected/disconnected-operators.md - SNO on OCP-V: cluster-installation/sno-on-ocpv/index.md - Agent-based (proxy): cluster-installation/agent-base-proxy/index.md - VMware/vSphere: