diff --git a/.add_worker_implementation_summary.md b/.add_worker_implementation_summary.md new file mode 100644 index 000000000..eaa54804c --- /dev/null +++ b/.add_worker_implementation_summary.md @@ -0,0 +1,234 @@ +# Implementation Summary: Post-Installation Worker Node Addition + +## What Was Created + +### Core Scripts + +1. **`add_worker_node.sh`** - Main script to add worker nodes + - Creates libvirt VM with customizable resources + - Configures virtual BMC (vbmc or sushy-tools) + - Generates BareMetalHost and Secret manifests + - Provides step-by-step instructions for completion + - Automatically finds available BMC ports and generates unique MAC addresses + +2. **`remove_worker_node.sh`** - Cleanup script to remove worker nodes + - Drains and deletes node from cluster + - Removes BareMetalHost and Machine resources + - Destroys VM and cleans up disk/NVRAM + - Removes BMC configuration + +3. **`auto_approve_csrs.sh`** - Helper script for CSR approval + - Auto-approves pending CSRs for specified duration + - Useful for development/testing scenarios + - Default 30-minute duration + +### Documentation + +1. **`docs/add-worker-post-install.md`** - Complete documentation + - Detailed usage instructions + - Configuration options + - Complete workflow examples + - Troubleshooting guide + - Architecture notes + +2. **`WORKER_QUICK_START.md`** - Quick reference guide + - TL;DR commands + - Common use cases + - Quick examples + +3. **`README.md`** - Updated main README + - Added new "Option 1: Add Workers Post-Installation" + - Kept existing pre-configuration method as "Option 2" + - Links to new documentation + +### Makefile Integration + +Added two new targets to `Makefile`: +- `make add_worker WORKER_NAME=` - Add a worker +- `make remove_worker WORKER_NAME=` - Remove a worker + +## Key Features + +### 1. No Pre-Planning Required +Unlike the existing methods, this solution allows adding workers **after** deployment without requiring `NUM_EXTRA_WORKERS` to be set beforehand. + +### 2. Flexible Configuration +Users can customize worker resources via environment variables: +```bash +export EXTRA_WORKER_MEMORY=32768 # 32GB +export EXTRA_WORKER_DISK=100 # 100GB +export EXTRA_WORKER_VCPU=16 # 16 cores +``` + +### 3. Smart Automation +- Automatically finds available BMC ports +- Generates unique MAC addresses +- Detects and starts BMC containers if needed +- Supports both IPMI and Redfish BMC protocols +- Handles UEFI/BIOS firmware automatically + +### 4. Complete Lifecycle Management +- Add workers: `add_worker_node.sh` +- Remove workers: `remove_worker_node.sh` +- Auto-approve CSRs: `auto_approve_csrs.sh` + +### 5. Safety Checks +- Validates cluster connectivity +- Checks for VM name conflicts +- Checks for BareMetalHost conflicts +- Validates worker name format + +## Usage Comparison + +### Old Method (Pre-Configuration Required) +```bash +# BEFORE initial deployment +export NUM_EXTRA_WORKERS=2 +export EXTRA_WORKERS_ONLINE_STATUS=false +make + +# AFTER deployment +oc apply -f ocp/ostest/extra_host_manifests.yaml +oc scale machineset ostest-worker-0 --replicas=3 -n openshift-machine-api +``` + +### New Method (Post-Installation) +```bash +# Deploy cluster normally +make + +# LATER: Add worker on demand +./add_worker_node.sh my-worker-1 +oc apply -f ocp/ostest/my-worker-1_bmh.yaml +./auto_approve_csrs.sh 30 & +oc scale machineset ostest-worker-0 --replicas=3 -n openshift-machine-api +``` + +## Architecture + +### VM Creation +- Uses libvirt XML to define VM +- Supports x86_64 and aarch64 architectures +- Configures UEFI or BIOS boot +- Connects to baremetal network +- Allocates disk storage in `/var/lib/libvirt/images/` + +### BMC Configuration +Two protocols supported: + +1. **IPMI** (via vbmc) + - Port range: 6230-6250 + - Protocol: `ipmi://:` + - Container: vbmc + +2. **Redfish** (via sushy-tools) + - Port: 8000 (HTTP) + - Protocol: `redfish-virtualmedia+http://:8000/` + - Container: sushy-tools + - Default choice + +### BareMetalHost Integration +Generated manifest includes: +- Secret with BMC credentials (admin/password) +- BareMetalHost spec with: + - BMC address and credentials + - Boot MAC address + - Online status (true) + - Automated cleaning mode (disabled) + +## Testing + +All scripts pass bash syntax validation: +```bash +bash -n add_worker_node.sh # ✓ PASS +bash -n remove_worker_node.sh # ✓ PASS +bash -n auto_approve_csrs.sh # ✓ PASS +``` + +## Files Modified/Created + +### New Files +- `add_worker_node.sh` (executable) +- `remove_worker_node.sh` (executable) +- `auto_approve_csrs.sh` (executable) +- `docs/add-worker-post-install.md` +- `WORKER_QUICK_START.md` +- `.add_worker_implementation_summary.md` (this file) + +### Modified Files +- `Makefile` - Added `add_worker` and `remove_worker` targets +- `README.md` - Updated "Testing with extra workers" section + +## Dependencies + +The scripts leverage existing dev-scripts infrastructure: +- `common.sh` - Environment variables and common functions +- `network.sh` - Network configuration +- `utils.sh` - Utility functions +- `ocp_install_env.sh` - OCP environment setup +- `logging.sh` - Logging functions + +## Compatibility + +- Works with standard installer flow (`make` or `make all`) +- Compatible with existing `NUM_EXTRA_WORKERS` workflow +- Supports both IPMI and Redfish BMC protocols +- Works with UEFI and BIOS boot modes +- Supports x86_64 and aarch64 architectures + +## Next Steps for Users + +1. **Basic Usage** + ```bash + ./add_worker_node.sh worker-1 + oc apply -f ocp/ostest/worker-1_bmh.yaml + oc scale machineset --replicas= -n openshift-machine-api + ``` + +2. **With Custom Resources** + ```bash + export EXTRA_WORKER_MEMORY=32768 EXTRA_WORKER_DISK=100 EXTRA_WORKER_VCPU=16 + ./add_worker_node.sh large-worker + ``` + +3. **Quick Start with Make** + ```bash + make add_worker WORKER_NAME=worker-1 + ``` + +4. **Auto-approve CSRs** + ```bash + ./auto_approve_csrs.sh 30 & + ``` + +## Limitations + +1. Only works with libvirt-based deployments +2. Requires BMC containers (vbmc/sushy-tools) to be available +3. BMC port range limited to available ports in 6230-6250 +4. Worker must be on same network as other cluster nodes +5. Resources (memory, disk, CPU) are set at VM creation time + +## Future Enhancements + +Possible improvements for future versions: +- Support for multiple workers in one command +- Interactive mode with prompts +- Integration with Ansible playbooks +- Support for additional disk attachment +- Network configuration customization +- BMC port range expansion +- Support for non-libvirt platforms + +## Support + +For issues or questions: +1. Check the documentation: `docs/add-worker-post-install.md` +2. Review quick start: `WORKER_QUICK_START.md` +3. Check script output for detailed instructions +4. Verify cluster connectivity and resources + +## Conclusion + +This implementation provides a complete, flexible solution for adding worker nodes to dev-scripts clusters post-installation, eliminating the need for pre-planning and making it easier to scale clusters on demand for testing and development purposes. + diff --git a/Makefile b/Makefile index 89cc76dc2..ae6e6a117 100644 --- a/Makefile +++ b/Makefile @@ -66,6 +66,12 @@ install_config: ocp_run: ./06_create_cluster.sh +add_worker: + ./add_worker_node.sh $(WORKER_NAME) + +remove_worker: + ./remove_worker_node.sh $(WORKER_NAME) + gather: ./must_gather.sh diff --git a/README.md b/README.md index dbe231ff5..ad54dd703 100644 --- a/README.md +++ b/README.md @@ -495,7 +495,31 @@ export IGNITION_EXTRA="ignition/file_example.ign" ### Testing with extra workers -It is possible to specify additional workers, which are not used in the initial +#### Option 1: Add Workers Post-Installation (No Pre-Planning Required) + +You can add worker nodes **after** your cluster is deployed without requiring pre-configuration: + +```bash +# Add a worker node +./add_worker_node.sh my-worker-1 + +# Apply the generated manifest +oc apply -f ocp/ostest/my-worker-1_bmh.yaml + +# Scale the machineset to provision it +oc scale machineset -worker-0 --replicas= -n openshift-machine-api +``` + +Or using Make: +```bash +make add_worker WORKER_NAME=my-worker-1 +``` + +See [WORKER_QUICK_START.md](WORKER_QUICK_START.md) for a quick guide or [docs/add-worker-post-install.md](docs/add-worker-post-install.md) for complete documentation. + +#### Option 2: Pre-Configure Extra Workers (Traditional Method) + +You can specify additional workers during initial deployment, which are not used in the initial deployment, and can then later be used e.g to test scale-out. The default online status of the extra workers is true, but can be changed to false using EXTRA_WORKERS_ONLINE_STATUS. diff --git a/WORKER_QUICK_START.md b/WORKER_QUICK_START.md new file mode 100644 index 000000000..5edec6eae --- /dev/null +++ b/WORKER_QUICK_START.md @@ -0,0 +1,66 @@ +# Quick Start: Adding Workers Post-Installation + +## TL;DR + +Add a worker node after your cluster is already deployed: + +```bash +# Add a worker +./add_worker_node.sh my-worker-1 + +# Apply the generated manifest +oc apply -f ocp/ostest/my-worker-1_bmh.yaml + +# Auto-approve CSRs in background +./auto_approve_csrs.sh 30 & + +# Scale up your machineset +oc get machineset -n openshift-machine-api +oc scale machineset -worker-0 --replicas= -n openshift-machine-api + +# Watch it join +oc get nodes -w +``` + +## Customizing Resources + +```bash +export EXTRA_WORKER_MEMORY=32768 # 32GB RAM +export EXTRA_WORKER_DISK=100 # 100GB disk +export EXTRA_WORKER_VCPU=16 # 16 vCPUs + +./add_worker_node.sh my-large-worker +``` + +## Using Make + +```bash +# Add worker +make add_worker WORKER_NAME=worker-1 + +# Remove worker +make remove_worker WORKER_NAME=worker-1 +``` + +## What Gets Created + +- ✅ Libvirt VM with specified resources +- ✅ Virtual BMC (IPMI or Redfish) +- ✅ BareMetalHost manifest +- ✅ Secret for BMC credentials +- ✅ Complete setup instructions + +## Removing a Worker + +```bash +# Remove from cluster and delete VM +./remove_worker_node.sh my-worker-1 + +# Don't forget to scale down the machineset +oc scale machineset -worker-0 --replicas= -n openshift-machine-api +``` + +## Full Documentation + +See [docs/add-worker-post-install.md](docs/add-worker-post-install.md) for detailed documentation. + diff --git a/add_worker_node.sh b/add_worker_node.sh new file mode 100755 index 000000000..dfeb6a959 --- /dev/null +++ b/add_worker_node.sh @@ -0,0 +1,306 @@ +#!/bin/bash +# +# Add a worker node to an existing cluster after deployment +# +# Usage: ./add_worker_node.sh [worker_name] +# +# This script: +# 1. Creates a new libvirt VM +# 2. Configures virtual BMC for the VM +# 3. Generates and applies BareMetalHost manifest +# 4. Provides instructions for scaling the machineset +# + +set -euo pipefail + +SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" + +source ${SCRIPTDIR}/logging.sh +source ${SCRIPTDIR}/common.sh +source ${SCRIPTDIR}/network.sh +source ${SCRIPTDIR}/utils.sh +source ${SCRIPTDIR}/ocp_install_env.sh + +# Parse arguments +WORKER_NAME=${1:-"extraworker-0"} + +# Validate worker name format +if [[ ! "$WORKER_NAME" =~ ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ ]]; then + echo "Error: Worker name must be lowercase alphanumeric with hyphens" + exit 1 +fi + +# Check if cluster is running +if ! oc get nodes &>/dev/null; then + echo "Error: Cannot connect to cluster. Ensure your cluster is running and KUBECONFIG is set." + exit 1 +fi + +# Check if worker VM already exists +if sudo virsh list --all | grep -q "${CLUSTER_NAME}_${WORKER_NAME}"; then + echo "Error: VM ${CLUSTER_NAME}_${WORKER_NAME} already exists" + exit 1 +fi + +# Check if BareMetalHost already exists +if oc get baremetalhost -n openshift-machine-api "${CLUSTER_NAME}-${WORKER_NAME}" &>/dev/null; then + echo "Error: BareMetalHost ${CLUSTER_NAME}-${WORKER_NAME} already exists" + exit 1 +fi + +echo "==========================================" +echo "Adding worker node: ${WORKER_NAME}" +echo "Cluster: ${CLUSTER_NAME}" +echo "==========================================" + +# Generate MAC address +WORKER_MAC=$(printf '52:54:00:%02x:%02x:%02x' $((RANDOM%256)) $((RANDOM%256)) $((RANDOM%256))) +echo "Generated MAC address: ${WORKER_MAC}" + +# Find available BMC port +VBMC_BASE_PORT=${VBMC_BASE_PORT:-6230} +VBMC_MAX_PORT=${VBMC_MAX_PORT:-6250} +BMC_PORT="" + +for port in $(seq $VBMC_BASE_PORT $VBMC_MAX_PORT); do + if ! sudo netstat -tuln | grep -q ":${port} "; then + BMC_PORT=$port + break + fi +done + +if [ -z "$BMC_PORT" ]; then + echo "Error: No available BMC ports in range ${VBMC_BASE_PORT}-${VBMC_MAX_PORT}" + exit 1 +fi + +echo "Using BMC port: ${BMC_PORT}" + +# Set worker resources +EXTRA_WORKER_MEMORY=${EXTRA_WORKER_MEMORY:-16384} +EXTRA_WORKER_DISK=${EXTRA_WORKER_DISK:-50} +EXTRA_WORKER_VCPU=${EXTRA_WORKER_VCPU:-8} + +# Determine BMC driver and protocol +BMC_DRIVER=${BMC_DRIVER:-"redfish-virtualmedia"} +if [[ "$BMC_DRIVER" == "ipmi" ]]; then + BMC_PROTOCOL="ipmi" + BMC_ADDRESS="${PROVISIONING_HOST_EXTERNAL_IP}:${BMC_PORT}" + BMC_FULL_ADDRESS="${BMC_PROTOCOL}://${BMC_ADDRESS}" +elif [[ "$BMC_DRIVER" =~ "redfish" ]]; then + BMC_PROTOCOL="redfish-virtualmedia" + BMC_ADDRESS="${PROVISIONING_HOST_EXTERNAL_IP}:8000/${WORKER_NAME}" + BMC_FULL_ADDRESS="${BMC_PROTOCOL}+http://${BMC_ADDRESS}" +else + echo "Error: Unsupported BMC driver: ${BMC_DRIVER}" + exit 1 +fi + +echo "BMC Address: ${BMC_FULL_ADDRESS}" + +# Create VM disk +DISK_PATH="/var/lib/libvirt/images/${CLUSTER_NAME}_${WORKER_NAME}.qcow2" +echo "Creating disk at ${DISK_PATH} (${EXTRA_WORKER_DISK}G)..." +sudo qemu-img create -f qcow2 "${DISK_PATH}" "${EXTRA_WORKER_DISK}G" + +# Get the baremetal network details +BAREMETAL_NETWORK=${BAREMETAL_NETWORK_NAME:-"baremetal"} + +# Determine firmware and NVRAM paths based on architecture +ARCH=$(uname -m) +case $ARCH in + x86_64) + if [ "${LIBVIRT_FIRMWARE:-uefi}" == "uefi" ]; then + FIRMWARE_PATH="/usr/share/OVMF/OVMF_CODE.fd" + NVRAM_PATH="/var/lib/libvirt/qemu/nvram/${CLUSTER_NAME}_${WORKER_NAME}_VARS.fd" + LOADER_TYPE="pflash" + fi + ;; + aarch64) + if [ "${LIBVIRT_FIRMWARE:-uefi}" == "uefi" ]; then + FIRMWARE_PATH="/usr/share/AAVMF/AAVMF_CODE.fd" + NVRAM_PATH="/var/lib/libvirt/qemu/nvram/${CLUSTER_NAME}_${WORKER_NAME}_VARS.fd" + LOADER_TYPE="pflash" + fi + ;; +esac + +# Create VM definition +echo "Creating VM ${CLUSTER_NAME}_${WORKER_NAME}..." + +VM_XML=$(cat < + ${CLUSTER_NAME}_${WORKER_NAME} + ${EXTRA_WORKER_MEMORY} + ${EXTRA_WORKER_VCPU} + + hvm +EOF +) + +if [ "${LIBVIRT_FIRMWARE:-uefi}" == "uefi" ]; then + VM_XML+=$(cat <${FIRMWARE_PATH} + ${NVRAM_PATH} +EOF +) +fi + +VM_XML+=$(cat < + + + + + + + + + + destroy + restart + destroy + + /usr/libexec/qemu-kvm + + + + + + + + + + + + + + + + + + /dev/urandom + + + +EOF +) + +# Define the VM +echo "$VM_XML" | sudo virsh define /dev/stdin + +echo "VM created successfully" + +# Setup virtual BMC +echo "Configuring virtual BMC..." + +if [ "$NODES_PLATFORM" = "libvirt" ]; then + # Ensure vbmc/sushy-tools are running + if [ "$BMC_DRIVER" = "ipmi" ]; then + if ! is_running vbmc; then + echo "Starting vbmc container..." + sudo rm -f $WORKING_DIR/virtualbmc/vbmc/master.pid + sudo podman run -d --net host --privileged --name vbmc --pod ironic-pod \ + -v "$WORKING_DIR/virtualbmc/vbmc":/root/.vbmc -v "/root/.ssh":/root/ssh \ + "${VBMC_IMAGE}" + sleep 5 + fi + + # Add vbmc entry + echo "Adding VBMC entry for ${CLUSTER_NAME}_${WORKER_NAME}..." + sudo podman exec vbmc vbmc add "${CLUSTER_NAME}_${WORKER_NAME}" \ + --port "${BMC_PORT}" \ + --username "admin" \ + --password "password" \ + --libvirt-uri "qemu+ssh://root@${PROVISIONING_HOST_EXTERNAL_IP}/system?&keyfile=/root/ssh/id_rsa_virt_power&no_verify=1&no_tty=1" + sudo podman exec vbmc vbmc start "${CLUSTER_NAME}_${WORKER_NAME}" + + else # redfish + if ! is_running sushy-tools; then + echo "Starting sushy-tools container..." + sudo podman run -d --net host --privileged --name sushy-tools --pod ironic-pod \ + -v "$WORKING_DIR/virtualbmc/sushy-tools":/root/sushy -v "/root/.ssh":/root/ssh \ + "${SUSHY_TOOLS_IMAGE}" + sleep 5 + fi + echo "Sushy-tools will automatically detect the new VM" + fi +fi + +echo "Virtual BMC configured" + +# Generate BareMetalHost manifest +BMH_MANIFEST="${OCP_DIR}/${WORKER_NAME}_bmh.yaml" + +echo "Generating BareMetalHost manifest at ${BMH_MANIFEST}..." + +cat > "${BMH_MANIFEST}" < --replicas= -n openshift-machine-api" +echo "" +echo "4. Monitor the machine creation:" +echo " oc get machines -n openshift-machine-api -w" +echo "" +echo "5. Approve CSRs when they appear:" +echo " oc get csr" +echo " oc adm certificate approve " +echo "" +echo " Or approve all pending CSRs:" +echo " oc get csr -o name | xargs oc adm certificate approve" +echo "" +echo "==========================================" + diff --git a/auto_approve_csrs.sh b/auto_approve_csrs.sh new file mode 100755 index 000000000..9b683ce22 --- /dev/null +++ b/auto_approve_csrs.sh @@ -0,0 +1,37 @@ +#!/bin/bash +# +# Auto-approve pending CSRs for a specified duration +# +# Usage: ./auto_approve_csrs.sh [duration_minutes] +# +# Default duration is 30 minutes +# + +set -euo pipefail + +DURATION_MINUTES=${1:-30} +DURATION_SECONDS=$((DURATION_MINUTES * 60)) + +echo "Auto-approving CSRs for ${DURATION_MINUTES} minutes..." +echo "Press Ctrl+C to stop" + +timeout=$DURATION_SECONDS +elapsed=0 + +while (( elapsed < timeout )); do + # Get pending CSRs + pending_csrs=$(oc get csr 2>/dev/null | grep Pending | awk '{print $1}' || true) + + if [ -n "$pending_csrs" ]; then + echo "[$(date '+%Y-%m-%d %H:%M:%S')] Found pending CSRs:" + echo "$pending_csrs" + echo "$pending_csrs" | xargs -r oc adm certificate approve + echo "Approved!" + fi + + elapsed=$((elapsed + 10)) + sleep 10 +done + +echo "Auto-approval period complete" + diff --git a/docs/add-worker-post-install.md b/docs/add-worker-post-install.md new file mode 100644 index 000000000..aa165dd35 --- /dev/null +++ b/docs/add-worker-post-install.md @@ -0,0 +1,251 @@ +# Adding Worker Nodes Post-Installation + +This guide explains how to add worker nodes to your cluster **after** the initial deployment is complete using the standard installer flow. + +## Overview + +The `add_worker_node.sh` script allows you to dynamically add worker nodes to an existing cluster without requiring `NUM_EXTRA_WORKERS` to be set before initial deployment. + +The script handles: +- Creating a new libvirt VM with appropriate resources +- Configuring virtual BMC (vbmc or sushy-tools) +- Generating BareMetalHost and Secret manifests +- Providing step-by-step instructions for cluster integration + +## Prerequisites + +- Cluster must be deployed and accessible (`oc` commands work) +- Virtual BMC containers (vbmc or sushy-tools) should be running (automatically started if needed) +- Sufficient system resources for additional VMs + +## Usage + +### Adding a Worker Node + +#### Option 1: Using the script directly + +```bash +./add_worker_node.sh [worker_name] +``` + +If no name is provided, defaults to `extraworker-0`. + +**Example:** +```bash +./add_worker_node.sh my-worker-1 +``` + +#### Option 2: Using Make + +```bash +make add_worker WORKER_NAME=my-worker-1 +``` + +### Configuration + +You can customize the worker resources by setting environment variables before running the script: + +```bash +export EXTRA_WORKER_MEMORY=32768 # Memory in MB (default: 16384) +export EXTRA_WORKER_DISK=100 # Disk size in GB (default: 50) +export EXTRA_WORKER_VCPU=16 # Number of vCPUs (default: 8) +export BMC_DRIVER=ipmi # BMC driver type (default: redfish-virtualmedia) + +./add_worker_node.sh my-large-worker +``` + +### Post-Script Steps + +After running the script, follow these steps to complete the worker addition: + +#### 1. Apply the BareMetalHost Manifest + +```bash +oc apply -f ocp/ostest/_bmh.yaml +``` + +#### 2. Wait for Host to Become Available + +```bash +oc get baremetalhost -n openshift-machine-api -w +``` + +Wait until the new BareMetalHost shows `STATE: available`. + +#### 3. Scale the Worker MachineSet + +```bash +# List current machinesets +oc get machineset -n openshift-machine-api + +# Scale up (increment the replica count) +oc scale machineset -worker-0 --replicas= -n openshift-machine-api +``` + +**Example:** +```bash +$ oc get machineset -n openshift-machine-api +NAME DESIRED CURRENT READY AVAILABLE AGE +ostest-worker-0 2 2 2 2 5h + +$ oc scale machineset ostest-worker-0 --replicas=3 -n openshift-machine-api +machineset.machine.openshift.io/ostest-worker-0 scaled +``` + +#### 4. Monitor Machine and Node Creation + +```bash +# Watch machines +oc get machines -n openshift-machine-api -w + +# Watch nodes +oc get nodes -w +``` + +#### 5. Approve Certificate Signing Requests (CSRs) + +When the node boots and starts joining, it will create CSRs that need approval. + +**Manual approval:** +```bash +# Check for pending CSRs +oc get csr + +# Approve each CSR +oc adm certificate approve +``` + +**Auto-approval (convenient for testing):** +```bash +# Auto-approve CSRs for 30 minutes +./auto_approve_csrs.sh 30 +``` + +#### 6. Verify Node is Ready + +```bash +oc get nodes +``` + +You should see your new worker node in the `Ready` state. + +## Removing a Worker Node + +To remove a worker node that was added: + +```bash +./remove_worker_node.sh +``` + +Or using Make: +```bash +make remove_worker WORKER_NAME=my-worker-1 +``` + +This will: +1. Drain and delete the node from the cluster +2. Delete the BareMetalHost and Secret +3. Delete the corresponding Machine +4. Destroy the VM and clean up disk/NVRAM files +5. Remove BMC configuration + +**Note:** Remember to scale down your machineset after removing workers: +```bash +oc scale machineset --replicas= -n openshift-machine-api +``` + +## Complete Example Workflow + +```bash +# 1. Add a new worker +./add_worker_node.sh worker-extra-1 + +# 2. Apply the manifest +oc apply -f ocp/ostest/worker-extra-1_bmh.yaml + +# 3. Wait for it to be available +oc get bmh -n openshift-machine-api -w +# (Ctrl+C when status is 'available') + +# 4. Start auto-approving CSRs in background +./auto_approve_csrs.sh 30 & + +# 5. Scale the machineset +oc get machineset -n openshift-machine-api +oc scale machineset ostest-worker-0 --replicas=3 -n openshift-machine-api + +# 6. Watch the node join +oc get nodes -w + +# Done! The worker should be ready in 10-15 minutes +``` + +## Troubleshooting + +### VM Won't Boot + +Check VM status: +```bash +sudo virsh list --all +sudo virsh start _ +``` + +Check VM console: +```bash +sudo virsh console _ +``` + +### BareMetalHost Stuck in "Registering" + +Check BMC connectivity: +```bash +# For IPMI +ipmitool -I lanplus -H -U admin -P password power status + +# Check BMC logs +oc logs -n openshift-machine-api deployment/metal3 -c baremetal-operator +``` + +### CSRs Not Appearing + +Check that the node is booting and has network connectivity: +```bash +# Check DHCP leases +sudo virsh net-dhcp-leases baremetal + +# Check VM is running +sudo virsh list --all +``` + +### BareMetalHost Shows "available" but Machine Won't Create + +Check machineset events: +```bash +oc describe machineset -n openshift-machine-api +``` + +Ensure you have available BareMetalHosts: +```bash +oc get baremetalhost -n openshift-machine-api +``` + +## Architecture Notes + +- **BMC Drivers**: The script supports both `ipmi` (via vbmc) and `redfish-virtualmedia` (via sushy-tools) +- **Default**: `redfish-virtualmedia` is used by default as it's more modern and feature-rich +- **Port Allocation**: The script automatically finds available BMC ports +- **MAC Generation**: MAC addresses are randomly generated to avoid conflicts +- **Firmware**: Supports both BIOS and UEFI boot modes based on your `LIBVIRT_FIRMWARE` setting + +## Limitations + +- Only works with libvirt-based deployments +- Requires the provisioning host to have connectivity to libvirt +- BMC port range is limited (default: 6230-6250) +- Worker resources are uniform (all workers use the same resource settings) + +## See Also + +- [Agent-based Add Nodes](../agent/docs/add-nodes.md) - For agent installer deployments +- [Remote Nodes](../README.md#deploying-dummy-remote-cluster-nodes) - For adding nodes on separate networks + diff --git a/examples/add_worker_example.sh b/examples/add_worker_example.sh new file mode 100644 index 000000000..5efc402f8 --- /dev/null +++ b/examples/add_worker_example.sh @@ -0,0 +1,196 @@ +#!/bin/bash +# +# Example: Adding Worker Nodes Post-Installation +# +# This script demonstrates various ways to add worker nodes +# to your cluster after deployment. +# + +# NOTE: This is an example file showing different usage patterns. +# Copy and modify as needed for your use case. + +set -e + +SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd )" +cd "$SCRIPTDIR" + +echo "==========================================" +echo "Example: Adding Workers Post-Installation" +echo "==========================================" +echo "" + +# ============================================================================ +# Example 1: Add a single worker with default resources +# ============================================================================ +echo "Example 1: Add worker with default resources (16GB RAM, 50GB disk, 8 vCPU)" +echo "" +echo "Commands:" +echo " ./add_worker_node.sh worker-default" +echo " oc apply -f ocp/ostest/worker-default_bmh.yaml" +echo " oc scale machineset \$(oc get machineset -n openshift-machine-api -o name | head -1 | cut -d/ -f2) --replicas=3 -n openshift-machine-api" +echo "" + +# ============================================================================ +# Example 2: Add a large worker with custom resources +# ============================================================================ +echo "Example 2: Add large worker with custom resources" +echo "" +echo "Commands:" +echo " export EXTRA_WORKER_MEMORY=32768 # 32GB RAM" +echo " export EXTRA_WORKER_DISK=100 # 100GB disk" +echo " export EXTRA_WORKER_VCPU=16 # 16 vCPUs" +echo " ./add_worker_node.sh worker-large" +echo " oc apply -f ocp/ostest/worker-large_bmh.yaml" +echo "" + +# ============================================================================ +# Example 3: Add multiple workers in sequence +# ============================================================================ +echo "Example 3: Add multiple workers (requires running commands sequentially)" +echo "" +echo "Commands:" +echo " # Add first worker" +echo " ./add_worker_node.sh worker-1" +echo " oc apply -f ocp/ostest/worker-1_bmh.yaml" +echo "" +echo " # Add second worker" +echo " ./add_worker_node.sh worker-2" +echo " oc apply -f ocp/ostest/worker-2_bmh.yaml" +echo "" +echo " # Add third worker" +echo " ./add_worker_node.sh worker-3" +echo " oc apply -f ocp/ostest/worker-3_bmh.yaml" +echo "" +echo " # Scale machineset to match (3 new workers)" +echo " oc scale machineset \$(oc get machineset -n openshift-machine-api -o name | head -1 | cut -d/ -f2) --replicas=5 -n openshift-machine-api" +echo "" + +# ============================================================================ +# Example 4: Complete workflow with auto-CSR approval +# ============================================================================ +echo "Example 4: Complete workflow with automatic CSR approval" +echo "" +echo "Commands:" +echo " # Start CSR auto-approval in background" +echo " ./auto_approve_csrs.sh 30 &" +echo " CSR_PID=\$!" +echo "" +echo " # Add and apply worker" +echo " ./add_worker_node.sh test-worker" +echo " oc apply -f ocp/ostest/test-worker_bmh.yaml" +echo "" +echo " # Wait for BareMetalHost to be available" +echo " echo 'Waiting for BareMetalHost to be available...'" +echo " oc wait --for=jsonpath='{.status.provisioning.state}'=available \\" +echo " baremetalhost/${CLUSTER_NAME}-test-worker \\" +echo " -n openshift-machine-api --timeout=10m" +echo "" +echo " # Scale machineset" +echo " CURRENT_REPLICAS=\$(oc get machineset -n openshift-machine-api -o jsonpath='{.items[0].spec.replicas}')" +echo " NEW_REPLICAS=\$((CURRENT_REPLICAS + 1))" +echo " oc scale machineset \$(oc get machineset -n openshift-machine-api -o name | head -1 | cut -d/ -f2) \\" +echo " --replicas=\$NEW_REPLICAS -n openshift-machine-api" +echo "" +echo " # Wait for node to be ready" +echo " echo 'Waiting for node to be ready...'" +echo " oc wait --for=condition=Ready node/${CLUSTER_NAME}-test-worker --timeout=20m" +echo "" +echo " # Stop CSR approval (it will auto-stop after 30 minutes anyway)" +echo " kill \$CSR_PID 2>/dev/null || true" +echo "" + +# ============================================================================ +# Example 5: Using Make targets +# ============================================================================ +echo "Example 5: Using Make targets (simplest method)" +echo "" +echo "Commands:" +echo " # Add worker" +echo " make add_worker WORKER_NAME=my-worker" +echo "" +echo " # Apply manifest (still manual step)" +echo " oc apply -f ocp/ostest/my-worker_bmh.yaml" +echo "" +echo " # Scale machineset (still manual step)" +echo " oc scale machineset --replicas= -n openshift-machine-api" +echo "" +echo " # Later: Remove worker" +echo " make remove_worker WORKER_NAME=my-worker" +echo "" + +# ============================================================================ +# Example 6: Remove a worker node +# ============================================================================ +echo "Example 6: Remove a worker node" +echo "" +echo "Commands:" +echo " # Scale down machineset first" +echo " CURRENT_REPLICAS=\$(oc get machineset -n openshift-machine-api -o jsonpath='{.items[0].spec.replicas}')" +echo " NEW_REPLICAS=\$((CURRENT_REPLICAS - 1))" +echo " oc scale machineset \$(oc get machineset -n openshift-machine-api -o name | head -1 | cut -d/ -f2) \\" +echo " --replicas=\$NEW_REPLICAS -n openshift-machine-api" +echo "" +echo " # Wait for machine to be deleted" +echo " echo 'Waiting for machine deletion...'" +echo " sleep 60" +echo "" +echo " # Remove the worker and clean up" +echo " ./remove_worker_node.sh my-worker" +echo "" + +# ============================================================================ +# Example 7: Check worker status +# ============================================================================ +echo "Example 7: Monitoring worker status" +echo "" +echo "Commands:" +echo " # Check BareMetalHosts" +echo " oc get baremetalhost -n openshift-machine-api" +echo "" +echo " # Check Machines" +echo " oc get machines -n openshift-machine-api" +echo "" +echo " # Check Nodes" +echo " oc get nodes" +echo "" +echo " # Watch node joining process" +echo " oc get nodes -w" +echo "" +echo " # Check pending CSRs" +echo " oc get csr | grep Pending" +echo "" +echo " # Check specific BareMetalHost details" +echo " oc describe baremetalhost ${CLUSTER_NAME}-worker-name -n openshift-machine-api" +echo "" + +# ============================================================================ +# Example 8: Troubleshooting +# ============================================================================ +echo "Example 8: Troubleshooting commands" +echo "" +echo "Commands:" +echo " # Check VM status" +echo " sudo virsh list --all | grep ${CLUSTER_NAME}" +echo "" +echo " # Start a VM if it's stopped" +echo " sudo virsh start ${CLUSTER_NAME}_worker-name" +echo "" +echo " # Check VM console" +echo " sudo virsh console ${CLUSTER_NAME}_worker-name" +echo "" +echo " # Check DHCP leases" +echo " sudo virsh net-dhcp-leases baremetal" +echo "" +echo " # Check baremetal operator logs" +echo " oc logs -n openshift-machine-api deployment/metal3 -c baremetal-operator --tail=50" +echo "" +echo " # Check machine controller logs" +echo " oc logs -n openshift-machine-api deployment/machine-api-controllers -c machine-controller --tail=50" +echo "" + +echo "==========================================" +echo "For more information, see:" +echo " - WORKER_QUICK_START.md" +echo " - docs/add-worker-post-install.md" +echo "==========================================" + diff --git a/remove_worker_node.sh b/remove_worker_node.sh new file mode 100755 index 000000000..5ce0b2366 --- /dev/null +++ b/remove_worker_node.sh @@ -0,0 +1,125 @@ +#!/bin/bash +# +# Remove a worker node from the cluster +# +# Usage: ./remove_worker_node.sh +# +# This script: +# 1. Deletes the BareMetalHost from the cluster +# 2. Deletes the corresponding Machine (if exists) +# 3. Removes the VM and its BMC configuration +# 4. Cleans up disk and NVRAM files +# + +set -euo pipefail + +SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" + +source ${SCRIPTDIR}/logging.sh +source ${SCRIPTDIR}/common.sh +source ${SCRIPTDIR}/network.sh +source ${SCRIPTDIR}/utils.sh +source ${SCRIPTDIR}/ocp_install_env.sh + +# Parse arguments +if [ $# -lt 1 ]; then + echo "Usage: $0 " + echo "Example: $0 extraworker-0" + exit 1 +fi + +WORKER_NAME=$1 + +echo "==========================================" +echo "Removing worker node: ${WORKER_NAME}" +echo "Cluster: ${CLUSTER_NAME}" +echo "==========================================" + +# Check if cluster is accessible +if ! oc get nodes &>/dev/null; then + echo "Warning: Cannot connect to cluster. Will only clean up local resources." + CLUSTER_ACCESSIBLE=false +else + CLUSTER_ACCESSIBLE=true +fi + +# Delete BareMetalHost +if [ "$CLUSTER_ACCESSIBLE" = true ]; then + BMH_NAME="${CLUSTER_NAME}-${WORKER_NAME}" + if oc get baremetalhost -n openshift-machine-api "$BMH_NAME" &>/dev/null; then + echo "Deleting BareMetalHost ${BMH_NAME}..." + oc delete baremetalhost -n openshift-machine-api "$BMH_NAME" || true + + # Also delete the secret + SECRET_NAME="${BMH_NAME}-bmc-secret" + if oc get secret -n openshift-machine-api "$SECRET_NAME" &>/dev/null; then + echo "Deleting secret ${SECRET_NAME}..." + oc delete secret -n openshift-machine-api "$SECRET_NAME" || true + fi + else + echo "BareMetalHost ${BMH_NAME} not found in cluster" + fi + + # Check for and delete corresponding Machine + echo "Checking for Machine resources..." + MACHINES=$(oc get machine -n openshift-machine-api -o name | grep -i "$WORKER_NAME" || true) + if [ -n "$MACHINES" ]; then + echo "Found machines: $MACHINES" + echo "$MACHINES" | xargs -r oc delete -n openshift-machine-api || true + fi + + # Check for and drain the node if it exists + if oc get node "${CLUSTER_NAME}-${WORKER_NAME}" &>/dev/null; then + echo "Draining node ${CLUSTER_NAME}-${WORKER_NAME}..." + oc adm drain "${CLUSTER_NAME}-${WORKER_NAME}" --ignore-daemonsets --delete-emptydir-data --force || true + echo "Deleting node ${CLUSTER_NAME}-${WORKER_NAME}..." + oc delete node "${CLUSTER_NAME}-${WORKER_NAME}" || true + fi +fi + +# Stop and destroy VM +VM_NAME="${CLUSTER_NAME}_${WORKER_NAME}" +if sudo virsh list --all | grep -q "$VM_NAME"; then + echo "Stopping VM ${VM_NAME}..." + sudo virsh destroy "$VM_NAME" 2>/dev/null || true + echo "Undefining VM ${VM_NAME}..." + sudo virsh undefine "$VM_NAME" --nvram 2>/dev/null || true +else + echo "VM ${VM_NAME} not found" +fi + +# Remove vbmc entry if using IPMI +if [ "${BMC_DRIVER:-redfish-virtualmedia}" = "ipmi" ]; then + if is_running vbmc; then + echo "Removing vbmc entry for ${VM_NAME}..." + sudo podman exec vbmc vbmc delete "$VM_NAME" 2>/dev/null || true + fi +fi + +# Remove disk file +DISK_PATH="/var/lib/libvirt/images/${VM_NAME}.qcow2" +if [ -f "$DISK_PATH" ]; then + echo "Removing disk ${DISK_PATH}..." + sudo rm -f "$DISK_PATH" +else + echo "Disk ${DISK_PATH} not found" +fi + +# Remove NVRAM file +NVRAM_PATH="/var/lib/libvirt/qemu/nvram/${VM_NAME}_VARS.fd" +if [ -f "$NVRAM_PATH" ]; then + echo "Removing NVRAM ${NVRAM_PATH}..." + sudo rm -f "$NVRAM_PATH" +fi + +# Remove manifest file if it exists +BMH_MANIFEST="${OCP_DIR}/${WORKER_NAME}_bmh.yaml" +if [ -f "$BMH_MANIFEST" ]; then + echo "Removing manifest ${BMH_MANIFEST}..." + rm -f "$BMH_MANIFEST" +fi + +echo "==========================================" +echo "Worker node ${WORKER_NAME} removed successfully" +echo "==========================================" +