Skip to content

OCPBUGS-97602: Use glance image with stable name to avoid race condition #81445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
winiciusallan wants to merge 1 commit into
base: main
Choose a base branch
from
+18 −53
Open

OCPBUGS-97602: Use glance image with stable name to avoid race condition #81445

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 14 additions & 50 deletions ci-operator/step-registry/openstack/conf/rhcosimage/openstack-conf-rhcosimage-commands.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,6 @@ set -o nounset
set -o errexit
set -o pipefail

# Get OCP version from the installer binary itself.
# This ensures the Glance image name matches the RHCOS content that the
# installer embeds, which is important for upgrade jobs where the installer
# version can differ from the release under test.
OCP_VERSION="$(openshift-install version 2>/dev/null | sed -n 's/^release image\s\+.*:\([0-9]\+\.[0-9]\+\).*/\1/p')"
if [[ ! "${OCP_VERSION}" =~ ^[0-9]+\.[0-9]+$ ]]; then
echo "ERROR: Failed to extract OCP version from openshift-install:" >&2
openshift-install version >&2
exit 1
fi

export OS_CLIENT_CONFIG_FILE="${SHARED_DIR}/clouds.yaml"

# For disconnected or otherwise unreachable environments, we want to
Expand All @@ -38,9 +27,15 @@ UNCOMPRESSED_SHA256="$(jq --raw-output '.architectures.x86_64.artifacts.openstac
IMAGE_VERSION="$(jq --raw-output '.architectures.x86_64.artifacts.openstack.release' "$COREOS_JSON")"
rm -f "$COREOS_JSON"

IMAGE_NAME="${OPENSTACK_RHCOS_IMAGE_NAME}-${OCP_VERSION}"
# Include the RHCOS stream version in the image name so that each
# RHCOS build gets its own immutable Glance image (e.g.
# "rhcos-422.94.202506120833-0"). The RHCOS version already encodes
# the OCP minor version (422 = 4.22, 500 = 5.0), so it is unique
# across releases. This makes each job pick their respective release.
IMAGE_NAME="${OPENSTACK_RHCOS_IMAGE_NAME}-${IMAGE_VERSION}"

echo "RHCOS version from installer: ${IMAGE_VERSION}"
echo "Target image name: ${IMAGE_NAME}"
echo "Image URL: ${IMAGE_URL}"

# Use `openstack image list` instead of `openstack image show` for all
Expand Down Expand Up @@ -72,24 +67,10 @@ if [[ "$EXISTING_COUNT" -gt 1 ]]; then
fi
fi

# Check if the (single remaining) image is up-to-date
CURRENT_SHA256=""
CURRENT_ID=""
if [[ "$EXISTING_COUNT" -ge 1 ]]; then
CURRENT_ID="$(echo "$EXISTING_IMAGES" | jq -r 'sort_by(.["Created At"]) | reverse | .[0].ID')"
if IMAGE_PROPS="$(openstack image show -c properties -f json "$CURRENT_ID" 2>/dev/null)"; then
CURRENT_SHA256="$(echo "$IMAGE_PROPS" | jq --raw-output '
.properties["owner_specified.openstack.sha256"] //
.properties["sha256"] //
empty
')"
fi
fi

if [[ "$CURRENT_SHA256" == "$UNCOMPRESSED_SHA256" ]]; then
echo "RHCOS image '${IMAGE_NAME}' already at the expected version (${IMAGE_VERSION}). Skipping upload."
if [[ "$EXISTING_COUNT" -eq 1 ]]; then
echo "RHCOS image '${IMAGE_NAME}' already exists. Skipping upload."
else
echo "RHCOS image '${IMAGE_NAME}' needs to be uploaded (current sha256: '${CURRENT_SHA256}', expected: '${UNCOMPRESSED_SHA256}')"
echo "RHCOS image '${IMAGE_NAME}' not found. Uploading..."
Comment on lines +70 to +73

@coderabbitai coderabbitai Bot Jul 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

sed -n '1,180p' ci-operator/step-registry/openstack/conf/rhcosimage/openstack-conf-rhcosimage-commands.sh

Repository: openshift/release

Length of output: 2105

🏁 Script executed:

rg -n "openstack image list -f json|\\.Status|\\.status|image list -f json" ci-operator/step-registry/openstack -S

Repository: openshift/release

Length of output: 2105

🏁 Script executed:

python3 - <<'PY'
import json, re, sys, pathlib
p = pathlib.Path("ci-operator/step-registry/openstack/conf/rhcosimage/openstack-conf-rhcosimage-commands.sh")
text = p.read_text()
for m in re.finditer(r'openstack image list -f json.*?jq -r \'([^\']+)\'', text, re.S):
    print(m.group(1))
PY

Repository: openshift/release

Length of output: 2105

🌐 Web query:

OpenStackClient openstack image list -f jsonJSON field names status casingStatus status docs

💡 Result:

When using the OpenStackClient command openstack image list -f json, the field name for the image status is lowercase status [1][2][3]. It is not capitalized as Status [1][3][4]. The OpenStack Image Service API (Glance), which backs this command, consistently uses the lowercase status key in its JSON response objects [2][5][3]. Similarly, the internal Python-OpenStackClient code that formats these results maps the field to status [1], and the OpenStack SDK resource definition for an image also uses status as the attribute name [4].

Citations:

Only skip when the existing image is active. openstack image list -f json uses the lowercase status field, so check .[0].status here and fail/refresh on queued, saving, or killed images instead of reusing them.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@ci-operator/step-registry/openstack/conf/rhcosimage/openstack-conf-rhcosimage-commands.sh`
around lines 70 - 73, The skip logic in the RHCOS image upload flow is too broad
and currently reuses any matching image instead of only a ready one. Update the
check in the image lookup/upload path to inspect the OpenStack image JSON status
via .[0].status and only skip when the existing image is active; for queued,
saving, or killed states, do not reuse it and instead refresh or fail so the
upload path can proceed correctly. Use the existing IMAGE_NAME/EXISTING_COUNT
handling in the openstack-conf-rhcosimage-commands.sh flow to locate the
condition.


WORK_DIR="$(mktemp -d)"

Expand All @@ -108,36 +89,19 @@ else
echo "Verifying uncompressed image checksum..."
echo "${UNCOMPRESSED_SHA256} ${UNCOMPRESSED_FILE}" | sha256sum --check --quiet

# Clean up leftover from any previous failed run (by ID to handle duplicates)
for id in $(openstack image list --name "${IMAGE_NAME}-new" -f value -c ID 2>/dev/null); do
openstack image delete "$id" 2>/dev/null || true
done

echo "Uploading image to '${OS_CLOUD}' as '${IMAGE_NAME}-new'..."
NEW_IMAGE_ID="$(openstack image create "${IMAGE_NAME}-new" \
echo "Uploading image to '${OS_CLOUD}' as '${IMAGE_NAME}'..."
IMAGE_ID=$(openstack image create "${IMAGE_NAME}" \
--container-format bare \
--disk-format qcow2 \
--file "$UNCOMPRESSED_FILE" \
--private \
--property sha256="$UNCOMPRESSED_SHA256" \
--property rhcos_version="$IMAGE_VERSION" \
--format value --column id)"

echo "Replacing old '${IMAGE_NAME}' image with new one..."
# Delete old images by ID
for id in $(openstack image list --name "${IMAGE_NAME}-old" -f value -c ID 2>/dev/null); do
openstack image delete "$id" 2>/dev/null || true
done
# Rename current image to -old (by ID, not name)
if [[ -n "$CURRENT_ID" ]]; then
openstack image set --name "${IMAGE_NAME}-old" "$CURRENT_ID" 2>/dev/null || true
fi
# Promote the new image (already using ID)
openstack image set --name "$IMAGE_NAME" "$NEW_IMAGE_ID"
--format value --column id)

rm -rf "$WORK_DIR"

echo "RHCOS image '${IMAGE_NAME}' updated to version ${IMAGE_VERSION}."
echo "RHCOS image '${IMAGE_NAME}' with ID '${IMAGE_ID}' uploaded."
fi

# Patch install-config.yaml to use the pre-uploaded image
Expand Down
7 changes: 4 additions & 3 deletions ci-operator/step-registry/openstack/conf/rhcosimage/openstack-conf-rhcosimage-ref.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,11 @@ ref:
- name: OPENSTACK_RHCOS_IMAGE_NAME
default: "rhcos"
documentation: |-
Prefix for the RHCOS image name in OpenStack Glance. The OCP minor
Prefix for the RHCOS image name in OpenStack Glance. The RHCOS stream
version from the openshift-install binary is appended automatically
(e.g. "rhcos" becomes "rhcos-4.18"). The step will create or update
this image to match the version embedded in openshift-install.
(e.g. "rhcos" becomes "rhcos-422.94.202506120833-0"). Each RHCOS build
gets its own immutable image, avoiding race conditions between
concurrent jobs.
documentation: |-
Pre-uploads the RHCOS image to OpenStack Glance when it is not already
at the version expected by the installer, then patches install-config.yaml
Expand Down