Skip to content

Rewrite create-prowjob README.md to emphasize using the script #75823

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tbuskey wants to merge 5 commits into
base: main
Choose a base branch
from
+155 −143
Open

Rewrite create-prowjob README.md to emphasize using the script #75823

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4931b01
Rewrite create-prowjob README.md to emphasize using the script
tbuskey Mar 6, 2026
3348413
Move enviroment variable table from README.md to script output
tbuskey Mar 6, 2026
37034c4
Remove PROW_RUN_TYPE from environment variables. It is derived from …
tbuskey Mar 6, 2026
4afdffc
Remove explanation of how catalog tag :latest gets translated
tbuskey Mar 6, 2026
b47c4f1
Warning that RPM install won't work on interactive PR
tbuskey Mar 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
196 changes: 88 additions & 108 deletions ci-operator/step-registry/sandboxed-containers-operator/create-prowjob/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,44 +1,68 @@
# Sandboxed Containers Operator - Prowjob Configuration Generator

This directory has a script to generate OpenShift CI prowjob configuration files for the Sandboxed Containers Operator with validation and error handling.
`sandboxed-containers-operator-create-prowjob-commands.sh` generates prowjob configuration files for the Sandboxed Containers Operator with validation and error handling. The existing files should not be edited directly. `sandboxed-containers-operator-create-prowjob-commands.sh` can also be used to run prowjobs with Gangway, like Konflux

## Overview

The `sandboxed-containers-operator-create-prowjob-commands.sh` script creates prowjob configuration files for the sandboxed containers operator CI pipeline. It supports both Pre-GA (development) and GA (production) release types with parameter validation.
The `sandboxed-containers-operator-create-prowjob-commands.sh` script creates prowjob configuration files. These prowjobs contain variations of provider/workload combinations. We will start with [Creating Prowjobs](#creating-prowjobs) and [Running Prowjobs](#running-prowjobs). More detail starts in [General Usage](#general-usage).

## Commands
## Creating Prowjobs
`sandboxed-containers-operator-create-prowjob-commands.sh update_templates` regenerates all exising prowjobs with the script defaults, these jobs are used directly by Konflux. It is preferred to make all changes in the script and not make changes in existing prowjobs. Also see [Update Templates](#update-templates)

The script supports the following commands:
`sandboxed-containers-operator-create-prowjob-commands.sh create` creates a yaml file in the current directory. Environment variables alter the yaml output. `create` is used by `update_templates` also.

- `create` - Create prowjob configuration files
- `run` - Run one prowjob from YAML (requires job YAML file and exactly one job name)

### Command Usage
## Running Prowjobs
### Konflux and Gangway
You cannot login to Gangway created clusters by design. They are useful for rerunning tests launched from Konflux without creating a new build. ex: infrastructure issues prevent the test from starting

Either use an existing YAMLFILE or `create` a new YAMLFILE in the current directory. From this YAMLFILE, we can `run` to create a cluster on PROVIDER and run the WORKLOAD tests using the Gangway API. You will need a Prow API token as explained in the [Gangway](#gangway) section
```bash
# Show help
./sandboxed-containers-operator-create-prowjob-commands.sh
sandboxed-containers-operator-create-prowjob-commands.sh run <YAMLFILE> <PROVIDER-ipi-WORKLOAD>
```
Konflux modifies a copy of the template files and uses Gangway to launch tests.

## Files
### Interactive
1. Create a branch `git checkout -b BRANCH` in the repo.
2. `create` the YAMLFILE in the current directory with your ENV changes. We want the cluster to stay around for 6 hours after the tests finish.
```bash
SLEEP_DURATION=6h OCP_VERSION=4.21 TEST_RELEASE_TYPE=Pre-GA `sandboxed-containers-operator-create-prowjob-commands.sh create`
```
`sandboxed-containers-operator-create-prowjob-commands.sh create` output has directions to mv the file, add to git and run a series of make commands. Don't follow them yet
3. You need modify the YAMLFILE to allow interaction
```bash
sed -ie 's/restrict_network_access: false/restrict_network_access: true/ ' YAMLFILE
Comment thread
tbuskey marked this conversation as resolved.
```
**NOTE: this will prevent the RPM upload if you need to install one!** You will need to upload the RPM to the nodes and install them after the kataconfig is installed. It will also make the automated tests invalid

- `sandboxed-containers-operator-create-prowjob-commands.sh` - Main script to generate prowjob configurations
- The output file is created in the current directory and named `openshift-sandboxed-containers-operator-devel__downstream-${PROW_RUN_TYPE}${OCP_VERSION}.yaml`
- `PROW_RUN_TYPE` is based on `TEST_RELEASE_TYPE`. It is `candidate` for `Pre-GA` and `release` otherwise
- If the output file exists, it will be moved to a `.backup` file
4. Now follow the directions to mv the file, add to git, run make commands
5. `git status -uno` and git add any changed files
6. `git commit -m '[DEBUG] [DO NOT MERGE]'`
7. `git push --set-upstream origin BRANCH`
8. Go to the PR and interact with it

## Key Features
#### Interactive in the PR
The openshift-ci-robot will list a number of test names you can can with `/pj-rehearse`

- Different behavior for Pre-GA vs GA releases
- Uses `:latest` tag by default for catalog images
- The actual latest tag (X.Y.Z-epoch_time format) is resolved at runtime by the `env-cm` step
- This ensures jobs always use the most recent catalog image when they run
- Generated files are not merged
- /PJ-REHEARSE in the PR to run prowjobs
If I created a prowjob as above, I will have a series like
```bash
periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidateOCP_VERSION-PROVIDER-ipi-WORKLOAD
```
Launch with
```bash
/pj-rehearse periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-aws-ipi-peerpods
```
This will launch the tests that konflux's run does. But you can get interactive access to the cluster.

## Usage
#### Interactive Cluster Access
In the PR, you will see a _pending check_ that looks like `ci/rehearse/periodic-ci-openshift-sandboxed-containers..` that is a URL to the **Spyglass** page for the prowjob. This page has all the logs and artifacts for your prowjob. In the **Build Log** section, you will see _Using namespace_ followed by a URL. This is the undercluster that is building your cluster. It will log you into the undercluster. Under **Inventory** you will see a pod for every step in the prowjob that has finished and the current running one. While it is running, the `PROVIDER-ipi-WORKLOAD-openshift-extended-test` pod will show the output of the test in its log.

### Basic Usage
Further under **Inventory** are secrets. The `PROVIDER-ipi-WORKLOAD` secret will eventually have the `kubeadmin-password` and `kubeconfig` files you can reveal and copy. This usually happends while the tests are running.

Once you have the credentials you can login to the cluster. However, you should only observe until the tests finish. The pod `PROVIDER_ipi-WORKLOAD-cucushift-installer-wait` will appear and stay running for `SLEEP_DURATION` hours. Then the pod will end and prow will deprovision. You can login to the pod and terminate the sleep loop early. You can only make the pod last 12 hours.


## General Usage
### Basic Usage of `sandboxed-containers-operator-create-prowjob-commands.sh`

Ensure you are in the __release__ directory of your fork of the [Prow repo](https://github.com/openshift/release)
The script uses environment variables exclusively for configuration:
Expand All @@ -58,37 +82,24 @@ PROW_API_TOKEN=your_token_here ci-operator/step-registry/sandboxed-containers-op
```

### Environment Variables
Run the script with no arguments or an invalid command to see full environment variable usage (descriptions, defaults, and validation). Example: `ci-operator/step-registry/sandboxed-containers-operator/create-prowjob/sandboxed-containers-operator-create-prowjob-commands.sh` or `$0 invalidcommand`
Comment thread
tbuskey marked this conversation as resolved.

### Pre-GA vs GA Configuration (`TEST_RELEASE_TYPE`)

| Variable | Default Value | Description | Validation |
| -------------------------- | ------------------------ | --------------------------------------------------------------------------- | ------------------------ |
| `OCP_VERSION` | Files for 4 OCP versions are pregenerated | OpenShift Container Platform version. Supports `X.Y` (latest), `X.Y.Z` (specific), or `X.Y.Z-rc.N`/`X.Y.Z-ec.N` (candidate). | If a specific version doesn't exist, error out.
| `OCP_CHANNEL` | `fast` | OCP release channel. Default is `fast` because it contains all versions that could become `stable`. Further explanation below | `stable`, `fast`, `candidate`, or `eus` |
| `AWS_REGION_OVERRIDE` | `us-east-2` | AWS region for testing | Any valid AWS region |
| `CUSTOM_AZURE_REGION` | `eastus` | Azure region for testing | Any valid Azure region |
| `OSC_CATALOG_TAG` | `latest` | Defaults to `:latest`. Actual tag resolved at runtime by `env-cm` step. Can override with specific version tag (e.g., `1.11.1-1766149846`) or SHA | repo tag or SHA |
| `INSTALL_KATA_RPM` | `true` | Whether to install Kata RPM | `true` or `false` |
| `KATA_RPM_VERSION` | `3.17.0-3.rhaos4.19.el9` | Kata RPM version (when `INSTALL_KATA_RPM=true`) | RPM version format |
| `PROW_RUN_TYPE` | `candidate` | Prow job run type | `candidate` or `release` |
| `SLEEP_DURATION` | `0h` | Time to keep cluster alive after tests. For manual testing. | 0-12 followed by 'h' |
| `TEST_RELEASE_TYPE` | `Pre-GA` | Release type for testing | `Pre-GA` or `GA` |
| `TEST_TIMEOUT` | `90` | Test timeout in minutes | Numeric value |

### Pre-GA vs GA Configuration

#### Pre-GA (Development) Mode
#### Pre-GA (Development) Mode (`OSC_CATALOG_TAG`)
- Uses `:latest` tag for catalog images by default
- The `env-cm` step resolves the actual latest tag (X.Y.Z-epoch_time format) at runtime
- This ensures jobs always test against the most recent build
- Creates `brew-catalog` source with the resolved catalog tag

#### GA (Production) Mode
#### GA (Production) Mode (`TEST_RELEASE_TYPE=GA`)
- Uses `redhat-operators` catalog source with GA images

### OCP Release Channels
### OCP Release Channels (`OCP_CHANNEL`)

The `OCP_CHANNEL` variable determines which OpenShift release channel to use. Use `candidate` for rc/ec versions

#### Channel Comparison
#### Channel Comparison (`OCP_CHANNEL` values)
Copy link
Copy Markdown
Contributor

@ldoktor ldoktor Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are not related to the commit

Copy link
Copy Markdown
Contributor

@ldoktor ldoktor Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I mean they seem useful, but should be separated and well described)


| Channel | Pre-Release | Use Case |
|---------|-------------|----------|
Expand All @@ -99,25 +110,23 @@ The `OCP_CHANNEL` variable determines which OpenShift release channel to use. U

### Advanced Configuration Examples

#### Pre-GA Development Testing
#### Pre-GA Development Testing (`TEST_RELEASE_TYPE`, `OCP_VERSION`)
```bash
# Test latest development builds
TEST_RELEASE_TYPE=Pre-GA \
PROW_RUN_TYPE=candidate \
OCP_VERSION=4.18 \
ci-operator/step-registry/sandboxed-containers-operator/create-prowjob/sandboxed-containers-operator-create-prowjob-commands.sh create
```

#### GA Production Testing
#### GA Production Testing (`TEST_RELEASE_TYPE`, `OCP_VERSION`)
```bash
# Test production releases
TEST_RELEASE_TYPE=GA \
PROW_RUN_TYPE=release \
OCP_VERSION=4.19 \
ci-operator/step-registry/sandboxed-containers-operator/create-prowjob/sandboxed-containers-operator-create-prowjob-commands.sh create
```

#### Custom Regions and Timeouts
#### Custom Regions and Timeouts (`AWS_REGION_OVERRIDE`, `CUSTOM_AZURE_REGION`, `SLEEP_DURATION`, `TEST_TIMEOUT`)
```bash
# Extended testing with custom regions
AWS_REGION_OVERRIDE=us-west-2 \
Expand All @@ -127,7 +136,7 @@ TEST_TIMEOUT=120 \
ci-operator/step-registry/sandboxed-containers-operator/create-prowjob/sandboxed-containers-operator-create-prowjob-commands.sh create
```

#### Kata RPM Testing
#### Kata RPM Testing (`INSTALL_KATA_RPM`, `KATA_RPM_VERSION`)
```bash
# Test without Kata RPM installation
INSTALL_KATA_RPM=false \
Expand All @@ -139,61 +148,54 @@ KATA_RPM_VERSION=3.18.0-3.rhaos4.20.el9 \
ci-operator/step-registry/sandboxed-containers-operator/create-prowjob/sandboxed-containers-operator-create-prowjob-commands.sh create
```

## Catalog Tag Discovery

### Runtime Tag Resolution
### CATALOG_SOURCE_IMAGE Tag Resolution

The `create-prowjob` script uses `:latest` as the default tag. The actual latest tag is resolved at **runtime** by the `env-cm` step when the job executes.
Copy link
Copy Markdown
Contributor

@ldoktor ldoktor Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about simply continuing in the sentence and say ...when the job executes. It can be overridden by any valid tag using "CATALOG_SOURCE_IMAGE" variable (not sure about the right env variable...)


### How It Works
1. **At config generation time**: `CATALOG_SOURCE_IMAGE` is set to `quay.io/redhat-user-workloads/ose-osc-tenant/osc-test-fbc:latest`
2. **At job runtime**: The `env-cm` step checks the tag:
- If the tag is `:latest`, it queries the Quay API to find the most recent `X.Y.Z-unix_epoch` tag
- If the tag is anything else (specific version or SHA), it is passed through unchanged
3. **Supported tag formats**:
- `latest` - resolved to newest `X.Y.Z-unix_epoch` tag at runtime
- `X.Y.Z-unix_epoch` (e.g., `1.11.1-1766149846`) - passed through unchanged
- SHA (e.g., `sha256:abc123...` or short SHA) - passed through unchanged
4. **Source**: `quay.io/redhat-user-workloads/ose-osc-tenant/osc-test-fbc`
It can be overridden with any other valid tag


## Run Command
## Gangway

The `run` command triggers a single ProwJob from a generated YAML configuration file. You must specify exactly one job name (the `as` value from the tests in the YAML, e.g. `azure-ipi-kata`, `aws-ipi-peerpods`). This command requires a valid Prow API token.
### Run Command Features

### Run Command Usage
- **Exactly one job**: You must provide the job YAML file and exactly one job name; neither more nor fewer are allowed.
- **Job Name Generation**: Constructs the full job name as `periodic-ci-{org}-{repo}-{branch}-{variant}-{job_name}`.
- **Metadata Extraction**: Extracts organization, repository, branch, and variant from the YAML file's `zz_generated_metadata` section.
- **API Integration**: Uses the Prow/Gangway API to trigger the job.
- **Job Status Monitoring**: Provides job ID and status information.


### Gangway `PROW_API_TOKEN`

To trigger ProwJobs via Gangway, you need a token for authentication. Tokens can be retrieved through the UI of the app.ci cluster at [OpenShift Console](https://console-openshift-console.apps.ci.l2s4.p1.openshiftapps.com).

Tf the app.ci cluster context is already configured:
```bash
# Run one job from a YAML file (exactly one job name required)
./sandboxed-containers-operator-create-prowjob-commands.sh run /path/to/job_yaml.yaml azure-ipi-kata
oc whoami -t
```

### Create Yaml for Run
Use the **create** command to create a yaml file in the current directory. Modify it for your case. Follow Option A in the output directions
Set `PROW_API_TOKEN` to the token

### Viewing the Run in Spyglass
```bash
export PROW_API_TOKEN=your_token_here
```

For complete information about triggering ProwJobs via REST, including permanent tokens for automation, see the [OpenShift CI documentation](https://docs.ci.openshift.org/docs/how-tos/triggering-prowjobs-via-rest/#obtaining-an-authentication-token).

The `run` command triggers a single ProwJob from a generated YAML configuration file. You must specify exactly one job name (the `as` value from the tests in the YAML, e.g. `azure-ipi-kata`, `aws-ipi-peerpods`). This command requires a valid Prow API token.

### Spyglass
#### Viewing the Run's URL
Go to [Prow configured jobs](https://prow.ci.openshift.org/configured-jobs/)
Scroll down to *sandboxed-containers-operator* and click on it
Search for the prow job you specified (ex aws-ipi-peerpods) and click on _Details_
Click on _History_
You will be taken to a list of the **Build** numbers, etc. Your job should be at the top. Clicking on that will show you the Spyglass of your job with the build log, artifacts, etc.
This URL is used by **dig&shift** for reporting and analysis. It will look something like [this](https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/75051/rehearse-75051-periodic-ci-openshift-sandboxed-containers-operator-devel-downstream-candidate421-azure-ipi-kata/2024178159888371712)

### Run Command Features

- **Exactly one job**: You must provide the job YAML file and exactly one job name; neither more nor fewer are allowed.
- **Job Name Generation**: Constructs the full job name as `periodic-ci-{org}-{repo}-{branch}-{variant}-{job_name}`.
- **Metadata Extraction**: Extracts organization, repository, branch, and variant from the YAML file's `zz_generated_metadata` section.
- **API Integration**: Uses the Prow/Gangway API to trigger the job.
- **Job Status Monitoring**: Provides job ID and status information.

### Run Command Environment Variables

| Variable | Required | Description |
|----------|----------|-------------|
| `PROW_API_TOKEN` | Yes | Prow API authentication token |

## Generated Test Matrix
## Update Templates

The script generates configuration for 5 test scenarios:

Expand All @@ -212,7 +214,7 @@ Each test includes:
## Output and Next Steps

### Generated Files
- **File Name**: `openshift-sandboxed-containers-operator-devel__downstream-{PROW_RUN_TYPE}{OCP_VERSION}.yaml`
- **File Name**: `openshift-sandboxed-containers-operator-devel__downstream-candidate{OCP**_VERSION**}.yaml` or `openshift-sandboxed-containers-operator-devel__downstream-release-{OCP**_VERSION**}.yaml`
- **Location**: Current directory
- **Backup**: Existing files are backed up with `.backup` extension

Expand Down Expand Up @@ -283,25 +285,3 @@ yq eval '.' openshift-sandboxed-containers-operator-devel__downstream-candidate4
- **Required**: `curl`, `jq`, `awk`, `sort`, `head`
- **Optional**: `yq` (for YAML syntax validation)
- **Network**: Access to `quay.io` API endpoints

## Prow API Token

To trigger ProwJobs via the REST API, you need an authentication token. Each SSO user is entitled to obtain a personal authentication token.

### Obtaining a Token

Tokens can be retrieved through the UI of the app.ci cluster at [OpenShift Console](https://console-openshift-console.apps.ci.l2s4.p1.openshiftapps.com). Alternatively, if the app.ci cluster context is already configured, you may execute:

```bash
oc whoami -t
```

### Using the Token

Once you have obtained a token, set it as an environment variable:

```bash
export PROW_API_TOKEN=your_token_here
```

For complete information about triggering ProwJobs via REST, including permanent tokens for automation, see the [OpenShift CI documentation](https://docs.ci.openshift.org/docs/how-tos/triggering-prowjobs-via-rest/#obtaining-an-authentication-token).
Loading