diff --git a/.cursor/rules/use-makefile.mdc b/.cursor/rules/use-makefile.mdc new file mode 100644 index 0000000..7f88d01 --- /dev/null +++ b/.cursor/rules/use-makefile.mdc @@ -0,0 +1,8 @@ +--- +description: Prefer Makefile targets for project commands +alwaysApply: true +--- + +# Use Makefile Commands + +Run all project operations through `make` targets (`make build`, `make shell`, `make run CMD="..."`) instead of invoking `podman`, `terraform`, or other CLIs directly. diff --git a/.cursor/skills/provision-rosa-aws/SKILL.md b/.cursor/skills/provision-rosa-aws/SKILL.md new file mode 100644 index 0000000..bc4bcee --- /dev/null +++ b/.cursor/skills/provision-rosa-aws/SKILL.md @@ -0,0 +1,126 @@ +--- +name: provision-rosa-aws +description: >- + Provision a ROSA HCP cluster on AWS using the containerized provisioner and + Terraform. Use when the user asks to deploy, provision, create, or spin up a + ROSA cluster, or mentions AWS provisioning, make build, make shell, or + terraform apply for this project. +disable-model-invocation: true +--- + +# Provision ROSA HCP on AWS + +## Prerequisites + +- AWS credentials (Access Key ID, Secret Access Key, Account ID, Region) +- Red Hat OCM token from https://console.redhat.com/openshift/token +- HuggingFace token for model access +- Podman installed on the host + +## Workflow + +### Step 1: Configure credentials + +Update `.env`: + +``` +AWS_ACCESS_KEY_ID= +AWS_SECRET_ACCESS_KEY= +AWS_ACCOUNT_ID= +AWS_DEFAULT_REGION=us-east-2 +RHCS_TOKEN= +HUGGINGFACE_TOKEN= +CLUSTER_ADMIN_PASSWORD= +``` + +Update `PCA_Deployment_ROSA/terraform/terraform.tfvars`: +- Set `aws_account_id`, `rhcs_token`, `huggingface_token`, `cluster_admin_password` +- Adjust cluster config as needed (name, version, instance types) + +### Step 2: Build the provisioner container + +```bash +make build +``` + +### Step 3: Run Terraform + +```bash +make run CMD="bash -c 'cd PCA_Deployment_ROSA/terraform && terraform init'" +make run CMD="bash -c 'cd PCA_Deployment_ROSA/terraform && terraform plan -out=tfplan'" +make run CMD="bash -c 'cd PCA_Deployment_ROSA/terraform && terraform apply tfplan'" +``` + +If container name conflicts: `podman rm -f pca` + +### Step 4: Verify cluster access + +Terraform uses `rosa create admin` (via OCM API) to create the cluster-admin user. After apply completes, wait 2-5 minutes for IDP propagation: + +```bash +make run CMD="bash -c 'cd PCA_Deployment_ROSA/terraform && terraform output -raw api_url'" +make run CMD="oc login --username cluster-admin --password '' --insecure-skip-tls-verify" +make run CMD="oc get nodes" +``` + +> **Note**: The admin IDP takes 2-5 minutes to propagate on ROSA HCP. Terraform waits 120s before attempting the grant. + +## Common Issues + +### Duplicate cluster name + +OCM rejects cluster creation if the name already exists in your org. Either choose a different `cluster_name` in tfvars or delete the old cluster first. + +### Invalid number of compute nodes + +`default_worker_replicas` must be a multiple of the number of private subnets (default: 3). E.g. use 3, 6, 9. + +### IAM roles already exist (EntityAlreadyExists) + +Account-level roles may exist from a prior deployment. Import them: + +```bash +terraform import 'module.account_iam_resources.aws_iam_role.account_role[0]' ManagedOpenShift-HCP-ROSA-Installer-Role +terraform import 'module.account_iam_resources.aws_iam_role.account_role[1]' ManagedOpenShift-HCP-ROSA-Support-Role +terraform import 'module.account_iam_resources.aws_iam_role.account_role[2]' ManagedOpenShift-HCP-ROSA-Worker-Role +``` + +### OAuth 500 — identity mapping conflict + +The username `cluster-admin` is reserved for `rosa create admin`. **Never** add it to custom HTPasswd IDPs. If you hit a 500 with "cannot be claimed by identity", fix via OCM API: + +```bash +rosa delete admin --cluster= --yes +# Create IDP with mapping_method "add" to bypass stale mapping: +ACCESS_TOKEN=$(curl -s -X POST "https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token" \ + -d "grant_type=refresh_token&client_id=cloud-services&refresh_token=$RHCS_TOKEN" | jq -r .access_token) +curl -X POST "https://api.openshift.com/api/clusters_mgmt/v1/clusters//identity_providers" \ + -H "Authorization: Bearer $ACCESS_TOKEN" -H "Content-Type: application/json" \ + -d '{"type":"HTPasswdIdentityProvider","name":"cluster-admin","mapping_method":"add","htpasswd":{"users":{"items":[{"username":"cluster-admin","password":""}]}}}' +# Then grant cluster-admin group: +curl -X POST "https://api.openshift.com/api/clusters_mgmt/v1/clusters//groups/cluster-admins/users" \ + -H "Authorization: Bearer $ACCESS_TOKEN" -H "Content-Type: application/json" \ + -d '{"id":"cluster-admin"}' +``` + +### OCM 500 error during cluster creation + +Transient server error. Wait a minute and re-run `terraform apply`. + +## Timing + +| Phase | Duration | +|-------|----------| +| VPC + Networking + IAM | ~2 min | +| ROSA HCP Cluster | 25-40 min | +| GPU Machine Pool + IDP | ~30 sec | +| IDP propagation + cluster-admin grant | ~2-5 min | +| **Total** | **~30-48 min** | + +## Verification + +```bash +rosa describe cluster --cluster= +oc login --username cluster-admin --password +oc get nodes +``` diff --git a/.cursor/skills/teardown-rosa-aws/SKILL.md b/.cursor/skills/teardown-rosa-aws/SKILL.md new file mode 100644 index 0000000..d44d043 --- /dev/null +++ b/.cursor/skills/teardown-rosa-aws/SKILL.md @@ -0,0 +1,134 @@ +--- +name: teardown-rosa-aws +description: >- + Tear down a ROSA HCP cluster on AWS, clean up leftover resources, and verify + complete removal. Use when the user asks to teardown, destroy, delete, remove, + or clean up a ROSA cluster, or mentions terraform destroy, make run destroy, + or AWS resource cleanup for this project. +disable-model-invocation: true +--- + +# Tear Down ROSA HCP on AWS + +## Prerequisites + +- `.env` with valid AWS credentials and `RHCS_TOKEN` +- Provisioner container image built (`make build`) +- Terraform state present in `PCA_Deployment_ROSA/terraform/` + +## Workflow + +### Step 1: Confirm terraform state has resources + +```bash +make run CMD="bash -c 'cd /workspace/PCA_Deployment_ROSA/terraform && terraform state list'" +``` + +If empty, skip to **Step 5: Verify** to check for orphaned AWS resources. + +### Step 2: Run terraform destroy + +```bash +make run CMD="bash -c 'cd /workspace/PCA_Deployment_ROSA/terraform && terraform init -upgrade && terraform destroy -auto-approve'" +``` + +This typically takes 15-30 minutes. Monitor for completion. + +**If destroy completes successfully**, skip to **Step 5: Verify**. + +### Step 3: Handle stuck VPC deletion + +`terraform destroy` commonly gets stuck on VPC resources (subnets, internet gateway) because ROSA leaves behind unmanaged resources. If the IGW or subnets are stuck destroying for >5 minutes: + +1. Kill the stuck container: `podman kill pca; podman rm pca` + +2. Get the VPC ID from state: + ```bash + make run CMD="bash -c 'cd /workspace/PCA_Deployment_ROSA/terraform && terraform state show aws_vpc.rosa[0]'" + ``` + +3. Run the cleanup script (substitute `VPC_ID` and `REGION`): + ```bash + make run CMD="bash /workspace/.cursor/skills/teardown-rosa-aws/scripts/cleanup-vpc.sh " + ``` + + This script removes, in order: + - ELBv2 and classic load balancers + - VPC endpoints + - EC2 instances (terminates and waits) + - ENIs (detach + delete) + - Non-default security groups + - Target groups + +4. Re-run terraform destroy: + ```bash + make run CMD="bash -c 'cd /workspace/PCA_Deployment_ROSA/terraform && terraform destroy -auto-approve'" + ``` + +### Step 4: Clean up orphaned OIDC providers + +ROSA may leave behind OIDC providers from current or previous deployments: + +```bash +make run CMD="bash -c 'aws iam list-open-id-connect-providers --query \"OpenIDConnectProviderList[?contains(Arn, \\\`openshiftapps.com\\\`)].Arn\" --output text'" +``` + +Delete any found: + +```bash +make run CMD="bash -c 'aws iam delete-open-id-connect-provider --open-id-connect-provider-arn '" +``` + +### Step 5: Verify complete cleanup + +Run the verification script (reads cluster name and region from `terraform.tfvars`): + +```bash +make run CMD="bash /workspace/.cursor/skills/teardown-rosa-aws/scripts/verify-cleanup.sh" +``` + +All 10 checks must pass. If any fail, delete the offending resources manually and re-verify. + +### Step 6: Confirm terraform state is empty + +```bash +make run CMD="bash -c 'cd /workspace/PCA_Deployment_ROSA/terraform && terraform state list'" +``` + +Should return no output. + +## Common Issues + +### Container name conflict + +```bash +podman kill pca; podman rm pca +``` + +### Subnets stuck deleting + +Usually caused by leftover ENIs from EC2 instances. Check for running instances: + +```bash +make run CMD="bash -c 'aws ec2 describe-instances --region --filters Name=vpc-id,Values= Name=instance-state-name,Values=running,pending,stopping,stopped --query \"Reservations[].Instances[].{ID:InstanceId,Name:Tags[?Key==\\\`Name\\\`].Value|[0]}\" --output table'" +``` + +Terminate them and wait before retrying. + +### IGW stuck deleting + +Usually caused by leftover NLBs. Check: + +```bash +make run CMD="bash -c 'aws elbv2 describe-load-balancers --region --query \"LoadBalancers[?VpcId==\\\`\\\`].LoadBalancerArn\" --output text'" +``` + +## Timing + +| Phase | Duration | +|-------|----------| +| Terraform destroy (cluster + IAM) | 10-20 min | +| VPC cleanup (if stuck) | 5-10 min | +| VPC destroy (after cleanup) | ~15 sec | +| Verification | ~15 sec | +| **Total** | **~15-30 min** | diff --git a/.cursor/skills/teardown-rosa-aws/scripts/cleanup-vpc.sh b/.cursor/skills/teardown-rosa-aws/scripts/cleanup-vpc.sh new file mode 100755 index 0000000..de06afe --- /dev/null +++ b/.cursor/skills/teardown-rosa-aws/scripts/cleanup-vpc.sh @@ -0,0 +1,95 @@ +#!/bin/bash +set -euo pipefail + +VPC_ID="${1:?Usage: cleanup-vpc.sh }" +REGION="${2:?Usage: cleanup-vpc.sh }" + +echo "=== Cleaning up leftover resources in VPC $VPC_ID ($REGION) ===" + +echo "--- Deleting ELBv2 load balancers..." +LB_ARNS=$(aws elbv2 describe-load-balancers --region "$REGION" \ + --query "LoadBalancers[?VpcId==\`$VPC_ID\`].LoadBalancerArn" --output text) +for arn in $LB_ARNS; do + echo " Deleting LB: $arn" + aws elbv2 delete-load-balancer --region "$REGION" --load-balancer-arn "$arn" +done + +echo "--- Deleting classic ELBs..." +CLB_NAMES=$(aws elb describe-load-balancers --region "$REGION" \ + --query "LoadBalancerDescriptions[?VPCId==\`$VPC_ID\`].LoadBalancerName" --output text) +for name in $CLB_NAMES; do + echo " Deleting classic LB: $name" + aws elb delete-load-balancer --region "$REGION" --load-balancer-name "$name" +done + +echo "--- Deleting VPC endpoints..." +VPCE_IDS=$(aws ec2 describe-vpc-endpoints --region "$REGION" \ + --filters "Name=vpc-id,Values=$VPC_ID" \ + --query "VpcEndpoints[].VpcEndpointId" --output text) +for vpce in $VPCE_IDS; do + echo " Deleting VPC endpoint: $vpce" + aws ec2 delete-vpc-endpoints --region "$REGION" --vpc-endpoint-ids "$vpce" +done + +echo "--- Terminating EC2 instances..." +INSTANCE_IDS=$(aws ec2 describe-instances --region "$REGION" \ + --filters "Name=vpc-id,Values=$VPC_ID" "Name=instance-state-name,Values=running,pending,stopping,stopped" \ + --query "Reservations[].Instances[].InstanceId" --output text) +if [ -n "$INSTANCE_IDS" ]; then + echo " Terminating: $INSTANCE_IDS" + aws ec2 terminate-instances --region "$REGION" --instance-ids $INSTANCE_IDS >/dev/null + echo " Waiting for termination..." + aws ec2 wait instance-terminated --region "$REGION" --instance-ids $INSTANCE_IDS + echo " All instances terminated." +fi + +echo "Waiting 15s for ENIs to release..." +sleep 15 + +echo "--- Detaching and deleting ENIs..." +ENI_IDS=$(aws ec2 describe-network-interfaces --region "$REGION" \ + --filters "Name=vpc-id,Values=$VPC_ID" \ + --query "NetworkInterfaces[].NetworkInterfaceId" --output text) +for eni in $ENI_IDS; do + ATT_ID=$(aws ec2 describe-network-interfaces --region "$REGION" \ + --network-interface-ids "$eni" \ + --query "NetworkInterfaces[0].Attachment.AttachmentId" --output text 2>/dev/null || echo "None") + if [ "$ATT_ID" != "None" ] && [ -n "$ATT_ID" ]; then + echo " Detaching $eni..." + aws ec2 detach-network-interface --region "$REGION" --attachment-id "$ATT_ID" --force 2>&1 || true + fi +done +sleep 10 +ENI_IDS=$(aws ec2 describe-network-interfaces --region "$REGION" \ + --filters "Name=vpc-id,Values=$VPC_ID" \ + --query "NetworkInterfaces[].NetworkInterfaceId" --output text) +for eni in $ENI_IDS; do + echo " Deleting ENI: $eni" + aws ec2 delete-network-interface --region "$REGION" --network-interface-id "$eni" 2>&1 || true +done + +echo "--- Deleting non-default security groups..." +SG_IDS=$(aws ec2 describe-security-groups --region "$REGION" \ + --filters "Name=vpc-id,Values=$VPC_ID" \ + --query 'SecurityGroups[?GroupName!=`default`].GroupId' --output text) +for sg in $SG_IDS; do + echo " Deleting SG: $sg" + aws ec2 delete-security-group --region "$REGION" --group-id "$sg" 2>&1 || true +done + +echo "--- Deleting target groups..." +TG_ARNS=$(aws elbv2 describe-target-groups --region "$REGION" \ + --query "TargetGroups[?VpcId==\`$VPC_ID\`].TargetGroupArn" --output text) +for tg in $TG_ARNS; do + echo " Deleting TG: $tg" + aws elbv2 delete-target-group --region "$REGION" --target-group-arn "$tg" 2>&1 || true +done + +echo "" +echo "--- Remaining ENIs:" +aws ec2 describe-network-interfaces --region "$REGION" \ + --filters "Name=vpc-id,Values=$VPC_ID" \ + --query "NetworkInterfaces[].{ID:NetworkInterfaceId,Status:Status}" --output table 2>/dev/null || echo " None" + +echo "" +echo "=== VPC cleanup complete ===" diff --git a/.cursor/skills/teardown-rosa-aws/scripts/verify-cleanup.sh b/.cursor/skills/teardown-rosa-aws/scripts/verify-cleanup.sh new file mode 100755 index 0000000..0624409 --- /dev/null +++ b/.cursor/skills/teardown-rosa-aws/scripts/verify-cleanup.sh @@ -0,0 +1,85 @@ +#!/bin/bash +set -euo pipefail + +TFVARS="/workspace/PCA_Deployment_ROSA/terraform/terraform.tfvars" + +if [ ! -f "$TFVARS" ]; then + echo "ERROR: $TFVARS not found. Pass CLUSTER_NAME and REGION as args." + echo "Usage: verify-cleanup.sh [cluster_name] [region]" + exit 1 +fi + +CLUSTER_NAME="${1:-$(grep '^cluster_name' "$TFVARS" | sed 's/.*= *"\(.*\)"/\1/')}" +REGION="${2:-$(grep '^aws_region' "$TFVARS" | sed 's/.*= *"\(.*\)"/\1/')}" + +FAIL=0 + +check() { + local label="$1" + local result="$2" + if [ -z "$result" ] || [ "$result" = "None" ]; then + echo " [OK] $label" + else + echo " [FAIL] $label: $result" + FAIL=1 + fi +} + +echo "=== Verifying teardown of cluster '$CLUSTER_NAME' in $REGION ===" +echo "" + +echo "1. VPC" +check "VPC" "$(aws ec2 describe-vpcs --region "$REGION" \ + --filters "Name=tag:Name,Values=${CLUSTER_NAME}-vpc" \ + --query "Vpcs[].VpcId" --output text)" + +echo "2. EC2 instances" +check "EC2" "$(aws ec2 describe-instances --region "$REGION" \ + --filters "Name=tag:Name,Values=${CLUSTER_NAME}*" "Name=instance-state-name,Values=running,pending,stopping,stopped" \ + --query "Reservations[].Instances[].InstanceId" --output text)" + +echo "3. HCP ROSA IAM roles" +check "IAM roles" "$(aws iam list-roles \ + --query 'Roles[?starts_with(RoleName, `ManagedOpenShift-HCP-ROSA-`)].RoleName' --output text)" + +echo "4. Operator IAM roles" +check "Operator roles" "$(aws iam list-roles \ + --query "Roles[?starts_with(RoleName, \`${CLUSTER_NAME}-\`)].RoleName" --output text)" + +echo "5. OIDC providers" +check "OIDC" "$(aws iam list-open-id-connect-providers \ + --query 'OpenIDConnectProviderList[?contains(Arn, `openshiftapps.com`)].Arn' --output text)" + +echo "6. Load balancers" +check "ELBv2" "$(aws elbv2 describe-load-balancers --region "$REGION" \ + --query "LoadBalancers[?contains(LoadBalancerName, \`${CLUSTER_NAME}\`) || contains(LoadBalancerName, \`k8s\`)].LoadBalancerName" --output text)" + +echo "7. NAT gateways" +check "NAT" "$(aws ec2 describe-nat-gateways --region "$REGION" \ + --filter "Name=tag:Name,Values=${CLUSTER_NAME}-nat" \ + --query 'NatGateways[?State!=`deleted`].NatGatewayId' --output text)" + +echo "8. Elastic IPs" +check "EIP" "$(aws ec2 describe-addresses --region "$REGION" \ + --filters "Name=tag:Name,Values=${CLUSTER_NAME}-nat-eip" \ + --query "Addresses[].AllocationId" --output text)" + +echo "9. Internet gateways" +check "IGW" "$(aws ec2 describe-internet-gateways --region "$REGION" \ + --filters "Name=tag:Name,Values=${CLUSTER_NAME}-igw" \ + --query "InternetGateways[].InternetGatewayId" --output text)" + +echo "10. Subnets" +check "Subnets" "$(aws ec2 describe-subnets --region "$REGION" \ + --filters "Name=tag:Name,Values=${CLUSTER_NAME}-*" \ + --query "Subnets[].SubnetId" --output text)" + +echo "" +echo "=====================================" +if [ "$FAIL" -eq 0 ]; then + echo "ALL CHECKS PASSED" +else + echo "SOME CHECKS FAILED (see above)" +fi +echo "=====================================" +exit $FAIL diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..a8bf4d9 --- /dev/null +++ b/.env.example @@ -0,0 +1,24 @@ +# ────────────────────────────────────────────── +# Private AI Code Assistant — Credentials +# Copy to .env and fill in your values. +# NEVER commit .env to version control. +# ────────────────────────────────────────────── + +# AWS Credentials +AWS_ACCESS_KEY_ID= +AWS_SECRET_ACCESS_KEY= +AWS_ACCOUNT_ID= +AWS_DEFAULT_REGION=us-east-2 + +# Red Hat OCM / ROSA Token (offline refresh token) +# Get yours at: https://console.redhat.com/openshift/token +RHCS_TOKEN= + +# HuggingFace Token +HUGGINGFACE_TOKEN= + +# Cluster Admin Password +CLUSTER_ADMIN_PASSWORD= + +# Azure (for ARO deployments) +# ARM_SUBSCRIPTION_ID= diff --git a/.github/workflows/pre-commit-autofix.yml b/.github/workflows/pre-commit-autofix.yml new file mode 100644 index 0000000..a4cc2e6 --- /dev/null +++ b/.github/workflows/pre-commit-autofix.yml @@ -0,0 +1,35 @@ +name: Pre-commit Autofix +on: + workflow_dispatch: +permissions: + contents: write + pull-requests: write +jobs: + autofix: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + - uses: actions/setup-go@v5 + with: + go-version: '1.22' + - run: go install github.com/google/yamlfmt/cmd/yamlfmt@v0.9.0 + - uses: pre-commit/action@v3.0.1 + id: pre-commit + continue-on-error: true + with: + extra_args: --all-files + - if: steps.pre-commit.outcome == 'failure' + uses: peter-evans/create-pull-request@v7 + with: + token: ${{ secrets.GITHUB_TOKEN }} + commit-message: "style: auto-fix pre-commit issues" + branch: pre-commit-autofix + title: "style: auto-fix pre-commit issues" + body: | + Automated PR created by the **Pre-commit Autofix** workflow. + + This applies all fixes from `pre-commit run --all-files`. + delete-branch: true diff --git a/.github/workflows/pre-commit-ci.yml b/.github/workflows/pre-commit-ci.yml new file mode 100644 index 0000000..a09918d --- /dev/null +++ b/.github/workflows/pre-commit-ci.yml @@ -0,0 +1,21 @@ +name: Pre-commit CI +on: + pull_request: + branches: [main] + push: + branches: [main] +jobs: + pre-commit: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + - uses: actions/setup-go@v5 + with: + go-version: "1.22" + - run: go install github.com/google/yamlfmt/cmd/yamlfmt@v0.9.0 + - uses: pre-commit/action@v3.0.1 + with: + extra_args: --all-files diff --git a/.gitignore b/.gitignore index 7e1e087..9a17f97 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,10 @@ assets/V1/ handoff.md +.env +terraform.tfvars +*.tfstate +*.tfstate.backup +.terraform/ +.terraform.lock.hcl +*.tfplan +tfplan diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 0000000..06186a3 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,27 @@ +repos: + - repo: https://github.com/adrienverge/yamllint + rev: v1.37.1 + hooks: + - id: yamllint + args: ["-d", "{extends: relaxed, rules: {line-length: {max: 250}, key-duplicates: enable}}"] + exclude: ^PCA_Deployment_.*/terraform/ + - repo: https://github.com/google/yamlfmt + rev: v0.9.0 + hooks: + - id: yamlfmt + exclude: ^PCA_Deployment_.*/terraform/ + - repo: https://github.com/pre-commit/pre-commit-hooks + rev: v5.0.0 + hooks: + - id: end-of-file-fixer + - id: detect-aws-credentials + args: ["--allow-missing-credentials"] + - id: detect-private-key + - repo: https://github.com/scop/pre-commit-shfmt + rev: v3.9.0-1 + hooks: + - id: shfmt + - repo: https://github.com/gitleaks/gitleaks + rev: v8.24.2 + hooks: + - id: gitleaks diff --git a/Containerfile.provisioner b/Containerfile.provisioner new file mode 100644 index 0000000..9a315a3 --- /dev/null +++ b/Containerfile.provisioner @@ -0,0 +1,53 @@ +FROM registry.fedoraproject.org/fedora:43 + +ARG TERRAFORM_VERSION=1.15.7 +ARG AWSCLI_VERSION=2.35.12 +ARG AZURE_CLI_VERSION=2.81.0 +ARG ROSA_VERSION=1.2.53 +ARG OC_VERSION=4.21.21 + +RUN dnf install -y \ + git \ + jq \ + curl \ + unzip \ + findutils \ + python3-pip \ + which \ + && dnf clean all + +RUN curl -fsSL "https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip" \ + -o /tmp/terraform.zip && \ + unzip /tmp/terraform.zip -d /usr/local/bin && \ + rm /tmp/terraform.zip && \ + chmod +x /usr/local/bin/terraform + +RUN curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64-${AWSCLI_VERSION}.zip" \ + -o /tmp/awscli.zip && \ + unzip -q /tmp/awscli.zip -d /tmp && \ + /tmp/aws/install && \ + rm -rf /tmp/awscli.zip /tmp/aws + +RUN rpm --import https://packages.microsoft.com/keys/microsoft.asc && \ + dnf install -y https://packages.microsoft.com/config/fedora/43/packages-microsoft-prod.rpm && \ + dnf install -y azure-cli-0:${AZURE_CLI_VERSION}-*.fc43 && \ + dnf clean all + +RUN curl -fsSL "https://mirror.openshift.com/pub/openshift-v4/clients/rosa/${ROSA_VERSION}/rosa-linux.tar.gz" \ + -o /tmp/rosa.tar.gz && \ + tar --no-same-owner -xzf /tmp/rosa.tar.gz -C /usr/local/bin rosa && \ + rm /tmp/rosa.tar.gz && \ + chmod +x /usr/local/bin/rosa + +RUN curl -fsSL "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/${OC_VERSION}/openshift-client-linux-${OC_VERSION}.tar.gz" \ + -o /tmp/oc.tar.gz && \ + tar --no-same-owner -xzf /tmp/oc.tar.gz -C /usr/local/bin oc kubectl && \ + rm /tmp/oc.tar.gz && \ + chmod +x /usr/local/bin/oc /usr/local/bin/kubectl + +RUN useradd -m -u 1000 pca + +USER pca +WORKDIR /workspace + +CMD ["/bin/bash"] diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..6afd826 --- /dev/null +++ b/Makefile @@ -0,0 +1,33 @@ +IMAGE_NAME ?= pca-provisioner +CONTAINER_NAME ?= pca +CONTAINERFILE ?= Containerfile.provisioner +PROJECT_DIR := $(shell pwd) + +ENV_FILE_FLAG := $(if $(wildcard .env),--env-file .env,) +AWS_MOUNT := $(if $(wildcard $(HOME)/.aws),-v $(HOME)/.aws:/home/pca/.aws:ro,) +AZURE_MOUNT := $(if $(wildcard $(HOME)/.azure),-v $(HOME)/.azure:/home/pca/.azure:ro,) +KUBE_MOUNT := $(if $(wildcard $(HOME)/.kube),-v $(HOME)/.kube:/home/pca/.kube:ro,) + +RUN_FLAGS := --rm \ + --user 0:0 \ + --name $(CONTAINER_NAME) \ + -v $(PROJECT_DIR):/workspace:Z \ + $(AWS_MOUNT) \ + $(AZURE_MOUNT) \ + $(KUBE_MOUNT) \ + $(ENV_FILE_FLAG) + +.PHONY: build shell run help + +help: ## Show this help + @grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | \ + awk 'BEGIN {FS = ":.*?## "}; {printf " \033[36m%-12s\033[0m %s\n", $$1, $$2}' + +build: ## Build the provisioner container image + podman build -t $(IMAGE_NAME) -f $(CONTAINERFILE) . + +shell: ## Start an interactive shell inside the container + podman run -it $(RUN_FLAGS) $(IMAGE_NAME) + +run: ## Run a one-shot command (usage: make run CMD="terraform plan") + podman run $(RUN_FLAGS) $(IMAGE_NAME) $(CMD) diff --git a/PCA_Deployment_ARO/argocd/00-app-of-apps.yaml b/PCA_Deployment_ARO/argocd/00-app-of-apps.yaml index 3f8f077..33e17c2 100644 --- a/PCA_Deployment_ARO/argocd/00-app-of-apps.yaml +++ b/PCA_Deployment_ARO/argocd/00-app-of-apps.yaml @@ -1,7 +1,6 @@ # Root App-of-Apps: ArgoCD discovers child Applications in this directory. # Each child Application points to a subdirectory of manifests. # Sync waves ensure correct ordering: operators → platform → workloads. ---- apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: diff --git a/PCA_Deployment_ARO/argocd/01-operators/cert-manager.yaml b/PCA_Deployment_ARO/argocd/01-operators/cert-manager.yaml index bc17020..b14a0a3 100644 --- a/PCA_Deployment_ARO/argocd/01-operators/cert-manager.yaml +++ b/PCA_Deployment_ARO/argocd/01-operators/cert-manager.yaml @@ -1,6 +1,5 @@ # cert-manager — required by KServe for webhook certificates # Installed via kustomize overlay from llm-d-playbook ---- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: diff --git a/PCA_Deployment_ARO/argocd/01-operators/leader-worker-set.yaml b/PCA_Deployment_ARO/argocd/01-operators/leader-worker-set.yaml index aba2e06..014d33e 100644 --- a/PCA_Deployment_ARO/argocd/01-operators/leader-worker-set.yaml +++ b/PCA_Deployment_ARO/argocd/01-operators/leader-worker-set.yaml @@ -1,5 +1,4 @@ # LeaderWorkerSet Operator — required by KServe for multi-node inference ---- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: diff --git a/PCA_Deployment_ARO/argocd/01-operators/lws-operator-cr.yaml b/PCA_Deployment_ARO/argocd/01-operators/lws-operator-cr.yaml index 6a2cb71..e5aa1d3 100644 --- a/PCA_Deployment_ARO/argocd/01-operators/lws-operator-cr.yaml +++ b/PCA_Deployment_ARO/argocd/01-operators/lws-operator-cr.yaml @@ -1,6 +1,5 @@ # LeaderWorkerSet Operator instance — enables the LWS controller. # Applied after the LWS CRD is installed (wave 1). ---- apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSetOperator metadata: diff --git a/PCA_Deployment_ARO/argocd/01-operators/nvidia-cluster-policy.yaml b/PCA_Deployment_ARO/argocd/01-operators/nvidia-cluster-policy.yaml index d4c2ca7..bed09be 100644 --- a/PCA_Deployment_ARO/argocd/01-operators/nvidia-cluster-policy.yaml +++ b/PCA_Deployment_ARO/argocd/01-operators/nvidia-cluster-policy.yaml @@ -2,7 +2,6 @@ # useOpenShiftDriverToolkit: true is required for ARO to use the Red Hat # Driver Toolkit (DTK) for in-cluster kernel module compilation, avoiding # the need for pre-compiled drivers or privileged node access. ---- apiVersion: nvidia.com/v1 kind: ClusterPolicy metadata: diff --git a/PCA_Deployment_ARO/argocd/01-operators/subscriptions.yaml b/PCA_Deployment_ARO/argocd/01-operators/subscriptions.yaml index 74380ff..19242db 100644 --- a/PCA_Deployment_ARO/argocd/01-operators/subscriptions.yaml +++ b/PCA_Deployment_ARO/argocd/01-operators/subscriptions.yaml @@ -1,6 +1,5 @@ # All operator Subscriptions managed by ArgoCD. # Sync wave 1: these must install before platform config (wave 2) uses their CRDs. ---- # Red Hat OpenShift AI (RHOAI) 3.3 — AI Gateway (llm-d) is GA at v0.4 apiVersion: v1 kind: Namespace diff --git a/PCA_Deployment_ARO/argocd/02-platform-config/checluster.yaml b/PCA_Deployment_ARO/argocd/02-platform-config/checluster.yaml index 5fb43cc..6d033eb 100644 --- a/PCA_Deployment_ARO/argocd/02-platform-config/checluster.yaml +++ b/PCA_Deployment_ARO/argocd/02-platform-config/checluster.yaml @@ -1,7 +1,6 @@ # OpenShift Dev Spaces instance — creates the dashboard, workspace controller, registries. # Configured with always-active workspaces (idle timeout disabled) to support # long-running OpenCode sessions without interruption. ---- apiVersion: org.eclipse.che/v2 kind: CheCluster metadata: diff --git a/PCA_Deployment_ARO/argocd/02-platform-config/datasciencecluster.yaml b/PCA_Deployment_ARO/argocd/02-platform-config/datasciencecluster.yaml index 88bdbb5..9aad5b6 100644 --- a/PCA_Deployment_ARO/argocd/02-platform-config/datasciencecluster.yaml +++ b/PCA_Deployment_ARO/argocd/02-platform-config/datasciencecluster.yaml @@ -1,6 +1,5 @@ # Patch DataScienceCluster for KServe "Headed" mode (required for llm-d + Gateway API) # The DSC is created by the RHOAI operator; this resource ensures the correct configuration. ---- apiVersion: datasciencecluster.opendatahub.io/v1 kind: DataScienceCluster metadata: diff --git a/PCA_Deployment_ARO/argocd/02-platform-config/hf-token-placeholder.yaml b/PCA_Deployment_ARO/argocd/02-platform-config/hf-token-placeholder.yaml index 78b59c9..eb680bd 100644 --- a/PCA_Deployment_ARO/argocd/02-platform-config/hf-token-placeholder.yaml +++ b/PCA_Deployment_ARO/argocd/02-platform-config/hf-token-placeholder.yaml @@ -1,7 +1,6 @@ # HuggingFace token secret — PLACEHOLDER # Replace REPLACE_WITH_BASE64_HF_TOKEN with: echo -n 'hf_your_token' | base64 # For production, use External Secrets Operator or Sealed Secrets instead. ---- apiVersion: v1 kind: Secret metadata: diff --git a/PCA_Deployment_ARO/argocd/02-platform-config/namespaces.yaml b/PCA_Deployment_ARO/argocd/02-platform-config/namespaces.yaml index a978e43..d2488b6 100644 --- a/PCA_Deployment_ARO/argocd/02-platform-config/namespaces.yaml +++ b/PCA_Deployment_ARO/argocd/02-platform-config/namespaces.yaml @@ -3,7 +3,6 @@ # NOTE: DevSpaces user namespaces are NOT defined here. DevSpaces # auto-provisions unique namespaces per user (e.g., dev1-devspaces-wk1ug6) # when they first log into the dashboard. See setup-devspaces-users.sh. ---- apiVersion: v1 kind: Namespace metadata: diff --git a/PCA_Deployment_ARO/argocd/02-platform-config/rbac.yaml b/PCA_Deployment_ARO/argocd/02-platform-config/rbac.yaml index ce09747..76d37cf 100644 --- a/PCA_Deployment_ARO/argocd/02-platform-config/rbac.yaml +++ b/PCA_Deployment_ARO/argocd/02-platform-config/rbac.yaml @@ -9,7 +9,6 @@ # # Users Dev1 and Dev2 are created via HTPasswd identity provider by the # setup-devspaces-users.sh script. ---- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: diff --git a/PCA_Deployment_ARO/argocd/03-ai-serving/inference-routing.yaml b/PCA_Deployment_ARO/argocd/03-ai-serving/inference-routing.yaml index d12d143..ac7aa22 100644 --- a/PCA_Deployment_ARO/argocd/03-ai-serving/inference-routing.yaml +++ b/PCA_Deployment_ARO/argocd/03-ai-serving/inference-routing.yaml @@ -9,7 +9,6 @@ # # To scale: simply increase InferenceService replicas (or add HPA). # The InferencePool automatically discovers new pods via label selector. ---- apiVersion: inference.networking.k8s.io/v1 kind: InferencePool metadata: diff --git a/PCA_Deployment_ARO/argocd/03-ai-serving/llm-d-epp.yaml b/PCA_Deployment_ARO/argocd/03-ai-serving/llm-d-epp.yaml index df42e07..1c16419 100644 --- a/PCA_Deployment_ARO/argocd/03-ai-serving/llm-d-epp.yaml +++ b/PCA_Deployment_ARO/argocd/03-ai-serving/llm-d-epp.yaml @@ -14,7 +14,6 @@ # In single-replica mode, the EPP passes through to the only available pod. # When scaled to multiple replicas, EPP provides prefix-aware scheduling # that significantly improves throughput. ---- apiVersion: v1 kind: ServiceAccount metadata: diff --git a/PCA_Deployment_ARO/argocd/03-ai-serving/llm-d-gateway.yaml b/PCA_Deployment_ARO/argocd/03-ai-serving/llm-d-gateway.yaml index 980ed22..4cc2e03 100644 --- a/PCA_Deployment_ARO/argocd/03-ai-serving/llm-d-gateway.yaml +++ b/PCA_Deployment_ARO/argocd/03-ai-serving/llm-d-gateway.yaml @@ -11,7 +11,6 @@ # # Internal DNS endpoint: # https://llm-d-gateway-data-science-gateway-class.ai-serving.svc.cluster.local ---- apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: diff --git a/PCA_Deployment_ARO/argocd/03-ai-serving/llminferenceservice.yaml b/PCA_Deployment_ARO/argocd/03-ai-serving/llminferenceservice.yaml index 0f6c4e3..e63903e 100644 --- a/PCA_Deployment_ARO/argocd/03-ai-serving/llminferenceservice.yaml +++ b/PCA_Deployment_ARO/argocd/03-ai-serving/llminferenceservice.yaml @@ -46,7 +46,6 @@ # Mistral / Mixtral mistral (none) 0.14+ # # ───────────────────────────────────────────────────────────────────────────── ---- apiVersion: infrastructure.opendatahub.io/v1 kind: HardwareProfile metadata: @@ -192,11 +191,11 @@ metadata: openshift.io/display-name: "Qwen 3.6-35B-A3B-FP8" serving.kserve.io/deploymentMode: RawDeployment opendatahub.io/accelerator-name: nvidia.com/gpu - # Scaling: increase maxReplicas and add GPU nodes to scale out. - # Each replica requires 1x H100 GPU. The InferencePool + EPP - # automatically discovers new replicas and routes intelligently. - # To scale: oc scale machineset --replicas=N - # then update maxReplicas below to match. + # Scaling: increase maxReplicas and add GPU nodes to scale out. + # Each replica requires 1x H100 GPU. The InferencePool + EPP + # automatically discovers new replicas and routes intelligently. + # To scale: oc scale machineset --replicas=N + # then update maxReplicas below to match. spec: predictor: minReplicas: 1 diff --git a/PCA_Deployment_ARO/argocd/03-ai-serving/pvcs.yaml b/PCA_Deployment_ARO/argocd/03-ai-serving/pvcs.yaml index d8c2c93..60bfa9a 100644 --- a/PCA_Deployment_ARO/argocd/03-ai-serving/pvcs.yaml +++ b/PCA_Deployment_ARO/argocd/03-ai-serving/pvcs.yaml @@ -1,7 +1,6 @@ # Persistent Volume Claims for model weight caching. # Using PVCs avoids re-downloading models on pod restarts (~30GB for Qwen3.6-35B-A3B-FP8). # Azure managed-csi (default ARO storage class) replaces AWS gp3-csi. ---- apiVersion: v1 kind: PersistentVolumeClaim metadata: diff --git a/PCA_Deployment_ARO/argocd/03-ai-serving/tls-secret-job.yaml b/PCA_Deployment_ARO/argocd/03-ai-serving/tls-secret-job.yaml index 43c275b..8eaad27 100644 --- a/PCA_Deployment_ARO/argocd/03-ai-serving/tls-secret-job.yaml +++ b/PCA_Deployment_ARO/argocd/03-ai-serving/tls-secret-job.yaml @@ -1,6 +1,5 @@ # Job to generate a self-signed TLS certificate for the llm-d Gateway. # Runs once after the namespace is created. ---- apiVersion: batch/v1 kind: Job metadata: diff --git a/PCA_Deployment_ARO/argocd/04-devspaces/devspaces-dashboard-samples.yaml b/PCA_Deployment_ARO/argocd/04-devspaces/devspaces-dashboard-samples.yaml index 259d208..55f7019 100644 --- a/PCA_Deployment_ARO/argocd/04-devspaces/devspaces-dashboard-samples.yaml +++ b/PCA_Deployment_ARO/argocd/04-devspaces/devspaces-dashboard-samples.yaml @@ -1,6 +1,5 @@ # DevSpaces dashboard configuration — custom samples and default image. # Makes the OpenCode workspace visible on the DevSpaces landing page. ---- apiVersion: v1 kind: ConfigMap metadata: diff --git a/PCA_Deployment_ARO/argocd/04-devspaces/devworkspaces.yaml b/PCA_Deployment_ARO/argocd/04-devspaces/devworkspaces.yaml index 31c58fd..a2d51db 100644 --- a/PCA_Deployment_ARO/argocd/04-devspaces/devworkspaces.yaml +++ b/PCA_Deployment_ARO/argocd/04-devspaces/devworkspaces.yaml @@ -33,7 +33,6 @@ # # Model endpoint (routed through llm-d EPP → vLLM): # https://llm-d-gateway-data-science-gateway-class.ai-serving.svc.cluster.local/v1 ---- apiVersion: workspace.devfile.io/v1alpha2 kind: DevWorkspace metadata: diff --git a/PCA_Deployment_ARO/argocd/04-devspaces/opencode-image-build.yaml b/PCA_Deployment_ARO/argocd/04-devspaces/opencode-image-build.yaml index b2b7b72..60119b6 100644 --- a/PCA_Deployment_ARO/argocd/04-devspaces/opencode-image-build.yaml +++ b/PCA_Deployment_ARO/argocd/04-devspaces/opencode-image-build.yaml @@ -4,7 +4,6 @@ # The BuildConfig creates the image from assets/Dockerfile.opencode # and stores it in the internal registry. DevWorkspaces reference # this image for OpenCode-enabled workspaces. ---- apiVersion: v1 kind: Namespace metadata: diff --git a/PCA_Deployment_ARO/argocd/04-devspaces/roo-code-configmaps.yaml b/PCA_Deployment_ARO/argocd/04-devspaces/roo-code-configmaps.yaml index 93ee44d..624d4a2 100644 --- a/PCA_Deployment_ARO/argocd/04-devspaces/roo-code-configmaps.yaml +++ b/PCA_Deployment_ARO/argocd/04-devspaces/roo-code-configmaps.yaml @@ -4,7 +4,6 @@ # global config pointing to the cluster-internal llm-d AI gateway. # OpenCode also reads OPENCODE_CONFIG_CONTENT env var (set in DevWorkspace) # as a runtime override, so this ConfigMap serves as a belt-and-suspenders backup. ---- apiVersion: v1 kind: ConfigMap metadata: diff --git a/PCA_Deployment_ARO/argocd/04-devspaces/vscode-extensions-config.yaml b/PCA_Deployment_ARO/argocd/04-devspaces/vscode-extensions-config.yaml index 1e6264b..4b8149a 100644 --- a/PCA_Deployment_ARO/argocd/04-devspaces/vscode-extensions-config.yaml +++ b/PCA_Deployment_ARO/argocd/04-devspaces/vscode-extensions-config.yaml @@ -5,7 +5,6 @@ # See: https://eclipse.dev/che/docs/stable/administration-guide/default-extensions-for-microsoft-visual-studio-code/ # This auto-install mechanism is configured in devworkspaces.yaml and # setup-devspaces-users.sh via DEFAULT_EXTENSIONS + postStart download. ---- apiVersion: v1 kind: ConfigMap metadata: diff --git a/PCA_Deployment_ARO/argocd/05-benchmarks/guidellm-sweep.yaml b/PCA_Deployment_ARO/argocd/05-benchmarks/guidellm-sweep.yaml index cbc7496..b117b40 100644 --- a/PCA_Deployment_ARO/argocd/05-benchmarks/guidellm-sweep.yaml +++ b/PCA_Deployment_ARO/argocd/05-benchmarks/guidellm-sweep.yaml @@ -1,7 +1,6 @@ # GuideLLM benchmark sweep for Qwen3.6-35B-A3B-FP8 on NVIDIA H100 NVL. # Runs multiple concurrency levels across code-assistant-relevant workloads. # Results are stored in a PVC and printed to Job logs for extraction. ---- apiVersion: v1 kind: PersistentVolumeClaim metadata: diff --git a/PCA_Deployment_ARO/scripts/create-gpu-machineset.sh b/PCA_Deployment_ARO/scripts/create-gpu-machineset.sh index 9454f15..af848fc 100755 --- a/PCA_Deployment_ARO/scripts/create-gpu-machineset.sh +++ b/PCA_Deployment_ARO/scripts/create-gpu-machineset.sh @@ -22,13 +22,16 @@ GREEN='\033[0;32m' YELLOW='\033[1;33m' NC='\033[0m' -info() { echo -e "${GREEN}[INFO]${NC} $*"; } -warn() { echo -e "${YELLOW}[WARN]${NC} $*"; } -error() { echo -e "${RED}[ERROR]${NC} $*" >&2; exit 1; } +info() { echo -e "${GREEN}[INFO]${NC} $*"; } +warn() { echo -e "${YELLOW}[WARN]${NC} $*"; } +error() { + echo -e "${RED}[ERROR]${NC} $*" >&2 + exit 1 +} # ── Prerequisites ───────────────────────────────────────────────────────────── -command -v oc &>/dev/null || error "'oc' CLI not found. Install from https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/" -command -v jq &>/dev/null || error "'jq' not found. Install with: brew install jq or dnf install jq" +command -v oc &>/dev/null || error "'oc' CLI not found. Install from https://mirror.openshift.com/pub/openshift-v4/clients/ocp/latest/" +command -v jq &>/dev/null || error "'jq' not found. Install with: brew install jq or dnf install jq" # Verify cluster access oc whoami &>/dev/null || error "Not logged in to OpenShift. Run 'oc login' first." @@ -43,31 +46,31 @@ CLUSTER_REGION=$(oc get infrastructure cluster -o jsonpath='{.status.platformSta info "Cluster region: ${LOCATION}" # Get the resource group from the cluster if not provided -if [[ -z "${RESOURCE_GROUP}" ]]; then - RESOURCE_GROUP=$(oc get infrastructure cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') - info "Detected resource group: ${RESOURCE_GROUP}" +if [[ -z ${RESOURCE_GROUP} ]]; then + RESOURCE_GROUP=$(oc get infrastructure cluster -o jsonpath='{.status.platformStatus.azure.resourceGroupName}') + info "Detected resource group: ${RESOURCE_GROUP}" fi # ── Clone an existing worker MachineSet as template ─────────────────────────── TEMPLATE_MS=$(oc get machineset -n openshift-machine-api -o name | grep "worker" | head -1) -[[ -z "${TEMPLATE_MS}" ]] && error "No worker MachineSet found to clone. Ensure the cluster has at least one worker MachineSet." +[[ -z ${TEMPLATE_MS} ]] && error "No worker MachineSet found to clone. Ensure the cluster has at least one worker MachineSet." info "Using template MachineSet: ${TEMPLATE_MS}" # Derive MachineSet name from VM SKU (a100 or h100) case "${GPU_VM_SIZE}" in - *H100*|*h100*) GPU_TAG="h100" ;; - *A100*|*a100*) GPU_TAG="a100" ;; - *) GPU_TAG="gpu" ;; +*H100* | *h100*) GPU_TAG="h100" ;; +*A100* | *a100*) GPU_TAG="a100" ;; +*) GPU_TAG="gpu" ;; esac GPU_MS_NAME="${INFRA_ID}-gpu-${GPU_TAG}" TEMPLATE_JSON=$(oc get "${TEMPLATE_MS}" -n openshift-machine-api -o json) # ── Build GPU MachineSet manifest ───────────────────────────────────────────── GPU_MS_JSON=$(echo "${TEMPLATE_JSON}" | jq --arg name "${GPU_MS_NAME}" \ - --arg vm_size "${GPU_VM_SIZE}" \ - --argjson replicas "${REPLICAS}" \ - ' + --arg vm_size "${GPU_VM_SIZE}" \ + --argjson replicas "${REPLICAS}" \ + ' # Strip runtime-managed fields del(.metadata.resourceVersion, .metadata.uid, .metadata.creationTimestamp, .status, .metadata.annotations["kubectl.kubernetes.io/last-applied-configuration"]) @@ -100,34 +103,34 @@ GPU_MS_JSON=$(echo "${TEMPLATE_JSON}" | jq --arg name "${GPU_MS_NAME}" \ # ── Apply the GPU MachineSet ────────────────────────────────────────────────── if oc get machineset "${GPU_MS_NAME}" -n openshift-machine-api &>/dev/null; then - warn "MachineSet ${GPU_MS_NAME} already exists. Applying update..." - echo "${GPU_MS_JSON}" | oc apply -f - + warn "MachineSet ${GPU_MS_NAME} already exists. Applying update..." + echo "${GPU_MS_JSON}" | oc apply -f - else - info "Creating GPU MachineSet ${GPU_MS_NAME}..." - echo "${GPU_MS_JSON}" | oc apply -f - + info "Creating GPU MachineSet ${GPU_MS_NAME}..." + echo "${GPU_MS_JSON}" | oc apply -f - fi # ── Wait for machines to be provisioned ─────────────────────────────────────── info "Waiting for GPU MachineSet to scale up (this takes 5-15 minutes for A100 VMs)..." for i in $(seq 1 90); do - READY=$(oc get machineset "${GPU_MS_NAME}" -n openshift-machine-api \ - -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0") - DESIRED=$(oc get machineset "${GPU_MS_NAME}" -n openshift-machine-api \ - -o jsonpath='{.spec.replicas}' 2>/dev/null || echo "${REPLICAS}") - if [[ "${READY}" == "${DESIRED}" && "${READY}" != "0" ]]; then - info "GPU MachineSet is ready: ${READY}/${DESIRED} nodes." - break - fi - echo " Waiting for GPU nodes... ready=${READY}/${DESIRED} (attempt ${i}/90)" - sleep 20 + READY=$(oc get machineset "${GPU_MS_NAME}" -n openshift-machine-api \ + -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0") + DESIRED=$(oc get machineset "${GPU_MS_NAME}" -n openshift-machine-api \ + -o jsonpath='{.spec.replicas}' 2>/dev/null || echo "${REPLICAS}") + if [[ ${READY} == "${DESIRED}" && ${READY} != "0" ]]; then + info "GPU MachineSet is ready: ${READY}/${DESIRED} nodes." + break + fi + echo " Waiting for GPU nodes... ready=${READY}/${DESIRED} (attempt ${i}/90)" + sleep 20 done FINAL_READY=$(oc get machineset "${GPU_MS_NAME}" -n openshift-machine-api \ - -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0") -if [[ "${FINAL_READY}" != "${REPLICAS}" ]]; then - warn "GPU MachineSet may still be provisioning. Check with:" - warn " oc get machineset ${GPU_MS_NAME} -n openshift-machine-api" - warn " oc get machines -n openshift-machine-api -l machine.openshift.io/cluster-api-machineset=${GPU_MS_NAME}" + -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0") +if [[ ${FINAL_READY} != "${REPLICAS}" ]]; then + warn "GPU MachineSet may still be provisioning. Check with:" + warn " oc get machineset ${GPU_MS_NAME} -n openshift-machine-api" + warn " oc get machines -n openshift-machine-api -l machine.openshift.io/cluster-api-machineset=${GPU_MS_NAME}" fi info "GPU MachineSet ${GPU_MS_NAME} created successfully." diff --git a/PCA_Deployment_ARO/scripts/deploy-full-stack.sh b/PCA_Deployment_ARO/scripts/deploy-full-stack.sh index c86c674..45b3e95 100755 --- a/PCA_Deployment_ARO/scripts/deploy-full-stack.sh +++ b/PCA_Deployment_ARO/scripts/deploy-full-stack.sh @@ -15,10 +15,13 @@ YELLOW='\033[1;33m' CYAN='\033[0;36m' NC='\033[0m' -info() { echo -e "${GREEN}[INFO]${NC} $*"; } -warn() { echo -e "${YELLOW}[WARN]${NC} $*"; } -step() { echo -e "${CYAN}[STEP]${NC} $*"; } -error() { echo -e "${RED}[ERROR]${NC} $*" >&2; exit 1; } +info() { echo -e "${GREEN}[INFO]${NC} $*"; } +warn() { echo -e "${YELLOW}[WARN]${NC} $*"; } +step() { echo -e "${CYAN}[STEP]${NC} $*"; } +error() { + echo -e "${RED}[ERROR]${NC} $*" >&2 + exit 1 +} oc whoami &>/dev/null || error "Not logged in to OpenShift. Run 'oc login' first." info "Deploying full PCA stack from: ${BASE_PATH}" @@ -32,29 +35,29 @@ oc apply -f "${BASE_PATH}/01-operators/leader-worker-set.yaml" 2>&1 | sed 's/^/ info "Waiting for operator CSVs to reach Succeeded (up to 15 min)..." for i in $(seq 1 90); do - SUCCEEDED=$(oc get csv -A --no-headers 2>/dev/null | grep -c "Succeeded" 2>/dev/null) || SUCCEEDED=0 - PENDING=$(oc get csv -A --no-headers 2>/dev/null | grep -c "Pending\|Installing" 2>/dev/null) || PENDING=0 + SUCCEEDED=$(oc get csv -A --no-headers 2>/dev/null | grep -c "Succeeded" 2>/dev/null) || SUCCEEDED=0 + PENDING=$(oc get csv -A --no-headers 2>/dev/null | grep -c "Pending\|Installing" 2>/dev/null) || PENDING=0 - if [[ "${PENDING}" -eq 0 && "${SUCCEEDED}" -gt 0 ]]; then - info "All operators installed: ${SUCCEEDED} CSVs in Succeeded state." - break - fi - echo " Operators: ${SUCCEEDED} succeeded, ${PENDING} pending (attempt ${i}/90)" - sleep 10 + if [[ ${PENDING} -eq 0 && ${SUCCEEDED} -gt 0 ]]; then + info "All operators installed: ${SUCCEEDED} CSVs in Succeeded state." + break + fi + echo " Operators: ${SUCCEEDED} succeeded, ${PENDING} pending (attempt ${i}/90)" + sleep 10 done # ── Wave 1b: Operator CRs (require CRDs from installed operators) ──────────── step "Wave 1b: Applying operator custom resources..." for cr_attempt in $(seq 1 12); do - FAILED=0 - oc apply -f "${BASE_PATH}/01-operators/nvidia-cluster-policy.yaml" 2>&1 | sed 's/^/ /' || FAILED=1 - oc apply -f "${BASE_PATH}/01-operators/lws-operator-cr.yaml" 2>&1 | sed 's/^/ /' || FAILED=1 - if [[ "${FAILED}" == "0" ]]; then - info "All operator CRs applied successfully." - break - fi - echo " Some CRDs not ready yet, retrying... (${cr_attempt}/12)" - sleep 15 + FAILED=0 + oc apply -f "${BASE_PATH}/01-operators/nvidia-cluster-policy.yaml" 2>&1 | sed 's/^/ /' || FAILED=1 + oc apply -f "${BASE_PATH}/01-operators/lws-operator-cr.yaml" 2>&1 | sed 's/^/ /' || FAILED=1 + if [[ ${FAILED} == "0" ]]; then + info "All operator CRs applied successfully." + break + fi + echo " Some CRDs not ready yet, retrying... (${cr_attempt}/12)" + sleep 15 done # ── Wave 2: Platform Config ────────────────────────────────────────────────── @@ -63,14 +66,14 @@ oc apply -f "${BASE_PATH}/02-platform-config/" 2>&1 | sed 's/^/ /' info "Waiting for CheCluster to become available..." for i in $(seq 1 60); do - CHE_STATUS=$(oc get checluster devspaces -n openshift-devspaces \ - -o jsonpath='{.status.chePhase}' 2>/dev/null || echo "NotReady") - if [[ "${CHE_STATUS}" == "Active" ]]; then - info "DevSpaces CheCluster is Active." - break - fi - echo " CheCluster phase: ${CHE_STATUS} (attempt ${i}/60)" - sleep 15 + CHE_STATUS=$(oc get checluster devspaces -n openshift-devspaces \ + -o jsonpath='{.status.chePhase}' 2>/dev/null || echo "NotReady") + if [[ ${CHE_STATUS} == "Active" ]]; then + info "DevSpaces CheCluster is Active." + break + fi + echo " CheCluster phase: ${CHE_STATUS} (attempt ${i}/60)" + sleep 15 done # ── Wave 3: AI Serving ─────────────────────────────────────────────────────── @@ -79,14 +82,14 @@ oc apply -f "${BASE_PATH}/03-ai-serving/" 2>&1 | sed 's/^/ /' info "Waiting for model to be ready (this can take 10-20 min for model download)..." for i in $(seq 1 120); do - MODEL_READY=$(oc get llminferenceservice qwen36-35b -n ai-serving \ - -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo "False") - if [[ "${MODEL_READY}" == "True" ]]; then - info "Model qwen36-35b is ready and serving." - break - fi - echo " Model status: not ready yet (attempt ${i}/120)" - sleep 15 + MODEL_READY=$(oc get llminferenceservice qwen36-35b -n ai-serving \ + -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo "False") + if [[ ${MODEL_READY} == "True" ]]; then + info "Model qwen36-35b is ready and serving." + break + fi + echo " Model status: not ready yet (attempt ${i}/120)" + sleep 15 done # ── Wave 4: DevSpaces ──────────────────────────────────────────────────────── @@ -95,12 +98,12 @@ step "Wave 4: Applying DevSpaces config (image build, dashboard samples, extensi # DevWorkspaces must be created PER USER via setup-devspaces-users.sh so the # controller.devfile.io/creator label is set correctly for dashboard visibility. for f in "${BASE_PATH}/04-devspaces/"*.yaml; do - FNAME=$(basename "$f") - if [[ "${FNAME}" == "devworkspaces.yaml" ]]; then - echo " Skipping ${FNAME} (template only — use setup-devspaces-users.sh)" - continue - fi - oc apply -f "$f" 2>&1 | sed 's/^/ /' + FNAME=$(basename "$f") + if [[ ${FNAME} == "devworkspaces.yaml" ]]; then + echo " Skipping ${FNAME} (template only — use setup-devspaces-users.sh)" + continue + fi + oc apply -f "$f" 2>&1 | sed 's/^/ /' done info "DevSpaces config applied." diff --git a/PCA_Deployment_ARO/scripts/post-terraform-fullstack.sh b/PCA_Deployment_ARO/scripts/post-terraform-fullstack.sh index 5755b60..94097cc 100755 --- a/PCA_Deployment_ARO/scripts/post-terraform-fullstack.sh +++ b/PCA_Deployment_ARO/scripts/post-terraform-fullstack.sh @@ -17,10 +17,13 @@ YELLOW='\033[1;33m' CYAN='\033[0;36m' NC='\033[0m' -info() { echo -e "${GREEN}[INFO]${NC} $*"; } -warn() { echo -e "${YELLOW}[WARN]${NC} $*"; } -step() { echo -e "${CYAN}[STEP]${NC} $*"; } -error() { echo -e "${RED}[ERROR]${NC} $*" >&2; exit 1; } +info() { echo -e "${GREEN}[INFO]${NC} $*"; } +warn() { echo -e "${YELLOW}[WARN]${NC} $*"; } +step() { echo -e "${CYAN}[STEP]${NC} $*"; } +error() { + echo -e "${RED}[ERROR]${NC} $*" >&2 + exit 1 +} oc whoami &>/dev/null || error "Not logged in to OpenShift. Run 'oc login' first." info "Connected to: $(oc whoami --show-server)" @@ -33,60 +36,60 @@ chmod +x "${SCRIPT_DIR}/deploy-full-stack.sh" # ── Step 1b: Trigger OpenCode custom image build ────────────────────── step "Step 1b: Triggering OpenCode DevSpaces image build..." for i in $(seq 1 30); do - if oc get buildconfig devspaces-opencode -n opencode-build &>/dev/null; then - oc start-build devspaces-opencode -n opencode-build 2>/dev/null || true - info "OpenCode image build started (runs in background)." - break - fi - echo " Waiting for BuildConfig to appear... (${i}/30)" - sleep 10 + if oc get buildconfig devspaces-opencode -n opencode-build &>/dev/null; then + oc start-build devspaces-opencode -n opencode-build 2>/dev/null || true + info "OpenCode image build started (runs in background)." + break + fi + echo " Waiting for BuildConfig to appear... (${i}/30)" + sleep 10 done # ── Step 1c: Create DevSpaces users and workspaces ──────────────────── step "Step 1c: Setting up DevSpaces users and workspaces..." if [[ -x "${SCRIPT_DIR}/setup-devspaces-users.sh" ]]; then - # Wait for image build to complete before creating workspaces - info "Waiting for OpenCode image build (up to 10 min)..." - for i in $(seq 1 60); do - BUILD_PHASE=$(oc get build -n opencode-build -l buildconfig=devspaces-opencode \ - --sort-by='.metadata.creationTimestamp' \ - -o jsonpath='{.items[-1].status.phase}' 2>/dev/null || echo "Unknown") - if [[ "${BUILD_PHASE}" == "Complete" ]]; then - info "OpenCode image build complete." - break - fi - echo " Build phase: ${BUILD_PHASE} (${i}/60)" - sleep 10 - done - "${SCRIPT_DIR}/setup-devspaces-users.sh" + # Wait for image build to complete before creating workspaces + info "Waiting for OpenCode image build (up to 10 min)..." + for i in $(seq 1 60); do + BUILD_PHASE=$(oc get build -n opencode-build -l buildconfig=devspaces-opencode \ + --sort-by='.metadata.creationTimestamp' \ + -o jsonpath='{.items[-1].status.phase}' 2>/dev/null || echo "Unknown") + if [[ ${BUILD_PHASE} == "Complete" ]]; then + info "OpenCode image build complete." + break + fi + echo " Build phase: ${BUILD_PHASE} (${i}/60)" + sleep 10 + done + "${SCRIPT_DIR}/setup-devspaces-users.sh" else - warn "setup-devspaces-users.sh not found. Run it manually after deployment." + warn "setup-devspaces-users.sh not found. Run it manually after deployment." fi # ── Step 2: Wait for GPU node to be ready ────────────────────────────── step "Step 2: Waiting for H100 GPU node..." for i in $(seq 1 90); do - GPU_NODES=$(oc get nodes -l nvidia.com/gpu.present=true --no-headers 2>/dev/null | wc -l | tr -d ' ') - if [[ "${GPU_NODES}" -ge 1 ]]; then - info "H100 GPU node is ready." - oc get nodes -l nvidia.com/gpu.present=true - break - fi - echo " Waiting for GPU node... (${i}/90)" - sleep 20 + GPU_NODES=$(oc get nodes -l nvidia.com/gpu.present=true --no-headers 2>/dev/null | wc -l | tr -d ' ') + if [[ ${GPU_NODES} -ge 1 ]]; then + info "H100 GPU node is ready." + oc get nodes -l nvidia.com/gpu.present=true + break + fi + echo " Waiting for GPU node... (${i}/90)" + sleep 20 done # ── Step 3: Ensure model is serving ─────────────────────────────────── step "Step 3: Waiting for Qwen model to be fully serving..." for i in $(seq 1 120); do - MODEL_READY=$(oc get llminferenceservice qwen36-35b -n ai-serving \ - -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo "False") - if [[ "${MODEL_READY}" == "True" ]]; then - info "Model qwen36-35b is ready and serving." - break - fi - echo " Model status: not ready (${i}/120)" - sleep 15 + MODEL_READY=$(oc get llminferenceservice qwen36-35b -n ai-serving \ + -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo "False") + if [[ ${MODEL_READY} == "True" ]]; then + info "Model qwen36-35b is ready and serving." + break + fi + echo " Model status: not ready (${i}/120)" + sleep 15 done # ── Step 4: Run GuideLLM sweep ──────────────────────────────────────── @@ -95,25 +98,25 @@ oc apply -f "${BASE_PATH}/05-benchmarks/guidellm-sweep.yaml" info "Waiting for GuideLLM job to complete (this takes 15-30 min)..." oc wait --for=condition=Complete job/guidellm-sweep-h100 -n ai-serving --timeout=3600s 2>/dev/null || { - warn "GuideLLM job timed out or failed. Checking status..." - oc get job guidellm-sweep-h100 -n ai-serving - oc get pods -n ai-serving -l app=guidellm + warn "GuideLLM job timed out or failed. Checking status..." + oc get job guidellm-sweep-h100 -n ai-serving + oc get pods -n ai-serving -l app=guidellm } # ── Step 5: Extract results ─────────────────────────────────────────── step "Step 5: Extracting GuideLLM results..." GUIDELLM_POD=$(oc get pods -n ai-serving -l app=guidellm --sort-by='.metadata.creationTimestamp' \ - -o jsonpath='{.items[-1].metadata.name}' 2>/dev/null || echo "") + -o jsonpath='{.items[-1].metadata.name}' 2>/dev/null || echo "") -if [[ -z "${GUIDELLM_POD}" ]]; then - warn "No GuideLLM pod found. Results extraction skipped." - exit 0 +if [[ -z ${GUIDELLM_POD} ]]; then + warn "No GuideLLM pod found. Results extraction skipped." + exit 0 fi info "Extracting logs from pod: ${GUIDELLM_POD}" LOGS=$(oc logs "${GUIDELLM_POD}" -n ai-serving 2>/dev/null || echo "Failed to get logs") -cat > "${RESULTS_FILE}" << HEADER +cat >"${RESULTS_FILE}" <
Generated: $(date -u '+%Y-%m-%d %H:%M UTC') > Cluster: aro-pca-aue (Australia East) diff --git a/PCA_Deployment_ARO/scripts/setup-devspaces-users.sh b/PCA_Deployment_ARO/scripts/setup-devspaces-users.sh index d3ae420..5db32c9 100755 --- a/PCA_Deployment_ARO/scripts/setup-devspaces-users.sh +++ b/PCA_Deployment_ARO/scripts/setup-devspaces-users.sh @@ -24,8 +24,8 @@ set -euo pipefail # ── User definitions ────────────────────────────────────────────────── # Format: "username:password" USERS=( - "Dev1:Dev1@PCA2026!" - "Dev2:Dev2@PCA2026!" + "Dev1:Dev1@PCA2026!" + "Dev2:Dev2@PCA2026!" ) SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" @@ -39,17 +39,20 @@ YELLOW='\033[1;33m' CYAN='\033[0;36m' NC='\033[0m' -info() { echo -e "${GREEN}[INFO]${NC} $*"; } -warn() { echo -e "${YELLOW}[WARN]${NC} $*"; } -step() { echo -e "${CYAN}[STEP]${NC} $*"; } -error() { echo -e "${RED}[ERROR]${NC} $*" >&2; exit 1; } +info() { echo -e "${GREEN}[INFO]${NC} $*"; } +warn() { echo -e "${YELLOW}[WARN]${NC} $*"; } +step() { echo -e "${CYAN}[STEP]${NC} $*"; } +error() { + echo -e "${RED}[ERROR]${NC} $*" >&2 + exit 1 +} API_URL=$(oc whoami --show-server 2>/dev/null) || error "Not logged in. Run 'oc login' first." info "Connected to: ${API_URL}" # Verify running as cluster-admin -oc auth can-i create oauth --all-namespaces &>/dev/null || \ - error "Must be logged in as cluster-admin (kubeadmin)." +oc auth can-i create oauth --all-namespaces &>/dev/null || + error "Must be logged in as cluster-admin (kubeadmin)." # ── Step 1: Create HTPasswd IDP ─────────────────────────────────────── step "Step 1: Creating HTPasswd identity provider..." @@ -59,20 +62,20 @@ trap "rm -f ${HTPASSWD_FILE}" EXIT FIRST=true for entry in "${USERS[@]}"; do - USERNAME="${entry%%:*}" - PASSWORD="${entry#*:}" - if $FIRST; then - htpasswd -cbB "${HTPASSWD_FILE}" "${USERNAME}" "${PASSWORD}" - FIRST=false - else - htpasswd -bB "${HTPASSWD_FILE}" "${USERNAME}" "${PASSWORD}" - fi + USERNAME="${entry%%:*}" + PASSWORD="${entry#*:}" + if $FIRST; then + htpasswd -cbB "${HTPASSWD_FILE}" "${USERNAME}" "${PASSWORD}" + FIRST=false + else + htpasswd -bB "${HTPASSWD_FILE}" "${USERNAME}" "${PASSWORD}" + fi done oc create secret generic htpass-secret \ - --from-file=htpasswd="${HTPASSWD_FILE}" \ - -n openshift-config \ - --dry-run=client -o yaml | oc apply -f - + --from-file=htpasswd="${HTPASSWD_FILE}" \ + -n openshift-config \ + --dry-run=client -o yaml | oc apply -f - oc patch oauth cluster --type=merge -p '{ "spec": { @@ -94,70 +97,70 @@ oc patch oauth cluster --type=merge -p '{ info "Waiting for OAuth pods to restart..." sleep 10 for i in $(seq 1 30); do - READY=$(oc get pods -n openshift-authentication -l app=oauth-openshift \ - --no-headers 2>/dev/null | grep -c "1/1.*Running" || echo 0) - if [[ "${READY}" -ge 3 ]]; then - info "OAuth pods ready (${READY}/3)." - break - fi - echo " OAuth pods ready: ${READY}/3 (${i}/30)" - sleep 10 + READY=$(oc get pods -n openshift-authentication -l app=oauth-openshift \ + --no-headers 2>/dev/null | grep -c "1/1.*Running" || echo 0) + if [[ ${READY} -ge 3 ]]; then + info "OAuth pods ready (${READY}/3)." + break + fi + echo " OAuth pods ready: ${READY}/3 (${i}/30)" + sleep 10 done # ── Step 2: Verify user logins ──────────────────────────────────────── step "Step 2: Verifying user logins..." for entry in "${USERS[@]}"; do - USERNAME="${entry%%:*}" - PASSWORD="${entry#*:}" - if oc login "${API_URL}" --username="${USERNAME}" --password="${PASSWORD}" \ - --insecure-skip-tls-verify=true &>/dev/null; then - info "User ${USERNAME} login OK." - else - error "User ${USERNAME} login FAILED." - fi + USERNAME="${entry%%:*}" + PASSWORD="${entry#*:}" + if oc login "${API_URL}" --username="${USERNAME}" --password="${PASSWORD}" \ + --insecure-skip-tls-verify=true &>/dev/null; then + info "User ${USERNAME} login OK." + else + error "User ${USERNAME} login FAILED." + fi done # Switch back to kubeadmin -if [[ -n "${KUBEADMIN_PASS:-}" ]]; then - oc login "${API_URL}" --username=kubeadmin --password="${KUBEADMIN_PASS}" \ - --insecure-skip-tls-verify=true &>/dev/null +if [[ -n ${KUBEADMIN_PASS:-} ]]; then + oc login "${API_URL}" --username=kubeadmin --password="${KUBEADMIN_PASS}" \ + --insecure-skip-tls-verify=true &>/dev/null else - warn "KUBEADMIN_PASS not set. Attempting to continue..." - oc login "${API_URL}" --username=kubeadmin \ - --insecure-skip-tls-verify=true &>/dev/null 2>&1 || true + warn "KUBEADMIN_PASS not set. Attempting to continue..." + oc login "${API_URL}" --username=kubeadmin \ + --insecure-skip-tls-verify=true &>/dev/null 2>&1 || true fi # ── Step 3: Trigger DevSpaces namespace provisioning ────────────────── step "Step 3: Triggering DevSpaces namespace provisioning..." DEVSPACES_ROUTE=$(oc get route devspaces -n openshift-devspaces \ - -o jsonpath='{.spec.host}' 2>/dev/null || echo "") -if [[ -z "${DEVSPACES_ROUTE}" ]]; then - DEVSPACES_ROUTE=$(oc get checluster devspaces -n openshift-devspaces \ - -o jsonpath='{.status.cheURL}' 2>/dev/null | sed 's|https://||') + -o jsonpath='{.spec.host}' 2>/dev/null || echo "") +if [[ -z ${DEVSPACES_ROUTE} ]]; then + DEVSPACES_ROUTE=$(oc get checluster devspaces -n openshift-devspaces \ + -o jsonpath='{.status.cheURL}' 2>/dev/null | sed 's|https://||') fi for entry in "${USERS[@]}"; do - USERNAME="${entry%%:*}" - PASSWORD="${entry#*:}" + USERNAME="${entry%%:*}" + PASSWORD="${entry#*:}" - # Login as user and hit DevSpaces API to trigger namespace provisioning - oc login "${API_URL}" --username="${USERNAME}" --password="${PASSWORD}" \ - --insecure-skip-tls-verify=true &>/dev/null - TOKEN=$(oc whoami -t) + # Login as user and hit DevSpaces API to trigger namespace provisioning + oc login "${API_URL}" --username="${USERNAME}" --password="${PASSWORD}" \ + --insecure-skip-tls-verify=true &>/dev/null + TOKEN=$(oc whoami -t) - curl -sk "https://${DEVSPACES_ROUTE}/api/kubernetes/namespace" \ - -H "Authorization: Bearer ${TOKEN}" &>/dev/null || true + curl -sk "https://${DEVSPACES_ROUTE}/api/kubernetes/namespace" \ + -H "Authorization: Bearer ${TOKEN}" &>/dev/null || true - info "Triggered namespace provisioning for ${USERNAME}." - sleep 3 + info "Triggered namespace provisioning for ${USERNAME}." + sleep 3 done # Switch back to kubeadmin -if [[ -n "${KUBEADMIN_PASS:-}" ]]; then - oc login "${API_URL}" --username=kubeadmin --password="${KUBEADMIN_PASS}" \ - --insecure-skip-tls-verify=true &>/dev/null +if [[ -n ${KUBEADMIN_PASS:-} ]]; then + oc login "${API_URL}" --username=kubeadmin --password="${KUBEADMIN_PASS}" \ + --insecure-skip-tls-verify=true &>/dev/null fi # Wait for namespaces to appear @@ -168,20 +171,20 @@ sleep 10 step "Step 4: Creating workspaces in DevSpaces-managed namespaces..." for entry in "${USERS[@]}"; do - USERNAME="${entry%%:*}" - PASSWORD="${entry#*:}" - USERNAME_LOWER=$(echo "${USERNAME}" | tr '[:upper:]' '[:lower:]') - - # Find the DevSpaces-managed namespace for this user - USER_NS=$(oc get ns -l app.kubernetes.io/component=workspaces-namespace \ - -o jsonpath="{range .items[*]}{.metadata.name}{'\t'}{.metadata.annotations.che\.eclipse\.org/username}{'\n'}{end}" \ - 2>/dev/null | grep -i " ${USERNAME}$" | awk '{print $1}' | head -1) - - if [[ -z "${USER_NS}" ]]; then - warn "No DevSpaces namespace found for ${USERNAME}. Creating one manually..." - RAND=$(head -c 3 /dev/urandom | xxd -p | head -c 6) - USER_NS="${USERNAME_LOWER}-devspaces-${RAND}" - cat </dev/null | grep -i " ${USERNAME}$" | awk '{print $1}' | head -1) + + if [[ -z ${USER_NS} ]]; then + warn "No DevSpaces namespace found for ${USERNAME}. Creating one manually..." + RAND=$(head -c 3 /dev/urandom | xxd -p | head -c 6) + USER_NS="${USERNAME_LOWER}-devspaces-${RAND}" + cat </dev/null || true - fi + # Grant admin to the user + oc create rolebinding "${USERNAME_LOWER}-admin" \ + --clusterrole=admin --user="${USERNAME}" -n "${USER_NS}" 2>/dev/null || true + fi - info "User ${USERNAME} → namespace ${USER_NS}" + info "User ${USERNAME} → namespace ${USER_NS}" - # Grant image-puller for the custom OpenCode image - oc policy add-role-to-group system:image-puller \ - "system:serviceaccounts:${USER_NS}" \ - --namespace=opencode-build 2>/dev/null || true + # Grant image-puller for the custom OpenCode image + oc policy add-role-to-group system:image-puller \ + "system:serviceaccounts:${USER_NS}" \ + --namespace=opencode-build 2>/dev/null || true - # Login as the user and create the workspace - oc login "${API_URL}" --username="${USERNAME}" --password="${PASSWORD}" \ - --insecure-skip-tls-verify=true &>/dev/null + # Login as the user and create the workspace + oc login "${API_URL}" --username="${USERNAME}" --password="${PASSWORD}" \ + --insecure-skip-tls-verify=true &>/dev/null - cat </dev/null +if [[ -n ${KUBEADMIN_PASS:-} ]]; then + oc login "${API_URL}" --username=kubeadmin --password="${KUBEADMIN_PASS}" \ + --insecure-skip-tls-verify=true &>/dev/null fi # ── Step 5: Wait for workspaces ─────────────────────────────────────── step "Step 5: Waiting for workspaces to start..." for entry in "${USERS[@]}"; do - USERNAME="${entry%%:*}" - USERNAME_LOWER=$(echo "${USERNAME}" | tr '[:upper:]' '[:lower:]') - USER_NS=$(oc get ns -l app.kubernetes.io/component=workspaces-namespace \ - -o jsonpath="{range .items[*]}{.metadata.name}{'\t'}{.metadata.annotations.che\.eclipse\.org/username}{'\n'}{end}" \ - 2>/dev/null | grep -i " ${USERNAME}$" | awk '{print $1}' | head -1) - - for i in $(seq 1 30); do - STATUS=$(oc get devworkspace "opencode-${USERNAME_LOWER}" -n "${USER_NS}" \ - -o jsonpath='{.status.phase}' 2>/dev/null || echo "Unknown") - if [[ "${STATUS}" == "Running" ]]; then - URL=$(oc get devworkspace "opencode-${USERNAME_LOWER}" -n "${USER_NS}" \ - -o jsonpath='{.status.mainUrl}' 2>/dev/null) - info "${USERNAME}: Running → ${URL}" - break - fi - echo " ${USERNAME}: ${STATUS} (${i}/30)" - sleep 10 - done + USERNAME="${entry%%:*}" + USERNAME_LOWER=$(echo "${USERNAME}" | tr '[:upper:]' '[:lower:]') + USER_NS=$(oc get ns -l app.kubernetes.io/component=workspaces-namespace \ + -o jsonpath="{range .items[*]}{.metadata.name}{'\t'}{.metadata.annotations.che\.eclipse\.org/username}{'\n'}{end}" \ + 2>/dev/null | grep -i " ${USERNAME}$" | awk '{print $1}' | head -1) + + for i in $(seq 1 30); do + STATUS=$(oc get devworkspace "opencode-${USERNAME_LOWER}" -n "${USER_NS}" \ + -o jsonpath='{.status.phase}' 2>/dev/null || echo "Unknown") + if [[ ${STATUS} == "Running" ]]; then + URL=$(oc get devworkspace "opencode-${USERNAME_LOWER}" -n "${USER_NS}" \ + -o jsonpath='{.status.mainUrl}' 2>/dev/null) + info "${USERNAME}: Running → ${URL}" + break + fi + echo " ${USERNAME}: ${STATUS} (${i}/30)" + sleep 10 + done done echo "" @@ -325,13 +328,13 @@ info " DevSpaces User Setup Complete" info "═══════════════════════════════════════════════════════" echo "" DASHBOARD=$(oc get checluster devspaces -n openshift-devspaces \ - -o jsonpath='{.status.cheURL}' 2>/dev/null || echo "https://devspaces.") + -o jsonpath='{.status.cheURL}' 2>/dev/null || echo "https://devspaces.") info "Dashboard: ${DASHBOARD}" echo "" for entry in "${USERS[@]}"; do - USERNAME="${entry%%:*}" - PASSWORD="${entry#*:}" - info " ${USERNAME} / ${PASSWORD}" + USERNAME="${entry%%:*}" + PASSWORD="${entry#*:}" + info " ${USERNAME} / ${PASSWORD}" done echo "" info "Factory URL (users can also create workspaces by opening this in their browser):" diff --git a/PCA_Deployment_ARO/scripts/validate.sh b/PCA_Deployment_ARO/scripts/validate.sh index be2fee4..eb067d0 100755 --- a/PCA_Deployment_ARO/scripts/validate.sh +++ b/PCA_Deployment_ARO/scripts/validate.sh @@ -9,7 +9,10 @@ YELLOW='\033[1;33m' NC='\033[0m' pass() { echo -e "${GREEN}[PASS]${NC} $1"; } -fail() { echo -e "${RED}[FAIL]${NC} $1"; FAILURES=$((FAILURES + 1)); } +fail() { + echo -e "${RED}[FAIL]${NC} $1" + FAILURES=$((FAILURES + 1)) +} warn() { echo -e "${YELLOW}[WARN]${NC} $1"; } FAILURES=0 @@ -22,11 +25,11 @@ echo "" # 1. Operators echo "--- Checking Operators ---" for op in "rhods-operator" "devspaces" "servicemeshoperator3" "gpu-operator-certified" "openshift-gitops-operator"; do - if oc get csv -A 2>/dev/null | grep -q "${op}.*Succeeded"; then - pass "Operator ${op} installed and healthy" - else - fail "Operator ${op} not found or not Succeeded" - fi + if oc get csv -A 2>/dev/null | grep -q "${op}.*Succeeded"; then + pass "Operator ${op} installed and healthy" + else + fail "Operator ${op} not found or not Succeeded" + fi done # 2. DataScienceCluster @@ -34,54 +37,54 @@ echo "" echo "--- Checking DataScienceCluster ---" DSC_MODE=$(oc get datasciencecluster default-dsc -o jsonpath='{.spec.components.kserve.rawDeploymentServiceConfig}' 2>/dev/null || echo "NOT_FOUND") if [ "$DSC_MODE" = "Headed" ]; then - pass "DataScienceCluster rawDeploymentServiceConfig = Headed" + pass "DataScienceCluster rawDeploymentServiceConfig = Headed" else - fail "DataScienceCluster rawDeploymentServiceConfig = ${DSC_MODE} (expected: Headed)" + fail "DataScienceCluster rawDeploymentServiceConfig = ${DSC_MODE} (expected: Headed)" fi # 3. GPU Operator and A100 nodes echo "" echo "--- Checking GPU Stack (NVIDIA A100) ---" if oc get clusterpolicy gpu-cluster-policy &>/dev/null; then - pass "NVIDIA ClusterPolicy exists" + pass "NVIDIA ClusterPolicy exists" else - warn "NVIDIA ClusterPolicy not found (expected if no GPU nodes)" + warn "NVIDIA ClusterPolicy not found (expected if no GPU nodes)" fi GPU_NODES=$(oc get nodes -l nvidia.com/gpu.present=true --no-headers 2>/dev/null | wc -l || echo "0") if [ "${GPU_NODES}" -gt 0 ]; then - pass "GPU nodes found: ${GPU_NODES} node(s) with nvidia.com/gpu.present=true" - oc get nodes -l nvidia.com/gpu.present=true -o custom-columns='NAME:.metadata.name,TYPE:.metadata.labels.node\.kubernetes\.io/instance-type,STATUS:.status.conditions[-1].type' 2>/dev/null || true + pass "GPU nodes found: ${GPU_NODES} node(s) with nvidia.com/gpu.present=true" + oc get nodes -l nvidia.com/gpu.present=true -o custom-columns='NAME:.metadata.name,TYPE:.metadata.labels.node\.kubernetes\.io/instance-type,STATUS:.status.conditions[-1].type' 2>/dev/null || true else - warn "No GPU nodes detected yet. MachineSet may still be provisioning." - warn "Check: oc get machineset -n openshift-machine-api" + warn "No GPU nodes detected yet. MachineSet may still be provisioning." + warn "Check: oc get machineset -n openshift-machine-api" fi # 4. Storage class (Azure managed-csi) echo "" echo "--- Checking Storage ---" if oc get storageclass managed-csi &>/dev/null; then - pass "StorageClass managed-csi exists" + pass "StorageClass managed-csi exists" else - fail "StorageClass managed-csi not found (required for model-cache PVC)" + fail "StorageClass managed-csi not found (required for model-cache PVC)" fi PVC_STATUS=$(oc get pvc model-cache -n ai-serving -o jsonpath='{.status.phase}' 2>/dev/null || echo "NOT_FOUND") if [ "$PVC_STATUS" = "Bound" ]; then - pass "PVC model-cache is Bound" + pass "PVC model-cache is Bound" else - warn "PVC model-cache status = ${PVC_STATUS}" + warn "PVC model-cache status = ${PVC_STATUS}" fi # 5. Namespaces echo "" echo "--- Checking Namespaces ---" for ns in "ai-serving" "dev1-devspaces" "dev2-devspaces" "openshift-devspaces"; do - if oc get ns "${ns}" &>/dev/null; then - pass "Namespace ${ns} exists" - else - fail "Namespace ${ns} missing" - fi + if oc get ns "${ns}" &>/dev/null; then + pass "Namespace ${ns} exists" + else + fail "Namespace ${ns} missing" + fi done # 6. CheCluster @@ -89,9 +92,9 @@ echo "" echo "--- Checking Dev Spaces ---" CHE_STATUS=$(oc get checluster devspaces -n openshift-devspaces -o jsonpath='{.status.chePhase}' 2>/dev/null || echo "NOT_FOUND") if [ "$CHE_STATUS" = "Active" ]; then - pass "CheCluster is Active" + pass "CheCluster is Active" else - fail "CheCluster phase = ${CHE_STATUS} (expected: Active)" + fail "CheCluster phase = ${CHE_STATUS} (expected: Active)" fi # 7. Model Serving @@ -99,9 +102,9 @@ echo "" echo "--- Checking Model Serving ---" LIS_STATUS=$(oc get llminferenceservice qwen36-35b -n ai-serving -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo "NOT_FOUND") if [ "$LIS_STATUS" = "True" ]; then - pass "LLMInferenceService qwen36-35b is Ready" + pass "LLMInferenceService qwen36-35b is Ready" else - warn "LLMInferenceService qwen36-35b Ready = ${LIS_STATUS} (may be starting up)" + warn "LLMInferenceService qwen36-35b Ready = ${LIS_STATUS} (may be starting up)" fi # 8. Gateway @@ -109,36 +112,36 @@ echo "" echo "--- Checking llm-d Gateway ---" GW_STATUS=$(oc get gateway llm-d-gateway -n ai-serving -o jsonpath='{.status.conditions[?(@.type=="Accepted")].status}' 2>/dev/null || echo "NOT_FOUND") if [ "$GW_STATUS" = "True" ]; then - pass "llm-d Gateway is Accepted" + pass "llm-d Gateway is Accepted" else - warn "llm-d Gateway Accepted = ${GW_STATUS}" + warn "llm-d Gateway Accepted = ${GW_STATUS}" fi # 9. DevWorkspaces echo "" echo "--- Checking DevWorkspaces ---" for ws in "code-workspace-1:dev1-devspaces" "code-workspace-2:dev2-devspaces"; do - NAME="${ws%%:*}" - NS="${ws##*:}" - WS_PHASE=$(oc get devworkspace "${NAME}" -n "${NS}" -o jsonpath='{.status.phase}' 2>/dev/null || echo "NOT_FOUND") - if [ "$WS_PHASE" = "Running" ]; then - pass "DevWorkspace ${NAME} in ${NS} is Running" - else - warn "DevWorkspace ${NAME} in ${NS} phase = ${WS_PHASE}" - fi + NAME="${ws%%:*}" + NS="${ws##*:}" + WS_PHASE=$(oc get devworkspace "${NAME}" -n "${NS}" -o jsonpath='{.status.phase}' 2>/dev/null || echo "NOT_FOUND") + if [ "$WS_PHASE" = "Running" ]; then + pass "DevWorkspace ${NAME} in ${NS} is Running" + else + warn "DevWorkspace ${NAME} in ${NS} phase = ${WS_PHASE}" + fi done # 10. ArgoCD Apps echo "" echo "--- Checking ArgoCD Applications ---" for app in "pca-operators" "pca-platform-config" "pca-ai-serving" "pca-devspaces"; do - SYNC=$(oc get application "${app}" -n openshift-gitops -o jsonpath='{.status.sync.status}' 2>/dev/null || echo "NOT_FOUND") - HEALTH=$(oc get application "${app}" -n openshift-gitops -o jsonpath='{.status.health.status}' 2>/dev/null || echo "NOT_FOUND") - if [ "$SYNC" = "Synced" ] && [ "$HEALTH" = "Healthy" ]; then - pass "ArgoCD app ${app}: Synced + Healthy" - else - fail "ArgoCD app ${app}: sync=${SYNC}, health=${HEALTH}" - fi + SYNC=$(oc get application "${app}" -n openshift-gitops -o jsonpath='{.status.sync.status}' 2>/dev/null || echo "NOT_FOUND") + HEALTH=$(oc get application "${app}" -n openshift-gitops -o jsonpath='{.status.health.status}' 2>/dev/null || echo "NOT_FOUND") + if [ "$SYNC" = "Synced" ] && [ "$HEALTH" = "Healthy" ]; then + pass "ArgoCD app ${app}: Synced + Healthy" + else + fail "ArgoCD app ${app}: sync=${SYNC}, health=${HEALTH}" + fi done # 11. Model endpoint connectivity @@ -151,43 +154,43 @@ echo " Model: ${MODEL_ID}" MODELS_RESULT=$(oc run validate-models --rm -i --restart=Never --image=curlimages/curl -- -sk "${GATEWAY_URL}/v1/models" 2>/dev/null || echo "CONN_FAILED") if echo "$MODELS_RESULT" | grep -q "${MODEL_ID}"; then - pass "Model endpoint reachable — ${MODEL_ID} listed" + pass "Model endpoint reachable — ${MODEL_ID} listed" else - fail "Model endpoint unreachable or model not listed" + fail "Model endpoint unreachable or model not listed" fi # 12. Tool calling verification echo "" echo "--- Checking Tool Calling (reasoning parser + tool parser) ---" TOOL_RESULT=$(oc run validate-toolcall --rm -i --restart=Never --image=curlimages/curl -- \ - -sk "${GATEWAY_URL}/v1/chat/completions" \ - -H "Content-Type: application/json" \ - -d "{\"model\":\"${MODEL_ID}\",\"messages\":[{\"role\":\"user\",\"content\":\"List files in /tmp\"}],\"tools\":[{\"type\":\"function\",\"function\":{\"name\":\"list_files\",\"description\":\"List directory contents\",\"parameters\":{\"type\":\"object\",\"properties\":{\"path\":{\"type\":\"string\",\"description\":\"Directory path\"}},\"required\":[\"path\"]}}}],\"tool_choice\":\"auto\",\"max_tokens\":200}" 2>/dev/null || echo "TOOL_FAILED") + -sk "${GATEWAY_URL}/v1/chat/completions" \ + -H "Content-Type: application/json" \ + -d "{\"model\":\"${MODEL_ID}\",\"messages\":[{\"role\":\"user\",\"content\":\"List files in /tmp\"}],\"tools\":[{\"type\":\"function\",\"function\":{\"name\":\"list_files\",\"description\":\"List directory contents\",\"parameters\":{\"type\":\"object\",\"properties\":{\"path\":{\"type\":\"string\",\"description\":\"Directory path\"}},\"required\":[\"path\"]}}}],\"tool_choice\":\"auto\",\"max_tokens\":200}" 2>/dev/null || echo "TOOL_FAILED") if echo "$TOOL_RESULT" | grep -q '"finish_reason":"tool_calls"'; then - pass "Tool calling works — finish_reason=tool_calls" + pass "Tool calling works — finish_reason=tool_calls" elif echo "$TOOL_RESULT" | grep -q '"tool_calls":\[{'; then - pass "Tool calling works — tool_calls array populated" + pass "Tool calling works — tool_calls array populated" else - if echo "$TOOL_RESULT" | grep -q ''; then - fail "Tool calling broken — tokens leaking into content (missing --reasoning-parser)" - elif echo "$TOOL_RESULT" | grep -q ''; then - fail "Tool calling broken — XML tool_call in content (wrong --tool-call-parser, need qwen3_xml)" - elif echo "$TOOL_RESULT" | grep -q 'TOOL_FAILED'; then - fail "Tool calling test failed — could not reach gateway" - else - warn "Tool calling inconclusive — model may have responded without tool use" - warn " Response snippet: $(echo "$TOOL_RESULT" | head -c 200)" - fi + if echo "$TOOL_RESULT" | grep -q ''; then + fail "Tool calling broken — tokens leaking into content (missing --reasoning-parser)" + elif echo "$TOOL_RESULT" | grep -q ''; then + fail "Tool calling broken — XML tool_call in content (wrong --tool-call-parser, need qwen3_xml)" + elif echo "$TOOL_RESULT" | grep -q 'TOOL_FAILED'; then + fail "Tool calling test failed — could not reach gateway" + else + warn "Tool calling inconclusive — model may have responded without tool use" + warn " Response snippet: $(echo "$TOOL_RESULT" | head -c 200)" + fi fi # Summary echo "" echo "=========================================" if [ $FAILURES -eq 0 ]; then - echo -e "${GREEN}All checks passed!${NC}" + echo -e "${GREEN}All checks passed!${NC}" else - echo -e "${RED}${FAILURES} check(s) failed.${NC}" + echo -e "${RED}${FAILURES} check(s) failed.${NC}" fi echo "=========================================" exit $FAILURES diff --git a/PCA_Deployment_ROSA/argocd/00-app-of-apps.yaml b/PCA_Deployment_ROSA/argocd/00-app-of-apps.yaml index b1df00c..395f84e 100644 --- a/PCA_Deployment_ROSA/argocd/00-app-of-apps.yaml +++ b/PCA_Deployment_ROSA/argocd/00-app-of-apps.yaml @@ -1,7 +1,6 @@ # Root App-of-Apps: ArgoCD discovers child Applications in this directory. # Each child Application below points to a subdirectory of manifests. # Sync waves ensure correct ordering: operators first, then platform, then workloads. ---- apiVersion: argoproj.io/v1alpha1 kind: AppProject metadata: diff --git a/PCA_Deployment_ROSA/argocd/01-operators/cert-manager.yaml b/PCA_Deployment_ROSA/argocd/01-operators/cert-manager.yaml index bc17020..b14a0a3 100644 --- a/PCA_Deployment_ROSA/argocd/01-operators/cert-manager.yaml +++ b/PCA_Deployment_ROSA/argocd/01-operators/cert-manager.yaml @@ -1,6 +1,5 @@ # cert-manager — required by KServe for webhook certificates # Installed via kustomize overlay from llm-d-playbook ---- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: diff --git a/PCA_Deployment_ROSA/argocd/01-operators/leader-worker-set.yaml b/PCA_Deployment_ROSA/argocd/01-operators/leader-worker-set.yaml index aba2e06..014d33e 100644 --- a/PCA_Deployment_ROSA/argocd/01-operators/leader-worker-set.yaml +++ b/PCA_Deployment_ROSA/argocd/01-operators/leader-worker-set.yaml @@ -1,5 +1,4 @@ # LeaderWorkerSet Operator — required by KServe for multi-node inference ---- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: diff --git a/PCA_Deployment_ROSA/argocd/01-operators/lws-operator-cr.yaml b/PCA_Deployment_ROSA/argocd/01-operators/lws-operator-cr.yaml index 6a2cb71..e5aa1d3 100644 --- a/PCA_Deployment_ROSA/argocd/01-operators/lws-operator-cr.yaml +++ b/PCA_Deployment_ROSA/argocd/01-operators/lws-operator-cr.yaml @@ -1,6 +1,5 @@ # LeaderWorkerSet Operator instance — enables the LWS controller. # Applied after the LWS CRD is installed (wave 1). ---- apiVersion: leaderworkerset.x-k8s.io/v1 kind: LeaderWorkerSetOperator metadata: diff --git a/PCA_Deployment_ROSA/argocd/01-operators/nvidia-cluster-policy.yaml b/PCA_Deployment_ROSA/argocd/01-operators/nvidia-cluster-policy.yaml index d664b55..f8395d6 100644 --- a/PCA_Deployment_ROSA/argocd/01-operators/nvidia-cluster-policy.yaml +++ b/PCA_Deployment_ROSA/argocd/01-operators/nvidia-cluster-policy.yaml @@ -1,6 +1,5 @@ # NVIDIA GPU Operator ClusterPolicy — activates the GPU stack. # Applied after the GPU Operator Subscription installs the CRD. ---- apiVersion: nvidia.com/v1 kind: ClusterPolicy metadata: diff --git a/PCA_Deployment_ROSA/argocd/01-operators/subscriptions.yaml b/PCA_Deployment_ROSA/argocd/01-operators/subscriptions.yaml index 65f492d..480d376 100644 --- a/PCA_Deployment_ROSA/argocd/01-operators/subscriptions.yaml +++ b/PCA_Deployment_ROSA/argocd/01-operators/subscriptions.yaml @@ -1,6 +1,5 @@ # All operator Subscriptions managed by ArgoCD. # Sync wave 1: these must install before platform config (wave 2) uses their CRDs. ---- # Red Hat OpenShift AI (RHOAI) 3.3+ apiVersion: v1 kind: Namespace diff --git a/PCA_Deployment_ROSA/argocd/02-platform-config/checluster.yaml b/PCA_Deployment_ROSA/argocd/02-platform-config/checluster.yaml index 19f9452..b7dc88a 100644 --- a/PCA_Deployment_ROSA/argocd/02-platform-config/checluster.yaml +++ b/PCA_Deployment_ROSA/argocd/02-platform-config/checluster.yaml @@ -1,5 +1,4 @@ # OpenShift Dev Spaces instance — creates the dashboard, workspace controller, registries ---- apiVersion: org.eclipse.che/v2 kind: CheCluster metadata: diff --git a/PCA_Deployment_ROSA/argocd/02-platform-config/datasciencecluster.yaml b/PCA_Deployment_ROSA/argocd/02-platform-config/datasciencecluster.yaml index 88bdbb5..9aad5b6 100644 --- a/PCA_Deployment_ROSA/argocd/02-platform-config/datasciencecluster.yaml +++ b/PCA_Deployment_ROSA/argocd/02-platform-config/datasciencecluster.yaml @@ -1,6 +1,5 @@ # Patch DataScienceCluster for KServe "Headed" mode (required for llm-d + Gateway API) # The DSC is created by the RHOAI operator; this resource ensures the correct configuration. ---- apiVersion: datasciencecluster.opendatahub.io/v1 kind: DataScienceCluster metadata: diff --git a/PCA_Deployment_ROSA/argocd/02-platform-config/hf-token-placeholder.yaml b/PCA_Deployment_ROSA/argocd/02-platform-config/hf-token-placeholder.yaml index 041ce3e..cde2ded 100644 --- a/PCA_Deployment_ROSA/argocd/02-platform-config/hf-token-placeholder.yaml +++ b/PCA_Deployment_ROSA/argocd/02-platform-config/hf-token-placeholder.yaml @@ -1,7 +1,6 @@ # HuggingFace token secret — PLACEHOLDER # Replace REPLACE_WITH_BASE64_HF_TOKEN with: echo -n 'hf_your_token' | base64 # For production, use External Secrets Operator or Sealed Secrets instead. ---- apiVersion: v1 kind: Secret metadata: diff --git a/PCA_Deployment_ROSA/argocd/02-platform-config/namespaces.yaml b/PCA_Deployment_ROSA/argocd/02-platform-config/namespaces.yaml index 44bdcb0..69fc401 100644 --- a/PCA_Deployment_ROSA/argocd/02-platform-config/namespaces.yaml +++ b/PCA_Deployment_ROSA/argocd/02-platform-config/namespaces.yaml @@ -1,5 +1,4 @@ # Namespaces for the AI platform and developer workspaces ---- apiVersion: v1 kind: Namespace metadata: diff --git a/PCA_Deployment_ROSA/argocd/02-platform-config/rbac.yaml b/PCA_Deployment_ROSA/argocd/02-platform-config/rbac.yaml index f8be4af..bc42e8d 100644 --- a/PCA_Deployment_ROSA/argocd/02-platform-config/rbac.yaml +++ b/PCA_Deployment_ROSA/argocd/02-platform-config/rbac.yaml @@ -1,5 +1,4 @@ # RBAC for DevSpaces users — grants namespace admin in their respective namespaces ---- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: diff --git a/PCA_Deployment_ROSA/argocd/03-ai-serving/hardware-profiles.yaml b/PCA_Deployment_ROSA/argocd/03-ai-serving/hardware-profiles.yaml index 518b334..db0bf75 100644 --- a/PCA_Deployment_ROSA/argocd/03-ai-serving/hardware-profiles.yaml +++ b/PCA_Deployment_ROSA/argocd/03-ai-serving/hardware-profiles.yaml @@ -1,5 +1,4 @@ # HardwareProfile for AWS Trainium (trn1.32xlarge) — optional, for future multi-accelerator deployments ---- apiVersion: infrastructure.opendatahub.io/v1 kind: HardwareProfile metadata: diff --git a/PCA_Deployment_ROSA/argocd/03-ai-serving/llm-d-gateway.yaml b/PCA_Deployment_ROSA/argocd/03-ai-serving/llm-d-gateway.yaml index 9cee793..2f3f14b 100644 --- a/PCA_Deployment_ROSA/argocd/03-ai-serving/llm-d-gateway.yaml +++ b/PCA_Deployment_ROSA/argocd/03-ai-serving/llm-d-gateway.yaml @@ -3,7 +3,6 @@ # # The resulting internal DNS endpoint is: # https://llm-d-gateway-data-science-gateway-class.ai-serving.svc.cluster.local ---- apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: diff --git a/PCA_Deployment_ROSA/argocd/03-ai-serving/llminferenceservice.yaml b/PCA_Deployment_ROSA/argocd/03-ai-serving/llminferenceservice.yaml index 15ad7d3..c0c9fc2 100644 --- a/PCA_Deployment_ROSA/argocd/03-ai-serving/llminferenceservice.yaml +++ b/PCA_Deployment_ROSA/argocd/03-ai-serving/llminferenceservice.yaml @@ -1,6 +1,5 @@ # LLMInferenceService — Qwen3-Coder-30B on NVIDIA GPU via KServe + llm-d # This single CR creates: Deployment, Service, InferencePool, EPP, HTTPRoute ---- apiVersion: serving.kserve.io/v1alpha1 kind: LLMInferenceService metadata: @@ -90,14 +89,7 @@ spec: env: - name: VLLM_ADDITIONAL_ARGS value: >- - --disable-uvicorn-access-log - --max-model-len 32768 - --gpu-memory-utilization 0.90 - --enable-prefix-caching - --enable-auto-tool-choice - --tool-call-parser qwen3_coder - --reasoning-parser qwen3 - --kv-cache-dtype fp8 + --disable-uvicorn-access-log --max-model-len 32768 --gpu-memory-utilization 0.90 --enable-prefix-caching --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 --kv-cache-dtype fp8 volumeMounts: - name: kserve-provision-location mountPath: /mnt/models diff --git a/PCA_Deployment_ROSA/argocd/03-ai-serving/pvcs.yaml b/PCA_Deployment_ROSA/argocd/03-ai-serving/pvcs.yaml index 2c694a2..d84f210 100644 --- a/PCA_Deployment_ROSA/argocd/03-ai-serving/pvcs.yaml +++ b/PCA_Deployment_ROSA/argocd/03-ai-serving/pvcs.yaml @@ -1,6 +1,5 @@ # Persistent Volume Claims for model weight caching. # Using PVCs avoids re-downloading models on pod restarts (~30GB for Qwen3-Coder-30B). ---- apiVersion: v1 kind: PersistentVolumeClaim metadata: diff --git a/PCA_Deployment_ROSA/argocd/03-ai-serving/tls-secret-job.yaml b/PCA_Deployment_ROSA/argocd/03-ai-serving/tls-secret-job.yaml index 5463512..0e23c0b 100644 --- a/PCA_Deployment_ROSA/argocd/03-ai-serving/tls-secret-job.yaml +++ b/PCA_Deployment_ROSA/argocd/03-ai-serving/tls-secret-job.yaml @@ -1,6 +1,5 @@ # Job to generate a self-signed TLS certificate for the llm-d Gateway. # Runs once after the namespace is created. ---- apiVersion: batch/v1 kind: Job metadata: diff --git a/PCA_Deployment_ROSA/argocd/03-ai-serving/vllm-neuron-runtime-template.yaml b/PCA_Deployment_ROSA/argocd/03-ai-serving/vllm-neuron-runtime-template.yaml index 46a569b..e31be08 100644 --- a/PCA_Deployment_ROSA/argocd/03-ai-serving/vllm-neuron-runtime-template.yaml +++ b/PCA_Deployment_ROSA/argocd/03-ai-serving/vllm-neuron-runtime-template.yaml @@ -1,5 +1,4 @@ # vLLM Neuron ServingRuntime Template — for Inferentia2/Trainium deployments ---- apiVersion: template.openshift.io/v1 kind: Template metadata: @@ -11,8 +10,7 @@ metadata: annotations: argocd.argoproj.io/sync-wave: "1" description: >- - vLLM ServingRuntime for AWS Inferentia2 / Trainium (Neuron SDK). - Supports dense and MoE models on NeuronCores. + vLLM ServingRuntime for AWS Inferentia2 / Trainium (Neuron SDK). Supports dense and MoE models on NeuronCores. openshift.io/display-name: "vLLM AWS Neuron (Inferentia2 / Trainium) ServingRuntime for KServe" openshift.io/provider-display-name: "Custom" opendatahub.io/apiProtocol: REST diff --git a/PCA_Deployment_ROSA/argocd/04-devspaces/devworkspaces.yaml b/PCA_Deployment_ROSA/argocd/04-devspaces/devworkspaces.yaml index 696def6..58aefa2 100644 --- a/PCA_Deployment_ROSA/argocd/04-devspaces/devworkspaces.yaml +++ b/PCA_Deployment_ROSA/argocd/04-devspaces/devworkspaces.yaml @@ -26,7 +26,6 @@ # --enable-auto-tool-choice required: Roo Code sends tool_choice:"auto"; vLLM rejects without this # --tool-call-parser qwen3_coder parses Qwen3-Coder XML tool calls into OpenAI tool_calls array # --reasoning-parser qwen3 isolates tokens from content ---- apiVersion: workspace.devfile.io/v1alpha2 kind: DevWorkspace metadata: diff --git a/PCA_Deployment_ROSA/argocd/04-devspaces/roo-code-configmaps.yaml b/PCA_Deployment_ROSA/argocd/04-devspaces/roo-code-configmaps.yaml index 88fb372..fd4267d 100644 --- a/PCA_Deployment_ROSA/argocd/04-devspaces/roo-code-configmaps.yaml +++ b/PCA_Deployment_ROSA/argocd/04-devspaces/roo-code-configmaps.yaml @@ -14,7 +14,6 @@ # # NOTE: The llm-d gateway URL uses HTTPS with a cluster-internal cert; extensions must set # NODE_TLS_REJECT_UNAUTHORIZED=0 or requestOptions.verifySsl=false. ---- # ── Continue config (source namespace — synced to all user namespaces) ────────────────────── apiVersion: v1 kind: ConfigMap diff --git a/PCA_Deployment_ROSA/argocd/04-devspaces/vscode-extensions-config.yaml b/PCA_Deployment_ROSA/argocd/04-devspaces/vscode-extensions-config.yaml index d5e5738..2eba6fb 100644 --- a/PCA_Deployment_ROSA/argocd/04-devspaces/vscode-extensions-config.yaml +++ b/PCA_Deployment_ROSA/argocd/04-devspaces/vscode-extensions-config.yaml @@ -1,5 +1,4 @@ # Cluster-wide VS Code extension recommendations for all DevSpaces workspaces ---- apiVersion: v1 kind: ConfigMap metadata: diff --git a/PCA_Deployment_ROSA/scripts/seal-secret.sh b/PCA_Deployment_ROSA/scripts/seal-secret.sh index df605bd..fbee6c5 100755 --- a/PCA_Deployment_ROSA/scripts/seal-secret.sh +++ b/PCA_Deployment_ROSA/scripts/seal-secret.sh @@ -16,15 +16,15 @@ INPUT="${1:?Usage: seal-secret.sh }" OUTPUT="${2:?Usage: seal-secret.sh }" if ! command -v kubeseal &>/dev/null; then - echo "ERROR: kubeseal CLI not found. Install from: https://github.com/bitnami-labs/sealed-secrets/releases" - exit 1 + echo "ERROR: kubeseal CLI not found. Install from: https://github.com/bitnami-labs/sealed-secrets/releases" + exit 1 fi kubeseal --format yaml \ - --controller-name sealed-secrets \ - --controller-namespace kube-system \ - < "${INPUT}" \ - > "${OUTPUT}" + --controller-name sealed-secrets \ + --controller-namespace kube-system \ + <"${INPUT}" \ + >"${OUTPUT}" echo "Sealed secret written to: ${OUTPUT}" echo "This file is safe to commit to Git." diff --git a/PCA_Deployment_ROSA/scripts/validate.sh b/PCA_Deployment_ROSA/scripts/validate.sh index 0301ed0..a8e0f1f 100755 --- a/PCA_Deployment_ROSA/scripts/validate.sh +++ b/PCA_Deployment_ROSA/scripts/validate.sh @@ -9,7 +9,10 @@ YELLOW='\033[1;33m' NC='\033[0m' pass() { echo -e "${GREEN}[PASS]${NC} $1"; } -fail() { echo -e "${RED}[FAIL]${NC} $1"; FAILURES=$((FAILURES + 1)); } +fail() { + echo -e "${RED}[FAIL]${NC} $1" + FAILURES=$((FAILURES + 1)) +} warn() { echo -e "${YELLOW}[WARN]${NC} $1"; } FAILURES=0 @@ -22,11 +25,11 @@ echo "" # 1. Operators echo "--- Checking Operators ---" for op in "rhods-operator" "devspaces" "servicemeshoperator3" "gpu-operator-certified" "openshift-gitops-operator"; do - if oc get csv -A 2>/dev/null | grep -q "${op}.*Succeeded"; then - pass "Operator ${op} installed and healthy" - else - fail "Operator ${op} not found or not Succeeded" - fi + if oc get csv -A 2>/dev/null | grep -q "${op}.*Succeeded"; then + pass "Operator ${op} installed and healthy" + else + fail "Operator ${op} not found or not Succeeded" + fi done # 2. DataScienceCluster @@ -34,29 +37,29 @@ echo "" echo "--- Checking DataScienceCluster ---" DSC_MODE=$(oc get datasciencecluster default-dsc -o jsonpath='{.spec.components.kserve.rawDeploymentServiceConfig}' 2>/dev/null || echo "NOT_FOUND") if [ "$DSC_MODE" = "Headed" ]; then - pass "DataScienceCluster rawDeploymentServiceConfig = Headed" + pass "DataScienceCluster rawDeploymentServiceConfig = Headed" else - fail "DataScienceCluster rawDeploymentServiceConfig = ${DSC_MODE} (expected: Headed)" + fail "DataScienceCluster rawDeploymentServiceConfig = ${DSC_MODE} (expected: Headed)" fi # 3. GPU Operator echo "" echo "--- Checking GPU Stack ---" if oc get clusterpolicy gpu-cluster-policy &>/dev/null; then - pass "NVIDIA ClusterPolicy exists" + pass "NVIDIA ClusterPolicy exists" else - warn "NVIDIA ClusterPolicy not found (expected if no GPU nodes)" + warn "NVIDIA ClusterPolicy not found (expected if no GPU nodes)" fi # 4. Namespaces echo "" echo "--- Checking Namespaces ---" for ns in "ai-serving" "dev1-devspaces" "dev2-devspaces" "openshift-devspaces"; do - if oc get ns "${ns}" &>/dev/null; then - pass "Namespace ${ns} exists" - else - fail "Namespace ${ns} missing" - fi + if oc get ns "${ns}" &>/dev/null; then + pass "Namespace ${ns} exists" + else + fail "Namespace ${ns} missing" + fi done # 5. CheCluster @@ -64,9 +67,9 @@ echo "" echo "--- Checking Dev Spaces ---" CHE_STATUS=$(oc get checluster devspaces -n openshift-devspaces -o jsonpath='{.status.chePhase}' 2>/dev/null || echo "NOT_FOUND") if [ "$CHE_STATUS" = "Active" ]; then - pass "CheCluster is Active" + pass "CheCluster is Active" else - fail "CheCluster phase = ${CHE_STATUS} (expected: Active)" + fail "CheCluster phase = ${CHE_STATUS} (expected: Active)" fi # 6. Model Serving @@ -74,9 +77,9 @@ echo "" echo "--- Checking Model Serving ---" LIS_STATUS=$(oc get llminferenceservice qwen3-coder -n ai-serving -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}' 2>/dev/null || echo "NOT_FOUND") if [ "$LIS_STATUS" = "True" ]; then - pass "LLMInferenceService qwen3-coder is Ready" + pass "LLMInferenceService qwen3-coder is Ready" else - warn "LLMInferenceService qwen3-coder Ready = ${LIS_STATUS} (may be starting up)" + warn "LLMInferenceService qwen3-coder Ready = ${LIS_STATUS} (may be starting up)" fi # 7. PVC @@ -84,9 +87,9 @@ echo "" echo "--- Checking Storage ---" PVC_STATUS=$(oc get pvc model-cache -n ai-serving -o jsonpath='{.status.phase}' 2>/dev/null || echo "NOT_FOUND") if [ "$PVC_STATUS" = "Bound" ]; then - pass "PVC model-cache is Bound" + pass "PVC model-cache is Bound" else - warn "PVC model-cache status = ${PVC_STATUS}" + warn "PVC model-cache status = ${PVC_STATUS}" fi # 8. Gateway @@ -94,36 +97,36 @@ echo "" echo "--- Checking llm-d Gateway ---" GW_STATUS=$(oc get gateway llm-d-gateway -n ai-serving -o jsonpath='{.status.conditions[?(@.type=="Accepted")].status}' 2>/dev/null || echo "NOT_FOUND") if [ "$GW_STATUS" = "True" ]; then - pass "llm-d Gateway is Accepted" + pass "llm-d Gateway is Accepted" else - warn "llm-d Gateway Accepted = ${GW_STATUS}" + warn "llm-d Gateway Accepted = ${GW_STATUS}" fi # 9. DevWorkspaces echo "" echo "--- Checking DevWorkspaces ---" for ws in "code-workspace-1:dev1-devspaces" "code-workspace-2:dev2-devspaces"; do - NAME="${ws%%:*}" - NS="${ws##*:}" - WS_PHASE=$(oc get devworkspace "${NAME}" -n "${NS}" -o jsonpath='{.status.phase}' 2>/dev/null || echo "NOT_FOUND") - if [ "$WS_PHASE" = "Running" ]; then - pass "DevWorkspace ${NAME} in ${NS} is Running" - else - warn "DevWorkspace ${NAME} in ${NS} phase = ${WS_PHASE}" - fi + NAME="${ws%%:*}" + NS="${ws##*:}" + WS_PHASE=$(oc get devworkspace "${NAME}" -n "${NS}" -o jsonpath='{.status.phase}' 2>/dev/null || echo "NOT_FOUND") + if [ "$WS_PHASE" = "Running" ]; then + pass "DevWorkspace ${NAME} in ${NS} is Running" + else + warn "DevWorkspace ${NAME} in ${NS} phase = ${WS_PHASE}" + fi done # 10. ArgoCD Apps echo "" echo "--- Checking ArgoCD Applications ---" for app in "pca-operators" "pca-platform-config" "pca-ai-serving" "pca-devspaces"; do - SYNC=$(oc get application "${app}" -n openshift-gitops -o jsonpath='{.status.sync.status}' 2>/dev/null || echo "NOT_FOUND") - HEALTH=$(oc get application "${app}" -n openshift-gitops -o jsonpath='{.status.health.status}' 2>/dev/null || echo "NOT_FOUND") - if [ "$SYNC" = "Synced" ] && [ "$HEALTH" = "Healthy" ]; then - pass "ArgoCD app ${app}: Synced + Healthy" - else - fail "ArgoCD app ${app}: sync=${SYNC}, health=${HEALTH}" - fi + SYNC=$(oc get application "${app}" -n openshift-gitops -o jsonpath='{.status.sync.status}' 2>/dev/null || echo "NOT_FOUND") + HEALTH=$(oc get application "${app}" -n openshift-gitops -o jsonpath='{.status.health.status}' 2>/dev/null || echo "NOT_FOUND") + if [ "$SYNC" = "Synced" ] && [ "$HEALTH" = "Healthy" ]; then + pass "ArgoCD app ${app}: Synced + Healthy" + else + fail "ArgoCD app ${app}: sync=${SYNC}, health=${HEALTH}" + fi done # 11. Model endpoint connectivity test @@ -137,9 +140,9 @@ echo " (Run this from within the cluster to test connectivity)" echo "" echo "=========================================" if [ $FAILURES -eq 0 ]; then - echo -e "${GREEN}All checks passed!${NC}" + echo -e "${GREEN}All checks passed!${NC}" else - echo -e "${RED}${FAILURES} check(s) failed.${NC}" + echo -e "${RED}${FAILURES} check(s) failed.${NC}" fi echo "=========================================" exit $FAILURES diff --git a/PCA_Deployment_ROSA/terraform/gitops-bootstrap.tf b/PCA_Deployment_ROSA/terraform/gitops-bootstrap.tf index 99d2b8e..532eba8 100644 --- a/PCA_Deployment_ROSA/terraform/gitops-bootstrap.tf +++ b/PCA_Deployment_ROSA/terraform/gitops-bootstrap.tf @@ -9,6 +9,12 @@ resource "null_resource" "install_gitops_operator" { command = <<-EOT set -e + echo "Logging into cluster..." + oc login ${rhcs_cluster_rosa_hcp.cluster.api_url} \ + --username=cluster-admin \ + --password='${var.cluster_admin_password}' \ + --insecure-skip-tls-verify=true + echo "Installing OpenShift GitOps operator..." cat <<'YAML' | oc apply -f - apiVersion: v1 diff --git a/PCA_Deployment_ROSA/terraform/main.tf b/PCA_Deployment_ROSA/terraform/main.tf index db57bc4..549571c 100644 --- a/PCA_Deployment_ROSA/terraform/main.tf +++ b/PCA_Deployment_ROSA/terraform/main.tf @@ -2,6 +2,8 @@ locals { operator_role_prefix = var.operator_role_prefix != "" ? var.operator_role_prefix : var.cluster_name billing_account_id = var.aws_billing_account_id != "" ? var.aws_billing_account_id : var.aws_account_id private_subnet_ids = var.use_existing_vpc ? var.existing_private_subnet_ids : aws_subnet.private[*].id + public_subnet_ids = var.use_existing_vpc ? [] : aws_subnet.public[*].id + all_subnet_ids = concat(local.private_subnet_ids, local.public_subnet_ids) rosa_creator_arn = data.aws_caller_identity.current.arn } @@ -103,7 +105,6 @@ module "account_iam_resources" { source = "terraform-redhat/rosa-hcp/rhcs//modules/account-iam-resources" account_role_prefix = var.account_role_prefix - openshift_version = var.openshift_version } module "oidc_config" { @@ -115,10 +116,8 @@ module "oidc_config" { module "operator_roles" { source = "terraform-redhat/rosa-hcp/rhcs//modules/operator-roles" - account_role_prefix = var.account_role_prefix operator_role_prefix = local.operator_role_prefix oidc_endpoint_url = module.oidc_config.oidc_endpoint_url - path = "/service-role/" } # ════════════════════════════════════════════════ @@ -129,26 +128,27 @@ resource "rhcs_cluster_rosa_hcp" "cluster" { cloud_region = var.aws_region aws_account_id = var.aws_account_id aws_billing_account_id = local.billing_account_id - aws_subnet_ids = local.private_subnet_ids + aws_subnet_ids = local.all_subnet_ids availability_zones = var.availability_zones version = var.openshift_version - replicas = var.default_worker_replicas - compute_machine_type = var.default_worker_instance_type properties = { rosa_creator_arn = local.rosa_creator_arn } sts = { - role_arn = module.account_iam_resources.account_roles_arn["Installer"] - support_role_arn = module.account_iam_resources.account_roles_arn["Support"] + role_arn = module.account_iam_resources.account_roles_arn["HCP-ROSA-Installer"] + support_role_arn = module.account_iam_resources.account_roles_arn["HCP-ROSA-Support"] instance_iam_roles = { - worker_role_arn = module.account_iam_resources.account_roles_arn["Worker"] + worker_role_arn = module.account_iam_resources.account_roles_arn["HCP-ROSA-Worker"] } operator_role_prefix = local.operator_role_prefix oidc_config_id = module.oidc_config.oidc_config_id } + compute_machine_type = var.default_worker_instance_type + replicas = var.default_worker_replicas + wait_for_create_complete = true wait_for_std_compute_nodes_complete = true @@ -184,7 +184,9 @@ resource "rhcs_hcp_machine_pool" "gpu" { min_replicas = var.gpu_pool_replicas max_replicas = var.gpu_pool_max_replicas } : { - enabled = false + enabled = false + min_replicas = null + max_replicas = null } replicas = var.gpu_pool_autoscaling ? null : var.gpu_pool_replicas @@ -196,9 +198,9 @@ resource "rhcs_hcp_machine_pool" "gpu" { } taints = [{ - key = "nvidia.com/gpu" - value = "present" - effect = "NoSchedule" + key = "nvidia.com/gpu" + value = "present" + schedule_type = "NoSchedule" }] depends_on = [rhcs_cluster_wait.wait] @@ -229,9 +231,9 @@ resource "rhcs_hcp_machine_pool" "inferentia" { } taints = [{ - key = "aws.amazon.com/neuroncore" - value = "present" - effect = "NoSchedule" + key = "aws.amazon.com/neuroncore" + value = "present" + schedule_type = "NoSchedule" }] depends_on = [rhcs_cluster_wait.wait] @@ -245,34 +247,31 @@ resource "rhcs_identity_provider" "htpasswd" { name = "devspaces-users" htpasswd = { - users = concat( - var.cluster_admin_password != "" ? [{ - username = "cluster-admin" - password = var.cluster_admin_password - }] : [], - [for u in var.devspaces_users : { - username = u.username - password = u.password - } if u.password != ""] - ) + users = [for u in var.devspaces_users : { + username = u.username + password = u.password + } if u.password != ""] } depends_on = [rhcs_cluster_wait.wait] } # Grant cluster-admin role to the admin user +resource "time_sleep" "wait_for_idp" { + create_duration = "120s" + depends_on = [rhcs_identity_provider.htpasswd] +} + resource "null_resource" "grant_cluster_admin" { count = var.cluster_admin_password != "" ? 1 : 0 provisioner "local-exec" { command = <<-EOT - oc login ${rhcs_cluster_rosa_hcp.cluster.api_url} \ - --username=cluster-admin \ - --password='${var.cluster_admin_password}' \ - --insecure-skip-tls-verify=true - oc adm policy add-cluster-role-to-user cluster-admin cluster-admin + set -e + rosa login --token='${var.rhcs_token}' + rosa create admin --cluster=${rhcs_cluster_rosa_hcp.cluster.id} --password='${var.cluster_admin_password}' || echo "Admin already exists, skipping." EOT } - depends_on = [rhcs_identity_provider.htpasswd] + depends_on = [time_sleep.wait_for_idp] } diff --git a/PCA_Deployment_ROSA/terraform/terraform.tfvars.example b/PCA_Deployment_ROSA/terraform/terraform.tfvars.example index af57795..2d561c6 100644 --- a/PCA_Deployment_ROSA/terraform/terraform.tfvars.example +++ b/PCA_Deployment_ROSA/terraform/terraform.tfvars.example @@ -39,8 +39,8 @@ inferentia_pool_replicas = 1 # Users cluster_admin_password = "" devspaces_users = [ - { username = "dev-user1", password = "ChangeMe123!" }, - { username = "dev-user2", password = "ChangeMe456!" }, + { username = "dev-user1", password = "ChangeMe1234!!" }, + { username = "dev-user2", password = "ChangeMe4567!!" }, ] # Secrets diff --git a/PCA_Deployment_ROSA/terraform/versions.tf b/PCA_Deployment_ROSA/terraform/versions.tf index cf2bd16..706b23e 100644 --- a/PCA_Deployment_ROSA/terraform/versions.tf +++ b/PCA_Deployment_ROSA/terraform/versions.tf @@ -14,6 +14,10 @@ terraform { source = "hashicorp/null" version = ">= 3.0" } + time = { + source = "hashicorp/time" + version = ">= 0.9" + } } # Uncomment to use remote state (recommended for teams) diff --git a/README.md b/README.md index f0c13fb..f540911 100644 --- a/README.md +++ b/README.md @@ -4097,4 +4097,4 @@ Three candidate fixes for the neuron-scheduler / OVN-Kubernetes annotation confl Document Version: 2.4 Last Updated: April 2026 -Status: Active Development -- Phases 1-4 verified, Phase 6 completed. Phase 6 GuideLLM sweep on L40S completed (April 20, 2026): Qwen3-Coder-30B-A3B-FP8, single L40S, peak 1,357 output tok/s, 93 tok/s single-user, 73ms TTFT. Phase 4b 2-replica scaling investigation completed (April 9, 2026): TP=16 requires all 16 NeuronDevices for NeuronLink topology — 8-device split causes NEFF execution failures and disk pressure; horizontal scaling across nodes recommended instead. Phase 4 Trainium trn1.32xlarge with DRA driver deployed (April 9, 2026): 16 NeuronCores via ResourceClaimTemplate, TP=16, 16k context, llm-d routing verified, GuideLLM sweep confirmed. Phase 3 heterogeneous routing verified (April 8, 2026): same-namespace, native TLS, 0 errors in GuideLLM sweep. Phase 5 (inf2.48xlarge dual-instance) planned. \ No newline at end of file +Status: Active Development -- Phases 1-4 verified, Phase 6 completed. Phase 6 GuideLLM sweep on L40S completed (April 20, 2026): Qwen3-Coder-30B-A3B-FP8, single L40S, peak 1,357 output tok/s, 93 tok/s single-user, 73ms TTFT. Phase 4b 2-replica scaling investigation completed (April 9, 2026): TP=16 requires all 16 NeuronDevices for NeuronLink topology — 8-device split causes NEFF execution failures and disk pressure; horizontal scaling across nodes recommended instead. Phase 4 Trainium trn1.32xlarge with DRA driver deployed (April 9, 2026): 16 NeuronCores via ResourceClaimTemplate, TP=16, 16k context, llm-d routing verified, GuideLLM sweep confirmed. Phase 3 heterogeneous routing verified (April 8, 2026): same-namespace, native TLS, 0 errors in GuideLLM sweep. Phase 5 (inf2.48xlarge dual-instance) planned. diff --git a/assets/GPU_Sizing_Considerations_for_AI_Code_Assistant_v2.html b/assets/GPU_Sizing_Considerations_for_AI_Code_Assistant_v2.html index 1e689be..7778249 100644 --- a/assets/GPU_Sizing_Considerations_for_AI_Code_Assistant_v2.html +++ b/assets/GPU_Sizing_Considerations_for_AI_Code_Assistant_v2.html @@ -1921,4 +1921,4 @@

Appendix A: Reco

Report generated April 2026. All calculations based on publicly available model configurations, GPU specifications, and Red Hat pricing. Actual performance should be validated with empirical benchmarks on the target infrastructure before procurement decisions. AWS pricing verified April 2026; OpenShift subscription rates should be confirmed with Red Hat sales.

- \ No newline at end of file + diff --git a/assets/GPU_Sizing_Considerations_for_AI_Code_Assistant_v2_print.html b/assets/GPU_Sizing_Considerations_for_AI_Code_Assistant_v2_print.html index ce4b7b0..4817f84 100644 --- a/assets/GPU_Sizing_Considerations_for_AI_Code_Assistant_v2_print.html +++ b/assets/GPU_Sizing_Considerations_for_AI_Code_Assistant_v2_print.html @@ -199,4 +199,4 @@

Cost Comparison: GPU-Only (100-Dev Team, 64K)

- \ No newline at end of file + diff --git a/assets/build-opencode-image.sh b/assets/build-opencode-image.sh index 4a53ffa..3f2052a 100755 --- a/assets/build-opencode-image.sh +++ b/assets/build-opencode-image.sh @@ -45,8 +45,8 @@ oc start-build "$BC_NAME" -n "$NAMESPACE" --follow echo "==> Granting image-puller to all DevSpaces namespaces" for ns in dev1-devspaces dev2-devspaces dev3-devspaces; do - oc policy add-role-to-group system:image-puller "system:serviceaccounts:$ns" \ - --namespace="$NAMESPACE" 2>/dev/null || true + oc policy add-role-to-group system:image-puller "system:serviceaccounts:$ns" \ + --namespace="$NAMESPACE" 2>/dev/null || true done echo "==> Image built and available at:"