Add pre-commit, containerized provisioner, and ROSA terraform fixes

.cursor/rules/use-makefile.mdc

@@ -0,0 +1,8 @@
+---
+description: Prefer Makefile targets for project commands
+alwaysApply: true
+---
+
+# Use Makefile Commands
+
+Run all project operations through `make` targets (`make build`, `make shell`, `make run CMD="..."`) instead of invoking `podman`, `terraform`, or other CLIs directly.

.cursor/skills/provision-rosa-aws/SKILL.md

@@ -0,0 +1,126 @@
+---
+name: provision-rosa-aws
+description: >-
+  Provision a ROSA HCP cluster on AWS using the containerized provisioner and
+  Terraform. Use when the user asks to deploy, provision, create, or spin up a
+  ROSA cluster, or mentions AWS provisioning, make build, make shell, or
+  terraform apply for this project.
+disable-model-invocation: true
+---
+
+# Provision ROSA HCP on AWS
+
+## Prerequisites
+
+- AWS credentials (Access Key ID, Secret Access Key, Account ID, Region)
+- Red Hat OCM token from https://console.redhat.com/openshift/token
+- HuggingFace token for model access
+- Podman installed on the host
+
+## Workflow
+
+### Step 1: Configure credentials
+
+Update `.env`:
+
+```
+AWS_ACCESS_KEY_ID=<key>
+AWS_SECRET_ACCESS_KEY=<secret>
+AWS_ACCOUNT_ID=<account-id>
+AWS_DEFAULT_REGION=us-east-2
+RHCS_TOKEN=<ocm-token>
+HUGGINGFACE_TOKEN=<hf-token>
+CLUSTER_ADMIN_PASSWORD=<password>
+```
+
+Update `PCA_Deployment_ROSA/terraform/terraform.tfvars`:
+- Set `aws_account_id`, `rhcs_token`, `huggingface_token`, `cluster_admin_password`
+- Adjust cluster config as needed (name, version, instance types)
+
+### Step 2: Build the provisioner container
+
+```bash
+make build
+```
+
+### Step 3: Run Terraform
+
+```bash
+make run CMD="bash -c 'cd PCA_Deployment_ROSA/terraform && terraform init'"
+make run CMD="bash -c 'cd PCA_Deployment_ROSA/terraform && terraform plan -out=tfplan'"
+make run CMD="bash -c 'cd PCA_Deployment_ROSA/terraform && terraform apply tfplan'"
+```
+
+If container name conflicts: `podman rm -f pca`
+
+### Step 4: Verify cluster access
+
+Terraform uses `rosa create admin` (via OCM API) to create the cluster-admin user. After apply completes, wait 2-5 minutes for IDP propagation:
+
+```bash
+make run CMD="bash -c 'cd PCA_Deployment_ROSA/terraform && terraform output -raw api_url'"
+make run CMD="oc login <api-url> --username cluster-admin --password '<password>' --insecure-skip-tls-verify"
+make run CMD="oc get nodes"
+```
+
+> **Note**: The admin IDP takes 2-5 minutes to propagate on ROSA HCP. Terraform waits 120s before attempting the grant.
+
+## Common Issues
+
+### Duplicate cluster name
+
+OCM rejects cluster creation if the name already exists in your org. Either choose a different `cluster_name` in tfvars or delete the old cluster first.
+
+### Invalid number of compute nodes
+
+`default_worker_replicas` must be a multiple of the number of private subnets (default: 3). E.g. use 3, 6, 9.
+
+### IAM roles already exist (EntityAlreadyExists)
+
+Account-level roles may exist from a prior deployment. Import them:
+
+```bash
+terraform import 'module.account_iam_resources.aws_iam_role.account_role[0]' ManagedOpenShift-HCP-ROSA-Installer-Role
+terraform import 'module.account_iam_resources.aws_iam_role.account_role[1]' ManagedOpenShift-HCP-ROSA-Support-Role
+terraform import 'module.account_iam_resources.aws_iam_role.account_role[2]' ManagedOpenShift-HCP-ROSA-Worker-Role
+```
+
+### OAuth 500 — identity mapping conflict
+
+The username `cluster-admin` is reserved for `rosa create admin`. **Never** add it to custom HTPasswd IDPs. If you hit a 500 with "cannot be claimed by identity", fix via OCM API:
+
+```bash
+rosa delete admin --cluster=<id> --yes
+# Create IDP with mapping_method "add" to bypass stale mapping:
+ACCESS_TOKEN=$(curl -s -X POST "https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token" \
+  -d "grant_type=refresh_token&client_id=cloud-services&refresh_token=$RHCS_TOKEN" | jq -r .access_token)
+curl -X POST "https://api.openshift.com/api/clusters_mgmt/v1/clusters/<cluster-id>/identity_providers" \
+  -H "Authorization: Bearer $ACCESS_TOKEN" -H "Content-Type: application/json" \
+  -d '{"type":"HTPasswdIdentityProvider","name":"cluster-admin","mapping_method":"add","htpasswd":{"users":{"items":[{"username":"cluster-admin","password":"<pw>"}]}}}'
+# Then grant cluster-admin group:
+curl -X POST "https://api.openshift.com/api/clusters_mgmt/v1/clusters/<cluster-id>/groups/cluster-admins/users" \
+  -H "Authorization: Bearer $ACCESS_TOKEN" -H "Content-Type: application/json" \
+  -d '{"id":"cluster-admin"}'
+```
+
+### OCM 500 error during cluster creation
+
+Transient server error. Wait a minute and re-run `terraform apply`.
+
+## Timing
+
+| Phase | Duration |
+|-------|----------|
+| VPC + Networking + IAM | ~2 min |
+| ROSA HCP Cluster | 25-40 min |
+| GPU Machine Pool + IDP | ~30 sec |
+| IDP propagation + cluster-admin grant | ~2-5 min |
+| **Total** | **~30-48 min** |
+
+## Verification
+
+```bash
+rosa describe cluster --cluster=<name>
+oc login <api-url> --username cluster-admin --password <password>
+oc get nodes
+```

.cursor/skills/teardown-rosa-aws/SKILL.md

@@ -0,0 +1,134 @@
+---
+name: teardown-rosa-aws
+description: >-
+  Tear down a ROSA HCP cluster on AWS, clean up leftover resources, and verify
+  complete removal. Use when the user asks to teardown, destroy, delete, remove,
+  or clean up a ROSA cluster, or mentions terraform destroy, make run destroy,
+  or AWS resource cleanup for this project.
+disable-model-invocation: true
+---
+
+# Tear Down ROSA HCP on AWS
+
+## Prerequisites
+
+- `.env` with valid AWS credentials and `RHCS_TOKEN`
+- Provisioner container image built (`make build`)
+- Terraform state present in `PCA_Deployment_ROSA/terraform/`
+
+## Workflow
+
+### Step 1: Confirm terraform state has resources
+
+```bash
+make run CMD="bash -c 'cd /workspace/PCA_Deployment_ROSA/terraform && terraform state list'"
+```
+
+If empty, skip to **Step 5: Verify** to check for orphaned AWS resources.
+
+### Step 2: Run terraform destroy
+
+```bash
+make run CMD="bash -c 'cd /workspace/PCA_Deployment_ROSA/terraform && terraform init -upgrade && terraform destroy -auto-approve'"
+```
+
+This typically takes 15-30 minutes. Monitor for completion.
+
+**If destroy completes successfully**, skip to **Step 5: Verify**.
+
+### Step 3: Handle stuck VPC deletion
+
+`terraform destroy` commonly gets stuck on VPC resources (subnets, internet gateway) because ROSA leaves behind unmanaged resources. If the IGW or subnets are stuck destroying for >5 minutes:
+
+1. Kill the stuck container: `podman kill pca; podman rm pca`
+
+2. Get the VPC ID from state:
+   ```bash
+   make run CMD="bash -c 'cd /workspace/PCA_Deployment_ROSA/terraform && terraform state show aws_vpc.rosa[0]'"
+   ```
+
+3. Run the cleanup script (substitute `VPC_ID` and `REGION`):
+   ```bash
+   make run CMD="bash /workspace/.cursor/skills/teardown-rosa-aws/scripts/cleanup-vpc.sh <VPC_ID> <REGION>"
+   ```
+
+   This script removes, in order:
+   - ELBv2 and classic load balancers
+   - VPC endpoints
+   - EC2 instances (terminates and waits)
+   - ENIs (detach + delete)
+   - Non-default security groups
+   - Target groups
+
+4. Re-run terraform destroy:
+   ```bash
+   make run CMD="bash -c 'cd /workspace/PCA_Deployment_ROSA/terraform && terraform destroy -auto-approve'"
+   ```
+
+### Step 4: Clean up orphaned OIDC providers
+
+ROSA may leave behind OIDC providers from current or previous deployments:
+
+```bash
+make run CMD="bash -c 'aws iam list-open-id-connect-providers --query \"OpenIDConnectProviderList[?contains(Arn, \\\`openshiftapps.com\\\`)].Arn\" --output text'"
+```
+
+Delete any found:
+
+```bash
+make run CMD="bash -c 'aws iam delete-open-id-connect-provider --open-id-connect-provider-arn <ARN>'"
+```
+
+### Step 5: Verify complete cleanup
+
+Run the verification script (reads cluster name and region from `terraform.tfvars`):
+
+```bash
+make run CMD="bash /workspace/.cursor/skills/teardown-rosa-aws/scripts/verify-cleanup.sh"
+```
+
+All 10 checks must pass. If any fail, delete the offending resources manually and re-verify.
+
+### Step 6: Confirm terraform state is empty
+
+```bash
+make run CMD="bash -c 'cd /workspace/PCA_Deployment_ROSA/terraform && terraform state list'"
+```
+
+Should return no output.
+
+## Common Issues
+
+### Container name conflict
+
+```bash
+podman kill pca; podman rm pca
+```
+
+### Subnets stuck deleting
+
+Usually caused by leftover ENIs from EC2 instances. Check for running instances:
+
+```bash
+make run CMD="bash -c 'aws ec2 describe-instances --region <REGION> --filters Name=vpc-id,Values=<VPC_ID> Name=instance-state-name,Values=running,pending,stopping,stopped --query \"Reservations[].Instances[].{ID:InstanceId,Name:Tags[?Key==\\\`Name\\\`].Value|[0]}\" --output table'"
+```
+
+Terminate them and wait before retrying.
+
+### IGW stuck deleting
+
+Usually caused by leftover NLBs. Check:
+
+```bash
+make run CMD="bash -c 'aws elbv2 describe-load-balancers --region <REGION> --query \"LoadBalancers[?VpcId==\\\`<VPC_ID>\\\`].LoadBalancerArn\" --output text'"
+```
+
+## Timing
+
+| Phase | Duration |
+|-------|----------|
+| Terraform destroy (cluster + IAM) | 10-20 min |
+| VPC cleanup (if stuck) | 5-10 min |
+| VPC destroy (after cleanup) | ~15 sec |
+| Verification | ~15 sec |
+| **Total** | **~15-30 min** |

.cursor/skills/teardown-rosa-aws/scripts/cleanup-vpc.sh

@@ -0,0 +1,95 @@
+#!/bin/bash
+set -euo pipefail
+
+VPC_ID="${1:?Usage: cleanup-vpc.sh <VPC_ID> <REGION>}"
+REGION="${2:?Usage: cleanup-vpc.sh <VPC_ID> <REGION>}"
+
+echo "=== Cleaning up leftover resources in VPC $VPC_ID ($REGION) ==="
+
+echo "--- Deleting ELBv2 load balancers..."
+LB_ARNS=$(aws elbv2 describe-load-balancers --region "$REGION" \
+	--query "LoadBalancers[?VpcId==\`$VPC_ID\`].LoadBalancerArn" --output text)
+for arn in $LB_ARNS; do
+	echo "  Deleting LB: $arn"
+	aws elbv2 delete-load-balancer --region "$REGION" --load-balancer-arn "$arn"
+done
+
+echo "--- Deleting classic ELBs..."
+CLB_NAMES=$(aws elb describe-load-balancers --region "$REGION" \
+	--query "LoadBalancerDescriptions[?VPCId==\`$VPC_ID\`].LoadBalancerName" --output text)
+for name in $CLB_NAMES; do
+	echo "  Deleting classic LB: $name"
+	aws elb delete-load-balancer --region "$REGION" --load-balancer-name "$name"
+done
+
+echo "--- Deleting VPC endpoints..."
+VPCE_IDS=$(aws ec2 describe-vpc-endpoints --region "$REGION" \
+	--filters "Name=vpc-id,Values=$VPC_ID" \
+	--query "VpcEndpoints[].VpcEndpointId" --output text)
+for vpce in $VPCE_IDS; do
+	echo "  Deleting VPC endpoint: $vpce"
+	aws ec2 delete-vpc-endpoints --region "$REGION" --vpc-endpoint-ids "$vpce"
+done
+
+echo "--- Terminating EC2 instances..."
+INSTANCE_IDS=$(aws ec2 describe-instances --region "$REGION" \
+	--filters "Name=vpc-id,Values=$VPC_ID" "Name=instance-state-name,Values=running,pending,stopping,stopped" \
+	--query "Reservations[].Instances[].InstanceId" --output text)
+if [ -n "$INSTANCE_IDS" ]; then
+	echo "  Terminating: $INSTANCE_IDS"
+	aws ec2 terminate-instances --region "$REGION" --instance-ids $INSTANCE_IDS >/dev/null
+	echo "  Waiting for termination..."
+	aws ec2 wait instance-terminated --region "$REGION" --instance-ids $INSTANCE_IDS
+	echo "  All instances terminated."
+fi
+
+echo "Waiting 15s for ENIs to release..."
+sleep 15
+
+echo "--- Detaching and deleting ENIs..."
+ENI_IDS=$(aws ec2 describe-network-interfaces --region "$REGION" \
+	--filters "Name=vpc-id,Values=$VPC_ID" \
+	--query "NetworkInterfaces[].NetworkInterfaceId" --output text)
+for eni in $ENI_IDS; do
+	ATT_ID=$(aws ec2 describe-network-interfaces --region "$REGION" \
+		--network-interface-ids "$eni" \
+		--query "NetworkInterfaces[0].Attachment.AttachmentId" --output text 2>/dev/null || echo "None")
+	if [ "$ATT_ID" != "None" ] && [ -n "$ATT_ID" ]; then
+		echo "  Detaching $eni..."
+		aws ec2 detach-network-interface --region "$REGION" --attachment-id "$ATT_ID" --force 2>&1 || true
+	fi
+done
+sleep 10
+ENI_IDS=$(aws ec2 describe-network-interfaces --region "$REGION" \
+	--filters "Name=vpc-id,Values=$VPC_ID" \
+	--query "NetworkInterfaces[].NetworkInterfaceId" --output text)
+for eni in $ENI_IDS; do
+	echo "  Deleting ENI: $eni"
+	aws ec2 delete-network-interface --region "$REGION" --network-interface-id "$eni" 2>&1 || true
+done
+
+echo "--- Deleting non-default security groups..."
+SG_IDS=$(aws ec2 describe-security-groups --region "$REGION" \
+	--filters "Name=vpc-id,Values=$VPC_ID" \
+	--query 'SecurityGroups[?GroupName!=`default`].GroupId' --output text)
+for sg in $SG_IDS; do
+	echo "  Deleting SG: $sg"
+	aws ec2 delete-security-group --region "$REGION" --group-id "$sg" 2>&1 || true
+done
+
+echo "--- Deleting target groups..."
+TG_ARNS=$(aws elbv2 describe-target-groups --region "$REGION" \
+	--query "TargetGroups[?VpcId==\`$VPC_ID\`].TargetGroupArn" --output text)
+for tg in $TG_ARNS; do
+	echo "  Deleting TG: $tg"
+	aws elbv2 delete-target-group --region "$REGION" --target-group-arn "$tg" 2>&1 || true
+done
+
+echo ""
+echo "--- Remaining ENIs:"
+aws ec2 describe-network-interfaces --region "$REGION" \
+	--filters "Name=vpc-id,Values=$VPC_ID" \
+	--query "NetworkInterfaces[].{ID:NetworkInterfaceId,Status:Status}" --output table 2>/dev/null || echo "  None"
+
+echo ""
+echo "=== VPC cleanup complete ==="