OSDOC-20065: AutoNode Post-GA Updates

_topic_maps/_topic_map_rosa_hcp.yml

@@ -401,6 +401,8 @@ Topics:
     File: rosa-troubleshooting-networking
   - Name: Verifying node health
     File: verifying-node-health
+  - Name: Troubleshooting Red Hat build of Karpenter
+    File: rosa-troubleshooting-autonode
 #  cannot create resource "namespaces", cannot patch resource "nodes"
 #  - Name: Troubleshooting CRI-O container runtime issues
 #    File: troubleshooting-crio-issues

modules/autonode-troubleshooting.adoc

@@ -0,0 +1,45 @@
+:_mod-docs-content-type: CONCEPT
+[id="autonode-troubleshooting-overview_{context}"]
+= About {autonode} troubleshooting
+
+[role="_abstract"]
+The {autonode} uses the Karpenter controller to provide automated node provisioning in OpenShift clusters. When the {autonode} fails to provision nodes or reports configuration errors, you can diagnose and resolve issues by examining Karpenter resource status, AWS infrastructure tags, and AWS Identity and and Access Management (IAM) permissions.
+
+{autonode} troubleshooting addresses four main failure categories:
+
+Controller deployment issues:: Karpenter Custom Resource Definitions (CRDs) do not appear after enabling the {autonode}, preventing you from creating or managing Karpenter resources.
+
+AWS resource discovery failures:: The EC2NodeClass resource cannot discover AWS subnets or security groups because required discovery tags are missing or incorrect.
+
+IAM permission issues:: The control plane operator lacks authorization to create tags on AWS resources, preventing Karpenter from properly configuring security groups.
+
+{cluster-manager} capability requirements:: Your organization does not have the required {cluster-manager-first} capabilities to create {autonode} clusters on the designated provision shard.
+
+[id="autonode-troubleshooting-diagnostic-workflow_{context}"]
+== General diagnostic workflow
+
+Follow this workflow to diagnose and resolve {autonode} issues:
+
+. Identify the symptom by checking which Karpenter resources are missing or not ready.
+. Gather diagnostic data using `oc` commands to inspect custom resource status and AWS CLI commands to verify infrastructure configuration.
+. Analyze the root cause by comparing observed resource state with expected configuration.
+. Apply the resolution by adding missing tags, updating IAM policies, or verifying capabilities.
+. Verify the fix by confirming that Karpenter resources reach ready status and automatic scaling functions correctly.
+
+[id="autonode-troubleshooting-escalating_{context}"]
+== When to escalate to Red{nbsp}Hat Support
+
+Escalate to Red{nbsp}Hat Support in these situations:
+
+* Karpenter CRDs do not appear after 5 minutes and the Karpenter controller pod is not running.
+* EC2NodeClass remains not ready after adding all required discovery tags and waiting 60 seconds.
+* IAM policies are correctly attached but CloudTrail continues showing authorization errors.
+* You have verified {cluster-manager} capabilities but cluster creation on the designated provision shard still fails.
+
+When escalating, provide:
+
+* Cluster ID
+* Output of `rosa describe cluster -c $CLUSTER_ID -o json`
+* Output of `oc get ec2nodeclass/default -o json`
+* Output of `oc get pods -n kube-system`
+* CloudTrail event details for authorization errors

modules/karpenter-resource-readiness.adoc

@@ -0,0 +1,59 @@
+:_mod-docs-content-type: REFERENCE
+[id="karpenter-resource-readiness-conditions_{context}"]
+= Karpenter resource readiness conditions
+
+[role="_abstract"]
+The EC2NodeClass resource reports readiness through status conditions that indicate whether Karpenter successfully discovered and validated AWS resources. Each condition reports a boolean status and explains why the condition is true or false. Understanding these conditions helps you identify which AWS resources require configuration changes.
+
+.EC2NodeClass status conditions
+[cols="1,1,3", options="header"]
+|===
+|Condition
+|Status values
+|Meaning
+
+|`AMIsReady`
+|`True` or `False`
+|The Amazon Machine Images (AMI) selector matched available AMIs in the AWS region.
+If `False`, the specified AMI selector did not match any AMIs accessible to your AWS account.
+
+|`SubnetsReady`
+|`True` or `False`
+|The subnet selector matched private subnets with the required Karpenter discovery tag.
+If `False`, no subnets in your VPC have the `karpenter.sh/discovery` tag with the value expected by the EC2NodeClass.
+
+|`SecurityGroupsReady`
+|`True` or `False`
+|The security group selector matched security groups with the required Karpenter discovery tag.
+If `False`, no security groups have the `karpenter.sh/discovery` tag with the value expected by the EC2NodeClass.
+
+|`InstanceProfileReady`
+|`True` or `False`
+|The IAM instance profile exists and is accessible to Karpenter.
+If `False`, the specified instance profile name is incorrect or the profile does not exist in your AWS account.
+
+|`ValidationSucceeded`
+|`True` or `False`
+|The EC2NodeClass specification passed validation.
+If `False`, the EC2NodeClass contains invalid configuration values or references non-existent resources.
+
+|`Ready`
+|`True` or `False`
+|Overall readiness status.
+This condition is `True` only when all dependent conditions (`AMIsReady`, `SubnetsReady`, `SecurityGroupsReady`, `InstanceProfileReady`, `ValidationSucceeded`) are `True`.
+If `False`, the message field lists which conditions are not ready.
+|===
+
+[id="karpenter-resource-readiness-conditions-common-failures_{context}"]
+== Common failure messages
+
+When EC2NodeClass conditions report `False` status, the message field provides details about the failure:
+
+SubnetSelector did not match any Subnets:: No subnets in your VPC have the required `karpenter.sh/discovery` tag. 
+Add the tag to all private subnets where Karpenter should provision nodes.
+
+SecurityGroupSelector did not match any SecurityGroups:: No security groups have the required `karpenter.sh/discovery` tag.
+Add the tag to the default security group and any additional security groups Karpenter should use.
+
+UnhealthyDependents:: The `Ready` condition shows this message when one or more dependent conditions are `False`.
+The message lists which specific conditions are not ready, for example: `SecurityGroupsReady=False, SubnetsReady=False`.

modules/query-ec2nodeclass-condition-status.adoc

@@ -0,0 +1,31 @@
+:_mod-docs-content-type: PROCEDURE
+[id="query-ec2nodeclass-condition-status_{context}"]
+= Query EC2NodeClass condition status
+
+[role="_abstract"]
+Use the `oc` command-line interface to inspect EC2NodeClass status conditions and identify which AWS resources require configuration changes.
+
+.Procedure
+
+. To view the full condition status for an EC2NodeClass, run:
++
+[source,terminal]
+----
+$ oc get ec2nodeclass/default -o json | jq .status.conditions
+----
++
+The output displays all condition types with their status, reason, and explanatory message.
+
+. Optional: To view only the overall `Ready` condition, run:
++
+[source,terminal]
+----
+$ oc get ec2nodeclass/default -o json | jq '.status.conditions[] | select(.type=="Ready")'
+----
++
+The output shows the `Ready` condition type, status, reason, and message explaining why the condition is in that state.
+
+.Verification
+
+* Review the output to identify which conditions show `"status": "False"`.
+* Check the `message` field for each failed condition to determine what AWS resources need configuration changes.

modules/resolve-ec2nodeclass-not-ready.adoc

@@ -0,0 +1,139 @@
+:_mod-docs-content-type: PROCEDURE
+[id="resolve-ec2nodeclass-not-ready-status_{context}"]
+= Resolve EC2NodeClass not ready status
+
+[role="_abstract"]
+Karpenter discovers AWS subnets and security groups by searching for resources tagged with a cluster-specific discovery tag. When the EC2NodeClass resource reports `SubnetsReady` or `SecurityGroupsReady` conditions as `False`, you need to add the required discovery tag to your AWS VPC resources.
+
+.Procedure
+
+. Check the EC2NodeClass status:
++
+[source,terminal]
+----
+$ oc get ec2nodeclass/default -o json | jq .status.conditions
+----
++
+This command displays all status conditions for the default EC2NodeClass.
+
+. Identify which conditions are not ready by looking for entries with `"status": "False"`.
++
+Common failures are `SubnetsReady` and `SecurityGroupsReady`.
+
+. If `SubnetsReady` shows `False` with message "SubnetSelector did not match any Subnets", private subnets in your VPC are missing the required discovery tag.
+Continue to the next step.
+
+. If `SecurityGroupsReady` shows `False` with message "SecurityGroupSelector did not match any SecurityGroups", security groups are missing the required discovery tag.
+The same resolution process applies for both subnets and security groups.
+
+. Get the Karpenter discovery tag value from the EC2NodeClass specification:
++
+[source,terminal]
+----
+$ oc get ec2nodeclass/default -o json | jq -r '.spec.subnetSelectorTerms[0].tags["karpenter.sh/discovery"]'
+----
++
+.Example output
+[source,terminal]
+----
+2m46sfl77ldkukbskhhs8uk5or2gm2p8
+----
++
+This cluster-specific value must be added as a tag to AWS resources.
+
+. List private subnets in your cluster VPC:
++
+[source,terminal]
+----
+$ aws ec2 describe-subnets --filters "Name=vpc-id,Values=<vpc_id>" \
+  --query 'Subnets[?MapPublicIpOnLaunch==`false`].[SubnetId,Tags]' --output table
+----
++
+Replace `<vpc_id>` with your cluster's VPC ID.
+
+. For each private subnet, verify whether the Karpenter discovery tag is present:
++
+[source,terminal]
+----
+$ aws ec2 describe-subnets --subnet-ids <subnet_id> \
+  --query 'Subnets[0].Tags[?Key==`karpenter.sh/discovery`].Value' --output text
+----
++
+Replace `<subnet_id>` with the subnet ID to check.
+
+. If the discovery tag is missing, add it to the subnet:
++
+[source,terminal]
+----
+$ aws ec2 create-tags --resources <subnet_id> \
+  --tags Key=karpenter.sh/discovery,Value=<discovery_tag_value>
+----
++
+Replace `<subnet_id>` with the subnet ID and `<discovery_tag_value>` with the value from step 5.
+
+. Repeat steps 7 and 8 for all private subnets in your VPC.
+
+. Verify that the default security group has the discovery tag:
++
+[source,terminal]
+----
+$ aws ec2 describe-security-groups --group-ids <default_sg_id> \
+  --query 'SecurityGroups[0].Tags[?Key==`karpenter.sh/discovery`].Value' --output text
+----
++
+Replace `<default_sg_id>` with your cluster's default security group ID.
+
+. If the discovery tag is missing from the security group, add it:
++
+[source,terminal]
+----
+$ aws ec2 create-tags --resources <default_sg_id> \
+  --tags Key=karpenter.sh/discovery,Value=<discovery_tag_value>
+----
++
+Replace `<default_sg_id>` with the security group ID and `<discovery_tag_value>` with the value from step 5.
+
+. Wait 30 to 60 seconds for Karpenter to re-scan AWS resources.
+
+. Verify that the EC2NodeClass Ready condition is now `True`:
++
+[source,terminal]
+----
+$ oc get ec2nodeclass/default -o json | jq '.status.conditions[] | select(.type=="Ready")'
+----
++
+The output should show `"status": "True"`.
+
+. If the Ready condition is still `False`, check for other condition failures in the output and repeat the appropriate steps.
+
+.Verification
+
+* The EC2NodeClass Ready condition reports `True` status:
++
+[source,terminal]
+----
+$ oc get ec2nodeclass/default -o json | jq '.status.conditions[] | select(.type=="Ready")'
+----
+
+* The SubnetsReady and SecurityGroupsReady conditions report `True` status:
++
+[source,terminal]
+----
+$ oc get ec2nodeclass/default -o json | jq '.status.conditions[] | select(.type=="SubnetsReady" or .type=="SecurityGroupsReady")'
+----
+
+* All required AWS resources have the correct `karpenter.sh/discovery` tag:
++
+[source,terminal]
+----
+$ aws ec2 describe-subnets --subnet-ids <subnet_id> \
+  --query 'Subnets[0].Tags[?Key==`karpenter.sh/discovery`].Value' --output text
+----
+
+.Troubleshooting
+
+If adding tags fails with authorization errors:: You need to update IAM permissions for the control plane operator.
+See "Resolve IAM permissions for AWS resource tagging".
+
+If the EC2NodeClass remains not ready after adding tags:: Verify that the tag value exactly matches the value in the EC2NodeClass specification.
+Tag values are case-sensitive and must match character-for-character.

modules/resolve-iam-permissions-tagging.adoc

@@ -0,0 +1,91 @@
+:_mod-docs-content-type: PROCEDURE
+[id="resolve-iam-permissions-for-tagging_{context}"]
+= Resolve IAM permissions for AWS resource tagging
+
+[role="_abstract"]
+The Karpenter controller requires permission to create tags on AWS security groups. When the control plane operator IAM role lacks the `ec2:CreateTags` permission, Karpenter cannot automatically tag security groups with the discovery tag.
+
+.Procedure
+
+. Identify the IAM role used by the control plane operator:
++
+[source,terminal]
+----
+$ aws iam list-roles | grep kube-system-control-plane-operator
+----
++
+The role name follows the pattern `<cluster_prefix>-kube-system-control-plane-operator`.
+
+. Optional: Verify the authorization error in AWS CloudTrail:
+.. Navigate to the AWS CloudTrail console.
+.. Search for events matching these criteria:
++
+--
+* Event name: `CreateTags`
+* Error code: `Client.UnauthorizedOperation`
+* User identity: Contains `control-plane-operator`
+--
++
+If you find matching events, the error message confirms that the IAM role lacks `ec2:CreateTags` permission.
+
+. Create an IAM policy document that allows creating tags:
++
+[source,terminal]
+----
+$ cat > /tmp/create-tags-policy.json << 'POLICY_EOF'
+{
+    "Version": "2012-10-17",
+    "Statement": [
+        {
+            "Sid": "AllowCreateTags",
+            "Effect": "Allow",
+            "Action": [
+                "ec2:CreateTags"
+            ],
+            "Resource": "*"
+        }
+    ]
+}
+POLICY_EOF
+----
+
+. Attach the policy to the control plane operator IAM role:
++
+[source,terminal]
+----
+$ aws iam put-role-policy \
+  --role-name <cluster_prefix>-kube-system-control-plane-operator \
+  --policy-name KarpenterCreateTags \
+  --policy-document file:///tmp/create-tags-policy.json
+----
++
+Replace `<cluster_prefix>` with your cluster's prefix from step 1.
+
+. Verify that the policy is attached to the role:
++
+[source,terminal]
+----
+$ aws iam get-role-policy \
+  --role-name <cluster_prefix>-kube-system-control-plane-operator \
+  --policy-name KarpenterCreateTags
+----
++
+The output shows the policy document you attached.
+
+.Verification
+
+* The IAM policy is successfully attached to the control plane operator role:
++
+[source,terminal]
+----
+$ aws iam get-role-policy \
+  --role-name <cluster_prefix>-kube-system-control-plane-operator \
+  --policy-name KarpenterCreateTags
+----
++
+This command returns the policy document without errors.
+
+* AWS CloudTrail no longer shows `Client.UnauthorizedOperation` errors for CreateTags operations performed by the control plane operator.
+
+* If your EC2NodeClass previously showed `SecurityGroupsReady = False`, you can now manually add the Karpenter discovery tag to the default security group.
+After you add the tag, the authorization error will not recur because the IAM role now has the required permissions.