Apache Ozone CSI - PVC Not Mounting, Pod Stuck in ContainerCreating

[pradippipaliya-inventyv]

Hi Team,

I have deployed Apache Ozone in a Kubernetes cluster following the official documentation: Apache Ozone CSI Documentation. My cluster setup consists of:

1 Master Node
3 Worker Nodes (Each running an Ozone Datanode)
All CSI-related components have been created successfully, and the PersistentVolume (PV) and PersistentVolumeClaim (PVC) are provisioned correctly as per the pv-test/ directory. However, the PVC is not mounting inside the pod, and the pod remains stuck in the ContainerCreating state.

Observed Issue
The pod events show the following error:

Warning FailedMount 3m49s (x72 over 135m) kubelet MountVolume.SetUp failed for volume "pvc-af7f72da-2c41-4199-b972-893b93bf909b" : rpc error: code = Unknown desc =

Upon inspecting the kubelet logs on the node where the pod is scheduled, I found these error messages:

Mar 06 09:18:57 ip-172-31-28-828 kubelet[957]: E0306 09:18:57.210571     957 pod_workers.go:1298] "Error syncing pod, skipping" err="unmounted volumes=[webroot], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="default/ozone-csi-test-webserver-59798c4494-n6pnz" podUID="73482f4f-3d6c-4ad1-86cf-8e7bb38be973"

Mar 06 09:20:28 ip-172-31-28-828 kubelet[957]: E0306 09:20:28.125912     957 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/org.apache.hadoop.ozone^pvc-af7f72da-2c41-4199-b972-893b93bf909b podName: nodeName:}" failed. No retries permitted until 2025-03-06 09:22:30.12589319 +0000 UTC m=+14147.136500718 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "pvc-af7f72da-2c41-4199-b972-893b93bf909b" (UniqueName: "kubernetes.io/csi/org.apache.hadoop.ozone^pvc-af7f72da-2c41-4199-b972-893b93bf909b") pod "ozone-csi-test-webserver-59798c4494-n6pnz" (UID: "73482f4f-3d6c-4ad1-86cf-8e7bb38be973") : rpc error: code = Unknown desc =

Troubleshooting Steps Taken
Verified CSI Driver Deployment: All components in the CSI directory are running as expected.
Checked PV and PVC: Both are in the Bound state.
Reviewed Kubelet Logs: Found timeout errors and rpc error: code = Unknown desc = messages.
Checked Node Status: All worker and master nodes are in a Ready state.
Examined StorageClass & Driver Configurations: No apparent misconfigurations were found.

Request for Assistance
Has anyone encountered a similar issue with the Apache Ozone CSI driver? Any guidance on resolving the PVC mounting failure and rpc error: code = Unknown desc = would be greatly appreciated.

Thanks in advance!

[ChenSammi]

Since HDDS-7307, Ozone has added a new port 19878 for the S3 gateway web UI. So S3 gateway now have two ports, 9878 and 19878. No sure if it's related since I don't have the handy environment to verify that.