Environment
- Kubernetes: v1.35.3
- Runtime: cri-dockerd
- Docker Engine: v29.4.0
- NVIDIA DRA Driver: v25.12.0
- NVIDIA Driver: 595.71.05
- OS: Ubuntu 22.04
Docker daemon configuration:
{
"data-root": "/data/docker",
"default-runtime": "nvidia",
"exec-opts": [
"native.cgroupdriver=systemd"
],
"features": {
"cdi": true
},
"log-opts": {
"max-file": "3",
"max-size": "5m"
},
"registry-mirrors": [
"https://docker.1ms.run"
],
"runtimes": {
"nvidia": {
"args": [],
"path": "nvidia-container-runtime"
}
}
}
Problem
I am testing Kubernetes DRA with the NVIDIA DRA driver and CDI enabled.
The DRA driver appears to work correctly:
- ResourceClaim is allocated successfully
- CDI YAML files are generated under
/var/run/cdi
- The kubelet plugin logs show successful preparation of CDI devices
Example plugin log:
Returning newly prepared devices for claim 'gpu-test2/pod-shared-gpu-xxxx':
[{[gpu] node2 gpu-0 [k8s.gpu.nvidia.com/claim=xxxx-gpu-0]}]
Generated CDI specs exist:
ls /var/run/cdi/
k8s.gpu.nvidia.com-claim_xxx.yaml
nvidia.yaml
However, inside the Kubernetes Pod:
/dev/nvidia* devices are missing
nvidia-smi is unavailable
- no GPU devices are injected into the container
Example:
ls /dev/
core fd null pts random shm stderr stdin stdout tty urandom zero
No NVIDIA devices are present.
Observation
This seems to indicate that:
- Kubernetes DRA allocation succeeds
- CDI specs are generated correctly
- But CDI devices are not propagated through cri-dockerd into Docker runtime
It looks like the CDIDevices field from CRI may not be forwarded by cri-dockerd.
Additional Notes
Docker CDI itself appears functional.
For example, Docker standalone CDI/device injection works correctly outside Kubernetes.
The issue only occurs in the Kubernetes + cri-dockerd path.
Questions
- Does cri-dockerd currently support Kubernetes DRA CDI device propagation?
- Is the CRI
CDIDevices field implemented and forwarded to Docker Engine?
- Is Kubernetes DRA officially supported with cri-dockerd?
- If not supported yet, are there plans to add support?
Thanks.
Environment
Docker daemon configuration:
{ "data-root": "/data/docker", "default-runtime": "nvidia", "exec-opts": [ "native.cgroupdriver=systemd" ], "features": { "cdi": true }, "log-opts": { "max-file": "3", "max-size": "5m" }, "registry-mirrors": [ "https://docker.1ms.run" ], "runtimes": { "nvidia": { "args": [], "path": "nvidia-container-runtime" } } }Problem
I am testing Kubernetes DRA with the NVIDIA DRA driver and CDI enabled.
The DRA driver appears to work correctly:
/var/run/cdiExample plugin log:
Generated CDI specs exist:
However, inside the Kubernetes Pod:
/dev/nvidia*devices are missingnvidia-smiis unavailableExample:
No NVIDIA devices are present.
Observation
This seems to indicate that:
It looks like the
CDIDevicesfield from CRI may not be forwarded by cri-dockerd.Additional Notes
Docker CDI itself appears functional.
For example, Docker standalone CDI/device injection works correctly outside Kubernetes.
The issue only occurs in the Kubernetes + cri-dockerd path.
Questions
CDIDevicesfield implemented and forwarded to Docker Engine?Thanks.