nvidia-ctk cdi generate: libdxcore.so not found on WSL2 despite being present

[tyeth-ai-assisted]

Description

nvidia-ctk cdi generate on WSL2 correctly auto-detects WSL mode but fails to locate libdxcore.so, logging:

level=warning msg="Could not locate libdxcore.so: pattern libdxcore.so not found"

The library IS present on the system at /usr/lib/x86_64-linux-gnu/libdxcore.so and is registered in ldconfig:

$ ldconfig -p | grep dxcore
libdxcore.so (libc6,x86-64) => /usr/lib/wsl/lib/libdxcore.so

Without libdxcore.so in the CDI spec, NVML initialization fails inside containers with Failed to initialize NVML: N/A, because libdxcore.so bridges the Linux NVML API to the Windows DirectX GPU Kernel via /dev/dxg.

Environment

• OS: Windows WSL2 (Linux 6.6.87.2-microsoft-standard-WSL2)
• GPU: NVIDIA GeForce RTX 5070 Laptop GPU
• Driver: NVIDIA 595.71, CUDA 13.2
• nvidia-container-toolkit: installed from nvidia apt repo (bundled in OpenShell cluster image)
• libdxcore.so locations:
  • /usr/lib/wsl/lib/libdxcore.so (WSL mount)
  • /usr/lib/x86_64-linux-gnu/libdxcore.so (symlink/copy in gateway container)

Steps to Reproduce

1. Run on WSL2 with NVIDIA GPU
2. nvidia-ctk cdi generate --output=/tmp/nvidia.yaml
3. Observe warning: "Could not locate libdxcore.so: pattern libdxcore.so not found"
4. Inspect generated CDI spec — libdxcore.so not in mounts
5. Start a container using the CDI spec
6. nvidia-smi inside container fails: Failed to initialize NVML: N/A

Expected Behavior

nvidia-ctk cdi generate should find libdxcore.so at /usr/lib/wsl/lib/libdxcore.so (or via ldconfig) and include it in the CDI spec mounts.

Workaround

Manually add libdxcore.so to the generated CDI spec:

# Add to containerEdits.mounts:
- hostPath: /usr/lib/x86_64-linux-gnu/libdxcore.so
  containerPath: /usr/lib/x86_64-linux-gnu/libdxcore.so
  options:
    - ro
    - nosuid
    - nodev
    - rbind
    - rprivate

And add --folder /usr/lib/x86_64-linux-gnu to the update-ldcache hook args.

After this fix plus setting mode = "cdi" in the nvidia container runtime config, nvidia-smi and NVML work correctly inside containers on WSL2.

Impact

This bug breaks the entire NVIDIA k8s device plugin stack on WSL2 when using CDI mode, since the device plugin requires NVML initialization which depends on libdxcore.so.

[tyeth]

I'm the human filing these issues, feel free to direct questions and/or rage my way.
Relates to setting up Nvidia NemoClaw issue NVIDIA/NemoClaw#208

[mattezell]

Can confirm this bug affects the full NemoClaw/OpenShell stack on WSL2. The missing libdxcore.so in the CDI spec causes nvidia-device-plugin to crash with Failed to initialize NVML: Not Supported.

Additionally, nvidia-ctk cdi generate in WSL mode only creates a name: all device entry. The k8s device plugin allocates by UUID, so pods requesting nvidia.com/gpu=GPU-xxxx fail with unresolvable CDI devices. A UUID-specific device entry must be added manually.

Workaround for both issues automated here: https://github.com/thenewguardai/tng-nemoclaw-quickstart/blob/main/scripts/wsl2-gpu-deploy.sh

[cdesiniotis]

@tyeth can you provide all logs when running nvidia-ctk --debug cdi generate? Is libdxcore the only library missing? What about the other libraries from this list?

[tyeth]

Hey, apologies that this is not quite what I imagine you want, which would be the same setup steps my agent did before to create a container image with the toolkit, but I haven't got access right now to examine fully what went on, so I've gone into WSL2 myself and followed the ubuntu install guide (add stable apt repo, update, install), then run that command (with Warnings for the sandbox utils and multplie drivers):

Full log

tyeth@framework16:~/dev-projects/misc/nvidia-ctk$ nvidia-ctk --debug cdi generate
DEBU[0000] Locating NVIDIA CDI Hook CLI as nvidia-cdi-hook
DEBU[0000] Locating "nvidia-cdi-hook" in [/home/tyeth/.local/bin /home/tyeth/.platformio/.cache /usr/bin /workspaces/gcc-arm-none-eabi/bin /home/tyeth/.vscode-server/data/User/globalStorage/github.copilot-chat/debugCommand /home/tyeth/.vscode-server/data/User/globalStorage/github.copilot-chat/copilotCli /home/tyeth/.vscode-server/bin/585eba7c0c34fd6b30faac7c62a42050bfbc0086/bin/remote-cli /usr/local/sbin /usr/local/bin /usr/sbin /sbin /bin /usr/games /usr/local/games /usr/lib/wsl/lib /mnt/c/Python314/Scripts /mnt/c/Python314 /mnt/c/WINDOWS/system32 /mnt/c/WINDOWS /mnt/c/WINDOWS/System32/Wbem /mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0 /mnt/c/WINDOWS/System32/OpenSSH /mnt/c/dev/software/vscode/bin /mnt/c/Program Files/Git/cmd /mnt/c/Program Files/NVIDIA Corporation/NVIDIA App/NvDLISR /mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common /mnt/c/Program Files/Tailscale /mnt/c/Program Files/cursor/resources/app/bin /mnt/c/Program Files/Arduino CLI /mnt/c/Program Files/nodejs /mnt/c/ProgramData/chocolatey/bin /Docker/host/bin /mnt/c/Program Files/GitHub CLI /mnt/c/Program Files/Microsoft SQL Server/170/Tools/Binn /mnt/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn /mnt/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit /mnt/c/Program Files/dotnet /mnt/c/Users/tyeth/.platformio/.cache /mnt/c/Users/tyeth/.local/bin /mnt/c/Users/tyeth/AppData/Local/Microsoft/WindowsApps /mnt/c/Users/tyeth/AppData/Local/Python/bin /mnt/c/Users/tyeth/AppData/Local/PowerToys/DSCModules /mnt/c/Users/tyeth/AppData/Local/Microsoft/WinGet/Links /mnt/c/Users/tyeth/AppData/Local/Python/pythoncore-3.13-64/Scripts /mnt/c/Users/tyeth/AppData/Roaming/npm /mnt/c/Users/tyeth/AppData/Roaming/Python/Python314/Scripts /mnt/c/Users/tyeth/.wokwi/bin /mnt/c/Users/tyeth/.dotnet/tools /snap/bin /home/tyeth/.nvm/versions/node/v22.22.1/bin]
DEBU[0000] Checking candidate '/usr/bin/nvidia-cdi-hook'
DEBU[0000] Found 1 candidates; ignoring further candidates
DEBU[0000] Found nvidia-cdi-hook candidates: [/usr/bin/nvidia-cdi-hook]
DEBU[0000] Using NVIDIA CDI Hook CLI path nvidia-cdi-hook
DEBU[0000] Opening ld.conf at /etc/ld.so.cache
DEBU[0000] Locating "libnvidia-sandboxutils.so.1" in [/ /usr/lib64 /usr/lib/x86_64-linux-gnu /usr/lib/aarch64-linux-gnu /usr/lib/x86_64-linux-gnu/nvidia/current /usr/lib/aarch64-linux-gnu/nvidia/current /lib64 /lib/x86_64-linux-gnu /lib/aarch64-linux-gnu /lib/x86_64-linux-gnu/nvidia/current /lib/aarch64-linux-gnu/nvidia/current]
WARN[0000] Ignoring error in locating libnvidia-sandboxutils.so.1: libnvidia-sandboxutils.so.1: not found
libnvidia-sandboxutils.so.1: not found
WARN[0000] Failed to init nvsandboxutils: ERROR_LIBRARY_LOAD; ignoring
DEBU[0000] Opening ld.conf at /etc/ld.so.cache
DEBU[0000] Locating "libnvidia-ml.so.1" in [/ /usr/lib64 /usr/lib/x86_64-linux-gnu /usr/lib/aarch64-linux-gnu /usr/lib/x86_64-linux-gnu/nvidia/current /usr/lib/aarch64-linux-gnu/nvidia/current /lib64 /lib/x86_64-linux-gnu /lib/aarch64-linux-gnu /lib/x86_64-linux-gnu/nvidia/current /lib/aarch64-linux-gnu/nvidia/current]
INFO[0000] Using /usr/lib/wsl/lib/libnvidia-ml.so.1
DEBU[0000] Is WSL-based system? true: found DXCore library
DEBU[0000] Is Tegra-based system? false: /sys/devices/soc0/family file not found
DEBU[0000] Is NVML-based system? true: found NVML library
DEBU[0000] Has an integrated GPU? false: no integrated GPUs found
INFO[0000] Auto-detected mode as 'wsl'
DEBU[0000] Locating /dev/dxg
DEBU[0000] Locating "/dev/dxg" in [/ /dev]
DEBU[0000] Checking candidate '/dev/dxg'
DEBU[0000] Located /dev/dxg as [/dev/dxg]
INFO[0000] Selecting /dev/dxg as /dev/dxg
WARN[0000] Found multiple driver store paths: [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2 /usr/lib/wsl/drivers/u0198975.inf_amd64_cab3843582148687 /usr/lib/wsl/drivers/kipudrv.inf_amd64_1a1aa059597c4810]
INFO[0000] Using WSL driver store paths: [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2 /usr/lib/wsl/drivers/u0198975.inf_amd64_cab3843582148687 /usr/lib/wsl/drivers/kipudrv.inf_amd64_1a1aa059597c4810]
DEBU[0000] Locating libcuda.so.1.1
DEBU[0000] Locating "libcuda.so.1.1" in [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2 /usr/lib/wsl/drivers/u0198975.inf_amd64_cab3843582148687 /usr/lib/wsl/drivers/kipudrv.inf_amd64_1a1aa059597c4810 /usr/lib/wsl/lib]
DEBU[0000] Checking candidate '/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libcuda.so.1.1'
DEBU[0000] Found 1 candidates; ignoring further candidates
DEBU[0000] Located libcuda.so.1.1 as [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libcuda.so.1.1]
INFO[0000] Selecting /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libcuda.so.1.1 as /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libcuda.so.1.1
DEBU[0000] Locating libcuda_loader.so
DEBU[0000] Locating "libcuda_loader.so" in [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2 /usr/lib/wsl/drivers/u0198975.inf_amd64_cab3843582148687 /usr/lib/wsl/drivers/kipudrv.inf_amd64_1a1aa059597c4810 /usr/lib/wsl/lib]
DEBU[0000] Checking candidate '/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libcuda_loader.so'
DEBU[0000] Found 1 candidates; ignoring further candidates
DEBU[0000] Located libcuda_loader.so as [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libcuda_loader.so]
INFO[0000] Selecting /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libcuda_loader.so as /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libcuda_loader.so
DEBU[0000] Locating libnvidia-ptxjitcompiler.so.1
DEBU[0000] Locating "libnvidia-ptxjitcompiler.so.1" in [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2 /usr/lib/wsl/drivers/u0198975.inf_amd64_cab3843582148687 /usr/lib/wsl/drivers/kipudrv.inf_amd64_1a1aa059597c4810 /usr/lib/wsl/lib]
DEBU[0000] Checking candidate '/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ptxjitcompiler.so.1'
DEBU[0000] Found 1 candidates; ignoring further candidates
DEBU[0000] Located libnvidia-ptxjitcompiler.so.1 as [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ptxjitcompiler.so.1]
INFO[0000] Selecting /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ptxjitcompiler.so.1 as /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ptxjitcompiler.so.1
DEBU[0000] Locating libnvidia-ml.so.1
DEBU[0000] Locating "libnvidia-ml.so.1" in [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2 /usr/lib/wsl/drivers/u0198975.inf_amd64_cab3843582148687 /usr/lib/wsl/drivers/kipudrv.inf_amd64_1a1aa059597c4810 /usr/lib/wsl/lib]
DEBU[0000] Checking candidate '/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ml.so.1'
DEBU[0000] Found 1 candidates; ignoring further candidates
DEBU[0000] Located libnvidia-ml.so.1 as [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ml.so.1]
INFO[0000] Selecting /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ml.so.1 as /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ml.so.1
DEBU[0000] Locating libnvidia-ml_loader.so
DEBU[0000] Locating "libnvidia-ml_loader.so" in [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2 /usr/lib/wsl/drivers/u0198975.inf_amd64_cab3843582148687 /usr/lib/wsl/drivers/kipudrv.inf_amd64_1a1aa059597c4810 /usr/lib/wsl/lib]
DEBU[0000] Checking candidate '/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ml_loader.so'
DEBU[0000] Found 1 candidates; ignoring further candidates
DEBU[0000] Located libnvidia-ml_loader.so as [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ml_loader.so]
INFO[0000] Selecting /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ml_loader.so as /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ml_loader.so
DEBU[0000] Locating libdxcore.so
DEBU[0000] Locating "libdxcore.so" in [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2 /usr/lib/wsl/drivers/u0198975.inf_amd64_cab3843582148687 /usr/lib/wsl/drivers/kipudrv.inf_amd64_1a1aa059597c4810 /usr/lib/wsl/lib]
DEBU[0000] Checking candidate '/usr/lib/wsl/lib/libdxcore.so'
DEBU[0000] Found 1 candidates; ignoring further candidates
DEBU[0000] Located libdxcore.so as [/usr/lib/wsl/lib/libdxcore.so]
INFO[0000] Selecting /usr/lib/wsl/lib/libdxcore.so as /usr/lib/wsl/lib/libdxcore.so
DEBU[0000] Locating libnvdxgdmal.so.1
DEBU[0000] Locating "libnvdxgdmal.so.1" in [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2 /usr/lib/wsl/drivers/u0198975.inf_amd64_cab3843582148687 /usr/lib/wsl/drivers/kipudrv.inf_amd64_1a1aa059597c4810 /usr/lib/wsl/lib]
DEBU[0000] Checking candidate '/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvdxgdmal.so.1'
DEBU[0000] Found 1 candidates; ignoring further candidates
DEBU[0000] Located libnvdxgdmal.so.1 as [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvdxgdmal.so.1]
INFO[0000] Selecting /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvdxgdmal.so.1 as /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvdxgdmal.so.1
DEBU[0000] Locating nvcubins.bin
DEBU[0000] Locating "nvcubins.bin" in [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2 /usr/lib/wsl/drivers/u0198975.inf_amd64_cab3843582148687 /usr/lib/wsl/drivers/kipudrv.inf_amd64_1a1aa059597c4810 /usr/lib/wsl/lib]
DEBU[0000] Checking candidate '/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/nvcubins.bin'
DEBU[0000] Found 1 candidates; ignoring further candidates
DEBU[0000] Located nvcubins.bin as [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/nvcubins.bin]
INFO[0000] Selecting /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/nvcubins.bin as /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/nvcubins.bin
DEBU[0000] Locating nvidia-smi
DEBU[0000] Locating "nvidia-smi" in [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2 /usr/lib/wsl/drivers/u0198975.inf_amd64_cab3843582148687 /usr/lib/wsl/drivers/kipudrv.inf_amd64_1a1aa059597c4810 /usr/lib/wsl/lib]
DEBU[0000] Checking candidate '/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/nvidia-smi'
DEBU[0000] Found 1 candidates; ignoring further candidates
DEBU[0000] Located nvidia-smi as [/usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/nvidia-smi]
INFO[0000] Selecting /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/nvidia-smi as /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/nvidia-smi
---
cdiVersion: 0.3.0
kind: [nvidia.com/gpu](http://nvidia.com/gpu)
devices:
    - name: all
      containerEdits:
        deviceNodes:
            - path: /dev/dxg
              major: 10
              minor: 125
              fileMode: 438
              permissions: rwm
containerEdits:
    env:
        - NVIDIA_VISIBLE_DEVICES=void
    hooks:
        - hookName: createContainer
          path: /usr/bin/nvidia-cdi-hook
          args:
            - nvidia-cdi-hook
            - create-symlinks
            - --link
            - /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/nvidia-smi::/usr/bin/nvidia-smi
          env:
            - NVIDIA_CTK_DEBUG=false
        - hookName: createContainer
          path: /usr/bin/nvidia-cdi-hook
          args:
            - nvidia-cdi-hook
            - update-ldcache
            - --folder
            - /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2
            - --folder
            - /usr/lib/wsl/lib
          env:
            - NVIDIA_CTK_DEBUG=false
    mounts:
        - hostPath: /usr/lib/wsl/lib/libdxcore.so
          containerPath: /usr/lib/wsl/lib/libdxcore.so
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate
        - hostPath: /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libcuda.so.1.1
          containerPath: /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libcuda.so.1.1
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate
        - hostPath: /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libcuda_loader.so
          containerPath: /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libcuda_loader.so
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate
        - hostPath: /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvdxgdmal.so.1
          containerPath: /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvdxgdmal.so.1
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate
        - hostPath: /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ml.so.1
          containerPath: /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ml.so.1
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate
        - hostPath: /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ml_loader.so
          containerPath: /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ml_loader.so
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate
        - hostPath: /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ptxjitcompiler.so.1
          containerPath: /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/libnvidia-ptxjitcompiler.so.1
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate
        - hostPath: /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/nvcubins.bin
          containerPath: /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/nvcubins.bin
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate
        - hostPath: /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/nvidia-smi
          containerPath: /usr/lib/wsl/drivers/nvfrwi.inf_amd64_63d60f682af8cba2/nvidia-smi
          options:
            - ro
            - nosuid
            - nodev
            - rbind
            - rprivate
INFO[0000] Generated CDI spec with version 0.3.0
tyeth@framework16:~/dev-projects/misc/nvidia-ctk$

Here's the state of the required files (so it looks like dxcore was modified today by install instructions):

I've got another computer that I can get a fresh WSL on (with a Quadro card iirc) so will have a go over the weekend if I can.

[cdesiniotis]

From your latest logs, nvidia-ctk was able to discover libdxcore.so and add it to the CDI spec. It would be good to understand what differs in the NemoClaw / OpenShell stack (which is where I presume the original issue was reproduced in).

[tyeth]

I've sadly been messing today in docker to fix something else, and killed all old containers+volumes. I saw the nemoclaw one go, but it's bound to be recreatable (I must remember to check if they have disabled nvidia GPU detection in a recent update for wsl users to avoid the issue).

[tyeth]

Mental note, when looking back at this, check out this PR in latest openshell release 0.2.0:
feat(bootstrap,cli): switch GPU injection to CDI where supported by @elezar in #495
feat(sandbox): switch device plugin to CDI injection mode by @elezar in #503