Everything looks fine but fails simpleP2P test

[meunal]

NVIDIA Open GPU Kernel Modules Version

580.95.05

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Ubuntu 24.04.3 LTS

Kernel Release

6.14.0-33-generic 33~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 19 17:02:30 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

GPU 0: NVIDIA GeForce RTX 4090 (UUID: GPU-e7ef4627-74b0-908a-1b24-cdefb9ad4d11), GPU 1: NVIDIA GeForce RTX 4090 (UUID: GPU-1b36253b-41fa-d8cc-a12f-309a5e368f68)

Describe the bug

I successfully installed the modules, everything looks fine but simpleP2P test fails:

Run kernel on GPU1, taking source data from GPU0 and writing to GPU1...
Run kernel on GPU0, taking source data from GPU1 and writing to GPU0...
Copy data back to host from GPU0 and verify results...
Verification error @ element 0: val = nan, ref = 0.000000
Verification error @ element 1: val = nan, ref = 4.000000
Verification error @ element 2: val = nan, ref = 8.000000
Verification error @ element 3: val = nan, ref = 12.000000
Verification error @ element 4: val = nan, ref = 16.000000
Verification error @ element 5: val = nan, ref = 20.000000
Verification error @ element 6: val = nan, ref = 24.000000
Verification error @ element 7: val = nan, ref = 28.000000
Verification error @ element 8: val = nan, ref = 32.000000
Verification error @ element 9: val = nan, ref = 36.000000
Verification error @ element 10: val = nan, ref = 40.000000
Verification error @ element 11: val = nan, ref = 44.000000
Disabling peer access...
Shutting down...
Test failed!

torch.cuda.can_device_access_peer returns True for both (0, 1) and (1, 0). I can move data between GPU without any issue:

a = tensor.to("cuda:0")
b = tensor.to("cuda:0).to("cuda:1")

a == b

yields to True but still cannot pass simpleP2P.

Hardware:

• 14700K
• 128GB DDR5 at 4600MT/s
• ROG Maximus Z790 Dark Hero
• 2 x RTX 4090 FE (x8/x8)
• IOMMU, VT-d is off.

To Reproduce

Bug Incidence

Once

nvidia-bug-report.log.gz

More Info

No response