Does this kernel support 5060ti/5070ti/5080/5090 p2p each other?

[gengchaogit]

NVIDIA Open GPU Kernel Modules Version

580.95.05

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

ubuntu 22.04

Kernel Release

ubuntu 22.04

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

5*5060ti+5070ti+5090

Describe the bug

only 5060ti(8x) can p2p, others like pcie 16x->8x not able to p2p

To Reproduce

only 5060ti(8x) can p2p, others like pcie 16x->8x not able to p2p

Bug Incidence

Once

nvidia-bug-report.log.gz

only 5060ti(8x) can p2p, others like pcie 16x->8x not able to p2p

More Info

No response

[gengchaogit]

(base) root@epyc:~/cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest/build# ./p2pBandwidthLatencyTest
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 5090, pciBusID: e1, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 5070 Ti, pciBusID: a1, pciDeviceID: 0, pciDomainID:0
Device: 2, NVIDIA GeForce RTX 5060 Ti, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 3, NVIDIA GeForce RTX 5060 Ti, pciBusID: 41, pciDeviceID: 0, pciDomainID:0
Device: 4, NVIDIA GeForce RTX 5060 Ti, pciBusID: e2, pciDeviceID: 0, pciDomainID:0
Device=0 CANNOT Access Peer Device=1
Device=0 CANNOT Access Peer Device=2
Device=0 CANNOT Access Peer Device=3
Device=0 CANNOT Access Peer Device=4
Device=1 CANNOT Access Peer Device=0
Device=1 CANNOT Access Peer Device=2
Device=1 CANNOT Access Peer Device=3
Device=1 CANNOT Access Peer Device=4
Device=2 CANNOT Access Peer Device=0
Device=2 CANNOT Access Peer Device=1
Device=2 CAN Access Peer Device=3
Device=2 CAN Access Peer Device=4
Device=3 CANNOT Access Peer Device=0
Device=3 CANNOT Access Peer Device=1
Device=3 CAN Access Peer Device=2
Device=3 CAN Access Peer Device=4
Device=4 CANNOT Access Peer Device=0
Device=4 CANNOT Access Peer Device=1
Device=4 CAN Access Peer Device=2
Device=4 CAN Access Peer Device=3

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
D\D 0 1 2 3 4
0 1 0 0 0 0
1 0 1 0 0 0
2 0 0 1 1 1
3 0 0 1 1 1
4 0 0 1 1 1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1 2 3 4
0 1531.86 14.22 14.22 14.22 13.24
1 14.03 769.70 15.95 15.94 13.98
2 13.94 15.78 388.01 15.71 13.99
3 14.03 16.60 15.87 388.49 13.98
4 13.18 14.24 14.22 14.26 387.91
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
D\D 0 1 2 3 4
0 1525.88 14.23 14.24 14.21 13.39
1 14.04 766.31 15.94 15.70 13.96
2 14.03 16.07 388.58 28.63 14.32
3 14.02 16.68 28.63 387.62 14.31
4 13.19 14.26 14.33 14.33 388.39
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
D\D 0 1 2 3 4
0 1529.57 15.30 15.29 15.20 14.54
1 15.26 769.50 15.98 16.21 15.05
2 15.30 16.03 387.33 15.99 15.00
3 15.35 16.00 15.95 387.14 15.21
4 14.83 15.56 15.50 15.31 387.38
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
D\D 0 1 2 3 4
0 1525.09 15.31 15.30 15.08 14.66
1 15.28 769.88 15.98 16.22 15.05
2 15.28 16.03 386.94 56.32 28.31
3 15.44 16.00 56.30 386.94 28.31
4 14.80 15.59 28.31 28.31 387.13
P2P=Disabled Latency Matrix (us)
GPU 0 1 2 3 4
0 2.07 14.32 14.31 14.31 14.32
1 14.44 2.07 14.34 14.34 14.34
2 14.17 14.36 2.07 14.35 14.25
3 14.35 13.77 14.26 2.07 14.34
4 14.34 14.31 14.31 14.35 2.08

CPU 0 1 2 3 4
0 2.44 7.17 6.14 6.12 6.94
1 7.02 2.30 5.98 6.01 6.88
2 6.49 6.25 2.04 5.53 6.46
3 6.43 6.30 5.56 2.13 6.47
4 6.87 6.75 6.10 6.05 2.33
P2P=Enabled Latency (P2P Writes) Matrix (us)
GPU 0 1 2 3 4
0 2.07 14.32 14.31 14.34 14.33
1 14.34 2.07 14.33 14.25 14.32
2 14.32 14.25 2.07 0.39 0.39
3 14.10 14.35 0.45 2.07 0.45
4 14.31 14.34 0.46 0.47 2.08

CPU 0 1 2 3 4
0 2.38 6.93 6.07 6.07 6.90
1 7.11 2.28 5.98 6.01 6.86
2 6.44 6.24 2.06 1.55 1.57
3 6.48 6.32 1.62 2.13 1.62
4 6.98 6.82 1.89 1.90 2.36

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

[gengchaogit]

(base) root@epyc:~/cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest/build# nvidia-smi topo -m
GPU0 GPU1 GPU2 GPU3 GPU4 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NODE NODE NODE NODE 0-63 0 N/A
GPU1 NODE X NODE NODE NODE 0-63 0 N/A
GPU2 NODE NODE X NODE NODE 0-63 0 N/A
GPU3 NODE NODE NODE X PHB 0-63 0 N/A
GPU4 NODE NODE NODE PHB X 0-63 0 N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
(base) root@epyc:/cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest/build# nvidia-smi topo -p2p -p
Option "-p2p" is missing its value.
(base) root@epyc:/cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest/build# nvidia-smi topo -p2p p
GPU0 GPU1 GPU2 GPU3 GPU4
GPU0 X OK OK OK OK
GPU1 OK X OK OK OK
GPU2 OK OK X OK OK
GPU3 OK OK OK X OK
GPU4 OK OK OK OK X

Legend:

X = Self
OK = Status Ok
CNS = Chipset not supported
GNS = GPU not supported
TNS = Topology not supported
NS = Not supported
U = Unknown
(base) root@epyc:~/cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest/build#

[gengchaogit]

Let me explain my situation. I previously inserted three 5060 Ti cards into the PCIe slots, and connected a 5070 Ti and a 5090 using the two 8654 sockets on my MZ73 motherboard (the two cards in the PHB topology). The test results showed that the 5060 Ti cards could P2P with each other, but the other two cards couldn't P2P with any other card. Later, I swapped the positions of the 5060 Ti and 5070 Ti cards, and the test results remained the same: the 5060 Ti cards could still P2P with each other. Even during the topology transition from NODE to PHB, the 5060 Ti cards didn't experience any impact on enabling P2P.

During this period, the 5070ti was changed from gen4 8x to gen5 16x and gen5 8x, and the 5060ti was changed from gen5 8x to gen4 8x, but none of this changed the test results.

[gengchaogit]

Device 0 [NVIDIA GeForce RTX 5060 Ti] PCIe GEN 5@ 8x RX: 372.0 KiB/s TX: 460.0 KiB/s
GPU 2625MHz MEM 13801MH TEMP 32fC FAN 30% POW 20 / 180 W
GPU[||||||||| 24%] MEM[|| 0.887Gi/15.929Gi]

Device 1 [NVIDIA GeForce RTX 5060 Ti] PCIe GEN 5@ 8x RX: 24.06 MiB/s TX: 16.58 GiB/s
GPU 2565MHz MEM 13801MH TEMP 33fC FAN 30% POW 20 / 180 W
GPU[|||||||||||||| 38%] MEM[|| 0.887Gi/15.929Gi]

Device 2 [NVIDIA GeForce RTX 5070 Ti] PCIe GEN 5@ 8x RX: 383.0 KiB/s TX: 468.0 KiB/s
GPU 2827MHz MEM 13801MH TEMP 30fC FAN 0% POW 52 / 300 W
GPU[||||||||| 25%] MEM[|| 0.979Gi/15.921Gi]

Device 3 [NVIDIA GeForce RTX 5090] PCIe GEN 4@ 8x RX: 360.0 KiB/s TX: 963.0 KiB/s
GPU 2962MHz MEM 13801MH TEMP 41fC FAN 0% POW 75 / 600 W
GPU[|||||||||||||||||||||||||||||||||97%] MEM[| 1.283Gi/31.843Gi]

Device 4 [NVIDIA GeForce RTX 5060 Ti] PCIe GEN 4@ 8x RX: 8.812 MiB/s TX: 13.72 GiB/s
GPU 2565MHz MEM 13801MH TEMP 32fC FAN 30% POW 20 / 180 W
GPU[||||||||||||||| 41%] MEM[|| 0.887Gi/15.929Gi]

[gengchaogit]

I have trie below grub config, but all not work.

GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt numa=off pci=acs_override=downstream,multifunction"
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=off iommu=soft numa=off pci=realloc pci=acs_override=downstream,multifunction"

[gengchaogit]

(base) root@epyc:~/cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest/build# nvidia-smi -q |grep -A 3 "BAR1"
BAR1 Memory Usage
Total : 16384 MiB
Used : 1 MiB
Free : 16383 MiB

BAR1 Memory Usage
    Total                             : 16384 MiB
    Used                              : 1 MiB
    Free                              : 16383 MiB

--
BAR1 Memory Usage
Total : 16384 MiB
Used : 1 MiB
Free : 16383 MiB

BAR1 Memory Usage
    Total                             : 32768 MiB
    Used                              : 1 MiB
    Free                              : 32767 MiB

--
BAR1 Memory Usage
Total : 16384 MiB
Used : 1 MiB
Free : 16383 MiB
(base) root@epyc:~/cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest/build#