nvidia 5090 p2p bandwidth so small

[wolftankk]

NVIDIA Open GPU Kernel Modules Version

580.82.09

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Ubuntu 22.04.5 LTS

Kernel Release

5.15.0-136-generic

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

NVIDIA GeForce RTX 5090

Describe the bug

install driver, p2p has enabled, but test simpleP2P failed and bandwidth only 3.39GB/s.

./simpleP2P
[./simpleP2P] - Starting...
Checking for multiple GPUs...
CUDA-capable device count: 8

Checking GPU(s) for support of peer to peer memory access...
> Peer access from NVIDIA GeForce RTX 5090 (GPU0) -> NVIDIA GeForce RTX 5090 (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU0) -> NVIDIA GeForce RTX 5090 (GPU2) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU0) -> NVIDIA GeForce RTX 5090 (GPU3) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU0) -> NVIDIA GeForce RTX 5090 (GPU4) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU0) -> NVIDIA GeForce RTX 5090 (GPU5) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU0) -> NVIDIA GeForce RTX 5090 (GPU6) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU0) -> NVIDIA GeForce RTX 5090 (GPU7) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU1) -> NVIDIA GeForce RTX 5090 (GPU0) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU1) -> NVIDIA GeForce RTX 5090 (GPU2) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU1) -> NVIDIA GeForce RTX 5090 (GPU3) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU1) -> NVIDIA GeForce RTX 5090 (GPU4) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU1) -> NVIDIA GeForce RTX 5090 (GPU5) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU1) -> NVIDIA GeForce RTX 5090 (GPU6) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU1) -> NVIDIA GeForce RTX 5090 (GPU7) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU2) -> NVIDIA GeForce RTX 5090 (GPU0) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU2) -> NVIDIA GeForce RTX 5090 (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU2) -> NVIDIA GeForce RTX 5090 (GPU3) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU2) -> NVIDIA GeForce RTX 5090 (GPU4) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU2) -> NVIDIA GeForce RTX 5090 (GPU5) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU2) -> NVIDIA GeForce RTX 5090 (GPU6) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU2) -> NVIDIA GeForce RTX 5090 (GPU7) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU3) -> NVIDIA GeForce RTX 5090 (GPU0) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU3) -> NVIDIA GeForce RTX 5090 (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU3) -> NVIDIA GeForce RTX 5090 (GPU2) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU3) -> NVIDIA GeForce RTX 5090 (GPU4) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU3) -> NVIDIA GeForce RTX 5090 (GPU5) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU3) -> NVIDIA GeForce RTX 5090 (GPU6) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU3) -> NVIDIA GeForce RTX 5090 (GPU7) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU4) -> NVIDIA GeForce RTX 5090 (GPU0) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU4) -> NVIDIA GeForce RTX 5090 (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU4) -> NVIDIA GeForce RTX 5090 (GPU2) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU4) -> NVIDIA GeForce RTX 5090 (GPU3) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU4) -> NVIDIA GeForce RTX 5090 (GPU5) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU4) -> NVIDIA GeForce RTX 5090 (GPU6) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU4) -> NVIDIA GeForce RTX 5090 (GPU7) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU5) -> NVIDIA GeForce RTX 5090 (GPU0) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU5) -> NVIDIA GeForce RTX 5090 (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU5) -> NVIDIA GeForce RTX 5090 (GPU2) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU5) -> NVIDIA GeForce RTX 5090 (GPU3) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU5) -> NVIDIA GeForce RTX 5090 (GPU4) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU5) -> NVIDIA GeForce RTX 5090 (GPU6) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU5) -> NVIDIA GeForce RTX 5090 (GPU7) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU6) -> NVIDIA GeForce RTX 5090 (GPU0) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU6) -> NVIDIA GeForce RTX 5090 (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU6) -> NVIDIA GeForce RTX 5090 (GPU2) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU6) -> NVIDIA GeForce RTX 5090 (GPU3) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU6) -> NVIDIA GeForce RTX 5090 (GPU4) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU6) -> NVIDIA GeForce RTX 5090 (GPU5) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU6) -> NVIDIA GeForce RTX 5090 (GPU7) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU7) -> NVIDIA GeForce RTX 5090 (GPU0) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU7) -> NVIDIA GeForce RTX 5090 (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU7) -> NVIDIA GeForce RTX 5090 (GPU2) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU7) -> NVIDIA GeForce RTX 5090 (GPU3) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU7) -> NVIDIA GeForce RTX 5090 (GPU4) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU7) -> NVIDIA GeForce RTX 5090 (GPU5) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU7) -> NVIDIA GeForce RTX 5090 (GPU6) : Yes
Enabling peer access between GPU0 and GPU1...
Allocating buffers (64MB on GPU0, GPU1 and CPU Host)...
Creating event handles...
cudaMemcpyPeer / cudaMemcpy between GPU0 and GPU1: 3.39GB/s
Preparing host buffer and memcpy to GPU0...
Run kernel on GPU1, taking source data from GPU0 and writing to GPU1...
Run kernel on GPU0, taking source data from GPU1 and writing to GPU0...
Copy data back to host from GPU0 and verify results...
Verification error @ element 0: val = nan, ref = 0.000000
Verification error @ element 1: val = nan, ref = 4.000000
Verification error @ element 2: val = nan, ref = 8.000000
Verification error @ element 3: val = nan, ref = 12.000000
Verification error @ element 4: val = nan, ref = 16.000000
Verification error @ element 5: val = nan, ref = 20.000000
Verification error @ element 6: val = nan, ref = 24.000000
Verification error @ element 7: val = nan, ref = 28.000000
Verification error @ element 8: val = nan, ref = 32.000000
Verification error @ element 9: val = nan, ref = 36.000000
Verification error @ element 10: val = nan, ref = 40.000000
Verification error @ element 11: val = nan, ref = 44.000000
Disabling peer access...
Shutting down...
Test failed!

p2pBandwidthLatencyTest

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 5090, pciBusID: 16, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 5090, pciBusID: 27, pciDeviceID: 0, pciDomainID:0
Device: 2, NVIDIA GeForce RTX 5090, pciBusID: 38, pciDeviceID: 0, pciDomainID:0
Device: 3, NVIDIA GeForce RTX 5090, pciBusID: 5a, pciDeviceID: 0, pciDomainID:0
Device: 4, NVIDIA GeForce RTX 5090, pciBusID: 98, pciDeviceID: 0, pciDomainID:0
Device: 5, NVIDIA GeForce RTX 5090, pciBusID: a8, pciDeviceID: 0, pciDomainID:0
Device: 6, NVIDIA GeForce RTX 5090, pciBusID: b8, pciDeviceID: 0, pciDomainID:0
Device: 7, NVIDIA GeForce RTX 5090, pciBusID: d8, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CAN Access Peer Device=2
Device=0 CAN Access Peer Device=3
Device=0 CAN Access Peer Device=4
Device=0 CAN Access Peer Device=5
Device=0 CAN Access Peer Device=6
Device=0 CAN Access Peer Device=7
Device=1 CAN Access Peer Device=0
Device=1 CAN Access Peer Device=2
Device=1 CAN Access Peer Device=3
Device=1 CAN Access Peer Device=4
Device=1 CAN Access Peer Device=5
Device=1 CAN Access Peer Device=6
Device=1 CAN Access Peer Device=7
Device=2 CAN Access Peer Device=0
Device=2 CAN Access Peer Device=1
Device=2 CAN Access Peer Device=3
Device=2 CAN Access Peer Device=4
Device=2 CAN Access Peer Device=5
Device=2 CAN Access Peer Device=6
Device=2 CAN Access Peer Device=7
Device=3 CAN Access Peer Device=0
Device=3 CAN Access Peer Device=1
Device=3 CAN Access Peer Device=2
Device=3 CAN Access Peer Device=4
Device=3 CAN Access Peer Device=5
Device=3 CAN Access Peer Device=6
Device=3 CAN Access Peer Device=7
Device=4 CAN Access Peer Device=0
Device=4 CAN Access Peer Device=1
Device=4 CAN Access Peer Device=2
Device=4 CAN Access Peer Device=3
Device=4 CAN Access Peer Device=5
Device=4 CAN Access Peer Device=6
Device=4 CAN Access Peer Device=7
Device=5 CAN Access Peer Device=0
Device=5 CAN Access Peer Device=1
Device=5 CAN Access Peer Device=2
Device=5 CAN Access Peer Device=3
Device=5 CAN Access Peer Device=4
Device=5 CAN Access Peer Device=6
Device=5 CAN Access Peer Device=7
Device=6 CAN Access Peer Device=0
Device=6 CAN Access Peer Device=1
Device=6 CAN Access Peer Device=2
Device=6 CAN Access Peer Device=3
Device=6 CAN Access Peer Device=4
Device=6 CAN Access Peer Device=5
Device=6 CAN Access Peer Device=7
Device=7 CAN Access Peer Device=0
Device=7 CAN Access Peer Device=1
Device=7 CAN Access Peer Device=2
Device=7 CAN Access Peer Device=3
Device=7 CAN Access Peer Device=4
Device=7 CAN Access Peer Device=5
Device=7 CAN Access Peer Device=6

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2     3     4     5     6     7
     0	     1     1     1     1     1     1     1     1
     1	     1     1     1     1     1     1     1     1
     2	     1     1     1     1     1     1     1     1
     3	     1     1     1     1     1     1     1     1
     4	     1     1     1     1     1     1     1     1
     5	     1     1     1     1     1     1     1     1
     6	     1     1     1     1     1     1     1     1
     7	     1     1     1     1     1     1     1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3      4      5      6      7
     0 1534.87  43.67  44.03  43.78  42.46  42.50  42.42  42.56
     1  43.25 1553.18  43.64  43.39  42.36  42.52  42.45  42.38
     2  43.65  43.34 1559.48  43.43  42.37  42.39  42.33  42.39
     3  43.95  43.68  43.21 1547.13  42.40  42.43  42.37  42.42
     4  42.50  42.66  42.30  42.57 1559.38  42.40  42.40  42.54
     5  42.43  42.55  42.39  42.56  42.47 1556.27  42.51  42.44
     6  42.27  42.53  42.32  42.60  42.45  42.45 1559.43  42.43
     7  42.54  42.62  42.33  42.64  42.55  42.44  42.67 1556.27
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2      3      4      5      6      7
     0 1522.90   2.00   1.99   2.00   1.99   1.99   1.99   1.99
     1  21.07 1537.99   1.99   2.00   1.99   1.99   1.99   1.99
     2   1.99  21.07 1543.97   1.99   1.99   1.99   1.99   1.99
     3   1.99   1.99   2.00 1543.97   1.99   1.99   1.99   1.99
     4   1.99   1.99   1.99   1.99 1540.93   2.00   1.99   2.00
     5   1.99   1.99   1.99   1.99  21.07 1540.93   1.99   2.00
     6   1.99   1.99   1.99   1.99   2.00  21.07 1540.93   2.00
     7   1.99   1.99   1.99   1.99   1.99   1.99   2.00 1556.27
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3      4      5      6      7
     0 1527.30  57.34  57.22  57.23  57.08  57.18  56.71  57.11
     1  57.50 1538.60  57.73  57.41  57.14  57.04  56.73  57.47
     2  57.36  57.22 1540.88  57.41  57.88  57.25  57.23  56.87
     3  57.42  57.24  57.91 1540.10  57.16  56.67  56.76  56.96
     4  56.87  57.13  56.84  56.64 1540.12  52.52  53.22  53.40
     5  56.45  56.41  56.40  57.51  52.90 1540.12  53.20  53.34
     6  56.44  56.51  57.17  56.93  53.30  53.24 1541.64  53.35
     7  56.85  57.70  56.77  56.86  53.37  53.36  53.30 1540.12
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3      4      5      6      7
     0 1527.30   3.99   3.99   3.99   3.99   3.99   3.99   3.99
     1   3.99 1539.36   3.99   3.99   3.99   3.99   3.99   3.99
     2   3.99   3.99 1541.62   3.99   3.99   3.99   3.99   3.99
     3   3.99   3.99   3.99 1540.86   3.99   3.99   3.99   3.99
     4   3.99   3.99   3.99   3.99 1540.88   3.99   3.99   3.99
     5   3.99   3.99   3.99   3.99   3.99 1540.10   3.99   3.99
     6   3.99   3.99   3.99   3.99   3.99   3.99 1540.12   3.99
     7   3.99   3.99   3.99   3.99   3.99   3.99   3.99 1540.88
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2      3      4      5      6      7
     0   2.07  14.31  14.30  14.29  14.32  14.32  14.36  14.31
     1  14.31   2.07  14.37  14.17  14.31  14.33  14.33  14.38
     2  13.19  13.02   2.07  14.32  14.34  14.34  14.31  14.40
     3  14.32  14.27  14.15   2.07  14.31  14.41  14.31  14.31
     4  14.33  14.34  14.37  14.37   2.07  14.33  14.12  14.25
     5  14.32  14.36  14.31  14.38  14.23   2.07  14.33  14.31
     6  14.34  14.51  14.34  14.31  14.32  14.33   2.07  14.33
     7  14.32  14.36  14.36  14.33  14.32  14.32  14.31   2.07

   CPU     0      1      2      3      4      5      6      7
     0   3.20   8.82   8.52   8.50   8.96   9.15   9.07   9.27
     1   8.98   3.01   8.63   8.74   9.25   9.36   9.26   9.41
     2   9.13   9.07   3.05   8.86   9.31   9.51   9.66   9.84
     3   9.43   9.15   9.27   3.16   9.90   9.96   9.93  10.03
     4   9.93  10.00   9.95  10.10   3.53  10.61  10.65  10.91
     5  10.50  10.40  10.41  10.48  11.26   3.68  11.32  11.31
     6  10.82  10.51  10.78  10.95  11.46  11.73   3.74  12.02
     7  11.84  11.54  11.41  11.47  12.05  12.33  12.23   3.90
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2      3      4      5      6      7
     0   2.07   0.44   0.37   0.37   0.38   0.38   0.38   0.44
     1   0.43   2.07   0.37   0.43   0.36   0.44   0.37   0.36
     2   0.38   0.43   2.07   0.36   0.37   0.37   0.37   0.44
     3   0.38   0.36   0.37   2.07   0.37   0.38   0.44   0.37
     4   0.40   0.40   0.46   0.42   2.07   0.47   0.45   0.40
     5   0.41   0.40   0.40   0.41   0.44   2.07   0.39   0.46
     6   0.47   0.40   0.39   0.40   0.46   0.36   2.07   0.44
     7   0.46   0.40   0.41   0.47   0.38   0.40   0.46   2.07

   CPU     0      1      2      3      4      5      6      7
     0   3.68   2.83   2.75   2.72   2.81   2.71   2.70   2.81
     1   2.85   3.60   2.75   2.74   2.73   2.74   2.73   2.72
     2   2.89   2.74   3.59   2.84   2.76   2.75   2.75   2.71
     3   2.92   2.75   2.84   3.72   2.77   2.73   2.84   2.76
     4   3.23   3.02   3.03   3.03   3.87   2.99   3.04   3.00
     5   3.17   3.14   3.05   3.03   3.05   4.03   3.02   3.02
     6   2.90   2.79   2.79   2.79   2.89   2.81   3.94   2.76
     7   3.19   3.10   3.13   3.07   3.07   3.07   3.05   3.91

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

in dmesg err:

[  291.329035] NVRM: nvAssertFailedNoLog: Assertion failed: ((base & RM_PAGE_MASK) == 0) @ kern_bus.c:295
[  291.329054] NVRM: nvAssertFailedNoLog: Assertion failed: remoteWMBoxLocalAddr != ~0ULL @ kern_bus_gm200.c:89

To Reproduce

in ubuntu, download nvidia driver from https://www.nvidia.com/en-us/drivers/details/254126/

install nvidia driver

exec ./install

reboot

Bug Incidence

Always

nvidia-bug-report.log.gz

nvidia-bug-report.log.gz

More Info

No response

[magikRUKKOLA]

Its seems line a memory address align issue. The likely cause is that you are using one of the GPUs for the video output while loading the OS so the memory allocator can't align the pages. Try adding nomodeset parameter to the kernel arguments (make sure you have an ssh access to the machine etc.).

You might also consider adding mitigations=off to boost the CPU performance slightly.

[3ricj]

Ok,

I had a similar problem and I was able to finally get to the bottom of it.

1. Make sure ACS is disabled -- I had it disabled in my bios, but it was still showing in sudo lspci -vvv as enabled (look for DirectTrans+ under ACSCtl under the host controller. If it shows "-" - your bios is lying to you, or it's on a bridge which can't be managed by the bios in my case -- I had a PLX bridge). I used the script below to force it to be disabled.

# must be root to access extended PCI config space
if [ "$EUID" -ne 0 ]; then
  echo "ERROR: $0 must be run as root"
  exit 1
fi

for BDF in `lspci -d "*:*:*" | awk '{print $1}'`; do

    # skip if it doesn't support ACS
    setpci -v -s ${BDF} ECAP_ACS+0x6.w > /dev/null 2>&1
    if [ $? -ne 0 ]; then
            #echo "${BDF} does not support ACS, skipping"
            continue
    fi

    logger "Disabling ACS on `lspci -s ${BDF}`"
    setpci -v -s ${BDF} ECAP_ACS+0x6.w=0000
    if [ $? -ne 0 ]; then
        logger "Error disabling ACS on ${BDF}"
            continue
    fi
    NEW_VAL=`setpci -v -s ${BDF} ECAP_ACS+0x6.w | awk '{print $NF}'`
    if [ "${NEW_VAL}" != "0000" ]; then
        logger "Failed to disable ACS on ${BDF}"
            continue
    fi
done
exit 0

2. IOMMU must be disabled -- or at least, all of your devices must be in the same IOMMU group. This you can do via kernel parameter -- in my case, when I disabled ACS in my bios, it put all my devices in one IOMMU group -- so it worked -- but I had slightly better performance when I disabled IOMMU directly.

3. PCIE bridge configuration -- You clearly have some monster machine with that many gpus - but they may all be sharing some north bridge PCIE switch or something -- so traffic between GPUS on a common switch may be faster than when talking to the devices hanging directly off the CPU, etc. The nvidia tools are helpful for getting a sense of this:

        GPU0    GPU1    GPU2    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      PIX     PXB     0-15    0               N/A
GPU1    PIX      X      PXB     0-15    0               N/A
GPU2    PXB     PXB      X      0-15    0               N/A

Legend:
  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge

After finally getting ACS disabled for real -- this driver performed well -- meaning, with p2p enabled it showed increased performance, not decreased performance as your example provided. I'm currently stuck at 8x -- and have a mixture of 4090 and 5090 gpus, so they can't all p2p, but here's the result:

After disabling ACS:

Device: 0, NVIDIA GeForce RTX 4090, pciBusID: 5, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 4090, pciBusID: 6, pciDeviceID: 0, pciDomainID:0
Device: 2, NVIDIA GeForce RTX 5090, pciBusID: 9, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CANNOT Access Peer Device=2
Device=1 CAN Access Peer Device=0
Device=1 CANNOT Access Peer Device=2
Device=2 CANNOT Access Peer Device=0
Device=2 CANNOT Access Peer Device=1

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2
     0       1     1     0
     1       1     1     0
     2       0     0     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2
     0 915.89   9.59  10.01
     1   9.60 927.30  10.04
     2  10.05  10.04 1548.56
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2
     0 919.12  13.17   9.93
     1  13.17 944.68   9.88
     2  10.04  10.04 1539.46
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2
     0 920.74  10.47  10.11
     1  10.51 925.93  10.14
     2  10.16  10.16 1541.64
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2
     0 919.39  25.94  10.07
     1  25.94 921.83  10.12
     2  10.15  10.13 1541.64
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2
     0   1.36  18.72  19.63
     1  22.63   1.34  19.96
     2  18.74  18.74   2.10

   CPU     0      1      2
     0   1.56   5.87   5.68
     1   5.63   1.49   5.77
     2   5.80   5.78   1.62
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2
     0   1.44   1.68  19.76
     1   1.64   1.34  19.74
     2  18.73  18.80   2.08

   CPU     0      1      2
     0   1.53   1.33   5.80
     1   1.37   1.51   5.83
     2   5.79   5.86   1.63

Update: I finally got my cabling sorted so I could have full 16x speeds (pcie 4.0), and here are the new results. As you would expect, the bandwidth doubles. 52GB/Second {bidirectional} makes me happy -- the max without overhead is 63GB/second. When I first installed the diver in this repo enabling it would give me 0.62GB/second (8x), but with p2p disabled I would see ~7. ACS was really killing p2p. A vast improvement.

Device: 0, NVIDIA GeForce RTX 4090, pciBusID: 5, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 4090, pciBusID: d, pciDeviceID: 0, pciDomainID:0
Device: 2, NVIDIA GeForce RTX 5090, pciBusID: 6, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CANNOT Access Peer Device=2
Device=1 CAN Access Peer Device=0
Device=1 CANNOT Access Peer Device=2
Device=2 CANNOT Access Peer Device=0
Device=2 CANNOT Access Peer Device=1

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2
     0       1     1     0
     1       1     1     0
     2       0     0     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2
     0 913.74   8.05   9.98
     1   9.36 925.65  10.00
     2   9.64   8.40 1554.73
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2
     0 919.12  26.29   9.95
     1  26.29 944.68   9.98
     2   9.64   8.41 1539.46
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2
     0 920.74   8.87   9.71
     1   8.93 925.65   9.40
     2   9.79   9.38 1540.12
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2
     0 920.20  52.04   9.73
     1  52.04 920.47   9.38
     2   9.79   9.40 1539.36
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2
     0   1.41  19.23  19.31
     1  22.08   1.41  19.18
     2  18.91  18.89   2.10

   CPU     0      1      2
     0   1.60   5.75   5.72
     1   5.70   1.55   6.04
     2   5.77   5.94   1.71
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2
     0   1.38   1.61  18.94
     1   1.61   1.38  19.33
     2  18.67  19.03   2.10

   CPU     0      1      2
     0   1.53   1.33   5.78
     1   1.39   1.54   5.78
     2   5.85   5.78   1.61

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

[aikitoria]

Interesting, maybe we should add disabling ACS to the instructions then. It should be enough to set IOMMU in passthrough mode, you do not need to disable it (this is actually impossible on systems with more than 255 logical cores)

[wolftankk]

Hi all,
I tested some cases, last case is working.

@3ricj tips it's right.

in bios

1. IOMMU must be disabled
2. ACS must be disabled
3. resizing BAR must be enabled.

# ./p2pBandwidthLatencyTest
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 5090, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 5090, pciBusID: 25, pciDeviceID: 0, pciDomainID:0
Device: 2, NVIDIA GeForce RTX 5090, pciBusID: 41, pciDeviceID: 0, pciDomainID:0
Device: 3, NVIDIA GeForce RTX 5090, pciBusID: 61, pciDeviceID: 0, pciDomainID:0
Device: 4, NVIDIA GeForce RTX 5090, pciBusID: 81, pciDeviceID: 0, pciDomainID:0
Device: 5, NVIDIA GeForce RTX 5090, pciBusID: a1, pciDeviceID: 0, pciDomainID:0
Device: 6, NVIDIA GeForce RTX 5090, pciBusID: c1, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CAN Access Peer Device=2
Device=0 CAN Access Peer Device=3
Device=0 CAN Access Peer Device=4
Device=0 CAN Access Peer Device=5
Device=0 CAN Access Peer Device=6
Device=1 CAN Access Peer Device=0
Device=1 CAN Access Peer Device=2
Device=1 CAN Access Peer Device=3
Device=1 CAN Access Peer Device=4
Device=1 CAN Access Peer Device=5
Device=1 CAN Access Peer Device=6
Device=2 CAN Access Peer Device=0
Device=2 CAN Access Peer Device=1
Device=2 CAN Access Peer Device=3
Device=2 CAN Access Peer Device=4
Device=2 CAN Access Peer Device=5
Device=2 CAN Access Peer Device=6
Device=3 CAN Access Peer Device=0
Device=3 CAN Access Peer Device=1
Device=3 CAN Access Peer Device=2
Device=3 CAN Access Peer Device=4
Device=3 CAN Access Peer Device=5
Device=3 CAN Access Peer Device=6
Device=4 CAN Access Peer Device=0
Device=4 CAN Access Peer Device=1
Device=4 CAN Access Peer Device=2
Device=4 CAN Access Peer Device=3
Device=4 CAN Access Peer Device=5
Device=4 CAN Access Peer Device=6
Device=5 CAN Access Peer Device=0
Device=5 CAN Access Peer Device=1
Device=5 CAN Access Peer Device=2
Device=5 CAN Access Peer Device=3
Device=5 CAN Access Peer Device=4
Device=5 CAN Access Peer Device=6
Device=6 CAN Access Peer Device=0
Device=6 CAN Access Peer Device=1
Device=6 CAN Access Peer Device=2
Device=6 CAN Access Peer Device=3
Device=6 CAN Access Peer Device=4
Device=6 CAN Access Peer Device=5

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2     3     4     5     6
     0	     1     1     1     1     1     1     1
     1	     1     1     1     1     1     1     1
     2	     1     1     1     1     1     1     1
     3	     1     1     1     1     1     1     1
     4	     1     1     1     1     1     1     1
     5	     1     1     1     1     1     1     1
     6	     1     1     1     1     1     1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3      4      5      6
     0 1525.88  19.77  19.91  19.28  16.58  18.52  18.45
     1  19.38 1553.23  19.95  19.44  16.18  18.11  17.91
     2  19.30  19.75 1556.32  19.47  16.53  18.50  18.45
     3  19.29  19.77  19.94 1553.28  16.54  18.46  18.01
     4  19.19  19.57  20.01  18.48 1556.27  18.43  18.59
     5  19.35  19.56  19.97  19.47  16.91 1559.38  18.23
     6  19.32  19.49  19.96  19.35  18.25  19.03 1556.27
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2      3      4      5      6
     0 1519.94  28.08  28.23  28.20  19.64  22.73  21.88
     1  28.08 1547.03  28.23  28.21  20.08  22.73  22.69
     2  28.08  28.23 1540.97  28.21  20.77  21.92  22.68
     3  28.05  28.23  28.23 1540.93  20.59  22.73  21.92
     4  22.73  21.87  22.73  22.73 1543.97  28.06  28.23
     5  10.09  10.22  15.65  22.73  28.19 1540.93  28.07
     6   9.20  15.63  22.72  21.87  27.93  28.23 1540.88
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3      4      5      6
     0 1528.05  20.12  20.10  20.07  17.74  19.61  19.55
     1  20.13 1541.64  20.24  20.14  18.39  19.01  19.32
     2  20.10  20.22 1540.88  20.19  18.41  18.99  19.71
     3  20.00  20.06  20.16 1538.58  18.45  19.06  19.35
     4  19.29  18.43  19.26  18.48 1539.36  18.85  19.06
     5  19.55  19.30  18.67  19.62  18.57 1540.10  19.17
     6  19.51  19.09  19.67  19.15  19.02  19.09 1540.12
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3      4      5      6
     0 1525.83  55.79  55.79  55.46  36.48  42.55  41.17
     1  55.50 1537.85  55.88  55.87  37.95  42.61  42.54
     2  55.50  55.86 1537.09  55.87  38.91  41.21  42.53
     3  55.51  55.86  55.86 1536.31  39.39  42.60  41.23
     4  42.60  41.11  42.54  42.52 1540.88  55.56  55.86
     5  39.25  42.55  41.25  42.61  55.50 1538.58  55.78
     6  37.83  42.61  42.52  41.24  55.50  55.84 1537.85
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2      3      4      5      6
     0   2.07  14.31  14.35  14.35  14.32  14.32  14.31
     1  14.34   2.07  14.32  14.32  14.34  14.29  14.35
     2  14.31  14.32   2.07  14.32  15.13  15.17  15.11
     3  14.31  14.34  14.32   2.07  14.88  15.14  15.19
     4  14.43  14.44  14.43  14.45   2.14  15.78  15.61
     5  14.40  14.46  14.39  14.40  15.62   2.13  15.76
     6  14.44  14.41  14.47  14.42  15.12  15.14   2.11

   CPU     0      1      2      3      4      5      6
     0   2.52   7.16   7.02   6.91   7.93   7.85   7.88
     1   7.08   2.40   6.89   6.79   7.91   7.71   7.83
     2   7.01   6.76   2.41   6.96  10.24   8.91   9.01
     3   7.33   7.10   7.24   2.48   9.07   8.93   8.94
     4   8.28   8.20   8.36   8.36   3.13  10.12  10.17
     5   8.18   8.06   8.19   8.16  10.06   3.07  10.06
     6   8.19   8.18   8.28   8.23  10.12   9.97   3.05
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2      3      4      5      6
     0   2.07   0.44   0.36   0.37   0.36   0.38   0.36
     1   0.36   2.07   0.37   0.44   0.44   0.37   0.43
     2   0.43   0.37   2.07   0.37   0.36   0.43   0.36
     3   0.44   0.38   0.44   2.07   0.39   0.37   0.38
     4   0.40   0.41   0.46   0.46   2.09   0.40   0.39
     5   0.39   0.47   0.46   0.46   0.39   2.10   0.47
     6   0.48   0.40   0.46   0.41   0.46   0.47   2.10

   CPU     0      1      2      3      4      5      6
     0   2.52   1.93   1.92   1.92   1.92   1.92   1.92
     1   2.04   2.51   1.93   1.93   1.95   1.95   1.94
     2   2.04   2.01   2.63   2.04   2.02   1.99   1.98
     3   2.05   2.02   1.97   2.57   2.01   1.99   2.06
     4   2.73   2.60   2.61   2.61   3.18   2.65   2.67
     5   2.68   2.59   2.59   2.61   2.54   3.14   2.55
     6   2.70   2.63   2.62   2.65   2.63   2.62   3.15

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

[NASLXTO]

Ok,

I had a similar problem and I was able to finally get to the bottom of it.

1. Make sure ACS is disabled -- I had it disabled in my bios, but it was still showing in sudo lspci -vvv as enabled (look for DirectTrans+ under ACSCtl under the host controller. If it shows "-" - your bios is lying to you, or it's on a bridge which can't be managed by the bios in my case -- I had a PLX bridge). I used the script below to force it to be disabled.

# must be root to access extended PCI config space
if [ "$EUID" -ne 0 ]; then
  echo "ERROR: $0 must be run as root"
  exit 1
fi

for BDF in `lspci -d "*:*:*" | awk '{print $1}'`; do

    # skip if it doesn't support ACS
    setpci -v -s ${BDF} ECAP_ACS+0x6.w > /dev/null 2>&1
    if [ $? -ne 0 ]; then
            #echo "${BDF} does not support ACS, skipping"
            continue
    fi

    logger "Disabling ACS on `lspci -s ${BDF}`"
    setpci -v -s ${BDF} ECAP_ACS+0x6.w=0000
    if [ $? -ne 0 ]; then
        logger "Error disabling ACS on ${BDF}"
            continue
    fi
    NEW_VAL=`setpci -v -s ${BDF} ECAP_ACS+0x6.w | awk '{print $NF}'`
    if [ "${NEW_VAL}" != "0000" ]; then
        logger "Failed to disable ACS on ${BDF}"
            continue
    fi
done
exit 0

2. IOMMU must be disabled -- or at least, all of your devices must be in the same IOMMU group. This you can do via kernel parameter -- in my case, when I disabled ACS in my bios, it put all my devices in one IOMMU group -- so it worked -- but I had slightly better performance when I disabled IOMMU directly.
3. PCIE bridge configuration -- You clearly have some monster machine with that many gpus - but they may all be sharing some north bridge PCIE switch or something -- so traffic between GPUS on a common switch may be faster than when talking to the devices hanging directly off the CPU, etc. The nvidia tools are helpful for getting a sense of this:

        GPU0    GPU1    GPU2    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      PIX     PXB     0-15    0               N/A
GPU1    PIX      X      PXB     0-15    0               N/A
GPU2    PXB     PXB      X      0-15    0               N/A

Legend:
  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge

After finally getting ACS disabled for real -- this driver performed well -- meaning, with p2p enabled it showed increased performance, not decreased performance as your example provided. I'm currently stuck at 8x -- and have a mixture of 4090 and 5090 gpus, so they can't all p2p, but here's the result:

After disabling ACS:

Device: 0, NVIDIA GeForce RTX 4090, pciBusID: 5, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 4090, pciBusID: 6, pciDeviceID: 0, pciDomainID:0
Device: 2, NVIDIA GeForce RTX 5090, pciBusID: 9, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CANNOT Access Peer Device=2
Device=1 CAN Access Peer Device=0
Device=1 CANNOT Access Peer Device=2
Device=2 CANNOT Access Peer Device=0
Device=2 CANNOT Access Peer Device=1

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2
     0       1     1     0
     1       1     1     0
     2       0     0     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2
     0 915.89   9.59  10.01
     1   9.60 927.30  10.04
     2  10.05  10.04 1548.56
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2
     0 919.12  13.17   9.93
     1  13.17 944.68   9.88
     2  10.04  10.04 1539.46
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2
     0 920.74  10.47  10.11
     1  10.51 925.93  10.14
     2  10.16  10.16 1541.64
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2
     0 919.39  25.94  10.07
     1  25.94 921.83  10.12
     2  10.15  10.13 1541.64
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2
     0   1.36  18.72  19.63
     1  22.63   1.34  19.96
     2  18.74  18.74   2.10

   CPU     0      1      2
     0   1.56   5.87   5.68
     1   5.63   1.49   5.77
     2   5.80   5.78   1.62
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2
     0   1.44   1.68  19.76
     1   1.64   1.34  19.74
     2  18.73  18.80   2.08

   CPU     0      1      2
     0   1.53   1.33   5.80
     1   1.37   1.51   5.83
     2   5.79   5.86   1.63

Update: I finally got my cabling sorted so I could have full 16x speeds (pcie 4.0), and here are the new results. As you would expect, the bandwidth doubles. 52GB/Second {bidirectional} makes me happy -- the max without overhead is 63GB/second. When I first installed the diver in this repo enabling it would give me 0.62GB/second (8x), but with p2p disabled I would see ~7. ACS was really killing p2p. A vast improvement.

Device: 0, NVIDIA GeForce RTX 4090, pciBusID: 5, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 4090, pciBusID: d, pciDeviceID: 0, pciDomainID:0
Device: 2, NVIDIA GeForce RTX 5090, pciBusID: 6, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CANNOT Access Peer Device=2
Device=1 CAN Access Peer Device=0
Device=1 CANNOT Access Peer Device=2
Device=2 CANNOT Access Peer Device=0
Device=2 CANNOT Access Peer Device=1

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2
     0       1     1     0
     1       1     1     0
     2       0     0     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2
     0 913.74   8.05   9.98
     1   9.36 925.65  10.00
     2   9.64   8.40 1554.73
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2
     0 919.12  26.29   9.95
     1  26.29 944.68   9.98
     2   9.64   8.41 1539.46
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2
     0 920.74   8.87   9.71
     1   8.93 925.65   9.40
     2   9.79   9.38 1540.12
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2
     0 920.20  52.04   9.73
     1  52.04 920.47   9.38
     2   9.79   9.40 1539.36
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2
     0   1.41  19.23  19.31
     1  22.08   1.41  19.18
     2  18.91  18.89   2.10

   CPU     0      1      2
     0   1.60   5.75   5.72
     1   5.70   1.55   6.04
     2   5.77   5.94   1.71
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2
     0   1.38   1.61  18.94
     1   1.61   1.38  19.33
     2  18.67  19.03   2.10

   CPU     0      1      2
     0   1.53   1.33   5.78
     1   1.39   1.54   5.78
     2   5.85   5.78   1.61

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

Hi @3ricj
supermicor told me: The 4125gs-tnrt2 has two BRCM PLX switches. Resize BAR does not work with PCIE slots from the PLX switch.

Is your Resizable BAR setting enabled?

[3ricj]

Hi @3ricj supermicor told me: The 4125gs-tnrt2 has two BRCM PLX switches. Resize BAR does not work with PCIE slots from the PLX switch.

Is your Resizable BAR setting enabled?

In short, yes, I have resizable bar enabled.

I don't have one of those boards -- but oddly, when I look at the manual it shows each of the two cpus are directly connected to 5x pcie lanes -- but it doesn't show the details:

-- it doesn't show a PLX bridge -- but it likely has one root bridge per cpu? But it's a guess. I'd look at the board in detail, or the output of lspci -vvvv as root. That will also show if Resizable Bar is currently enabled -- which is what you are asking about here.
-- it also doesn't show an ISL -- inter switch link - which would enable the two pci roots to directly talk to each other.

Most likely, this system would need to communicate via the CPUs -- but there is considerable bandwidth at the fabric layer here. I'd be curious about what the output of nvidia-smi topo- m shows.

One additional detail: PLX switches can't have "Large Memory Forwarding" turned on for dual CPU systems, as it requires them to be in "Transparent" mode not NTB/dual-host mode -- which disables large memory forwarding. That being said, you can likely get p2p access working on each group of GPUs tied to each respective root complex/cpu.

[NASLXTO]

Hi @3ricj supermicor told me: The 4125gs-tnrt2 has two BRCM PLX switches. Resize BAR does not work with PCIE slots from the PLX switch.
Is your Resizable BAR setting enabled?

In short, yes, I have resizable bar enabled.

I don't have one of those boards -- but oddly, when I look at the manual it shows each of the two cpus are directly connected to 5x pcie lanes -- but it doesn't show the details:

-- it doesn't show a PLX bridge -- but it likely has one root bridge per cpu? But it's a guess. I'd look at the board in detail, or the output of lspci -vvvv as root. That will also show if Resizable Bar is currently enabled -- which is what you are asking about here. -- it also doesn't show an ISL -- inter switch link - which would enable the two pci roots to directly talk to each other.

Most likely, this system would need to communicate via the CPUs -- but there is considerable bandwidth at the fabric layer here. I'd be curious about what the output of nvidia-smi topo- m shows.

One additional detail: PLX switches can't have "Large Memory Forwarding" turned on for dual CPU systems, as it requires them to be in "Transparent" mode not NTB/dual-host mode -- which disables large memory forwarding. That being said, you can likely get p2p access working on each group of GPUs tied to each respective root complex/cpu.

Thank you for getting back to me. I have very limited knowledge about motherboards and hardware in general.

nvidia-smi topo -m
GPU0 GPU1 GPU2 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X SYS SYS 0-47,96-143 0 N/A
GPU1 SYS X NODE 48-95,144-191 1 N/A
GPU2 SYS NODE X 48-95,144-191 1 N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

Here are the results from lspci -vvvv:
lspci-vvvv.log

Interestingly, when I installed the driver from the Ubuntu Server 24.04 repository, enabling Resizable BAR didn’t produce any errors in dmesg, but I couldn’t test P2P, and CUDA didn’t work properly either.
However, when I installed the NVIDIA-XXX.run package and enabled Resizable BAR, I encountered the following error:

[ 2.132194] kernel: pci 0000:e4:00.0: [10de:2b85] type 00 class 0x030000 PCIe Legacy Endpoint
[ 2.133137] kernel: pci 0000:e4:00.0: BAR 0 [mem 0xc0000000-0xc3ffffff]
[ 2.133150] kernel: pci 0000:e4:00.0: BAR 1 [mem 0xb0530000000-0xb053fffffff 64bit pref]
[ 2.133163] kernel: pci 0000:e4:00.0: BAR 3 [mem 0xb0542000000-0xb0543ffffff 64bit pref]
[ 2.133170] kernel: pci 0000:e4:00.0: BAR 5 [io 0x7000-0x707f]
[ 2.133176] kernel: pci 0000:e4:00.0: ROM [mem 0xc4000000-0xc407ffff pref]
[ 2.135906] kernel: pci 0000:e4:00.0: Enabling HDA controller
[ 2.136743] kernel: pci 0000:e4:00.0: PME# supported from D0 D3hot
[ 2.137295] kernel: pci 0000:e4:00.0: VF BAR 0 [mem 0xb0544000000-0xb054403ffff 64bit pref]
[ 2.137295] kernel: pci 0000:e4:00.0: VF BAR 0 [mem 0xb0544000000-0xb054403ffff 64bit pref]: contains BAR 0 for 1 VFs
[ 2.137308] kernel: pci 0000:e4:00.0: VF BAR 2 [mem 0xb0520000000-0xb052fffffff 64bit pref]
[ 2.137309] kernel: pci 0000:e4:00.0: VF BAR 2 [mem 0xb0520000000-0xb052fffffff 64bit pref]: contains BAR 2 for 1 VFs
[ 2.137322] kernel: pci 0000:e4:00.0: VF BAR 4 [mem 0xb0540000000-0xb0541ffffff 64bit pref]
[ 2.137322] kernel: pci 0000:e4:00.0: VF BAR 4 [mem 0xb0540000000-0xb0541ffffff 64bit pref]: contains BAR 4 for 1 VFs
[ 2.183939] kernel: pci 0000:e4:00.0: vgaarb: bridge control possible
[ 2.183939] kernel: pci 0000:e4:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[ 2.209675] kernel: pci 0000:e4:00.1: D0 power state depends on 0000:e4:00.0
[ 12.862345] kernel: nouveau 0000:e4:00.0: unknown chipset (1b2000a1)
[ 13.055685] kernel: nvidia 0000:e4:00.0: BAR 1 [mem 0xb0530000000-0xb053fffffff 64bit pref]: releasing
[ 13.055687] kernel: nvidia 0000:e4:00.0: BAR 3 [mem 0xb0542000000-0xb0543ffffff 64bit pref]: releasing
[ 13.055958] kernel: nvidia 0000:e4:00.0: BAR 1 [mem size 0x800000000 64bit pref]: can't assign; no space
[ 13.055959] kernel: nvidia 0000:e4:00.0: BAR 1 [mem size 0x800000000 64bit pref]: failed to assign
[ 13.055960] kernel: nvidia 0000:e4:00.0: BAR 3 [mem 0xb0530000000-0xb0531ffffff 64bit pref]: assigned
[ 13.056132] kernel: nvidia 0000:e4:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=none:owns=none
[ 13.085701] kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:e4:00.0 on minor 4
[ 16.394910] kernel: NVOC: __nvoc_objDelete: Child class KernelVideoEngine not freed from parent class OBJGPU.NVRM: GPU 0000:e4:00.0: RmInitAdapter failed! (0x24:0x72:1207)
[ 16.395590] kernel: NVRM: GPU 0000:e4:00.0: rm_init_adapter failed, device minor number 2
[ 16.561983] kernel: NVRM: GPU 0000:e4:00.0: RmInitAdapter failed! (0x62:0x40:2015)
[ 16.562622] kernel: NVRM: GPU 0000:e4:00.0: rm_init_adapter failed, device minor number 2

[ 14.495505] kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 14.495532] kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
[ 15.501657] kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 15.501676] kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM
[ 16.561510] kernel: NVRM: _kgspBootGspRm: unexpected WPR2 already up, cannot proceed with booting GSP
[ 16.561534] kernel: NVRM: RmInitAdapter: Cannot initialize GSP firmware RM