Assertion failed 580.95.05

[R-omk]

NVIDIA Open GPU Kernel Modules Version

580.95.05

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Ubuntu 24.04.3 LTS

Kernel Release

Linux aiserver 6.8.0-85-generic

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

2x GPU 0: NVIDIA GeForce RTX 5090

Describe the bug

When running the test ./p2pBandwidthLatencyTest or when running vllm using two GPUs (--tensor-parallel-size 2), the following message appears in the system log.

kernel: NVRM: nvAssertFailedNoLog: Assertion failed: ((base & RM_PAGE_MASK) == 0) @ kern_bus.c:295
kernel: NVRM: nvAssertFailedNoLog: Assertion failed: remoteWMBoxLocalAddr != ~0ULL @ kern_bus_gm200.c

The p2p test shows results, but vllm cannot start. Before installing the patch, the p2p test shows no connectivity (as expected), and vllm works.

A similar problem occurs when connecting 2x GPUs 3090.

./p2pBandwidthLatencyTest 
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 5090, pciBusID: a1, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 5090, pciBusID: c1, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1
     0       1     1
     1       1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 1531.86  27.50 
     1  27.39 1559.43 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1 
     0 1525.88  12.17 
     1  12.18 1547.08 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 1528.82  37.61 
     1  37.02 1540.12 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 1528.82  24.33 
     1  24.33 1540.12 
P2P=Disabled Latency Matrix (us)
   GPU     0      1 
     0   2.07  14.30 
     1  14.31   2.07 

   CPU     0      1 
     0   2.08   5.55 
     1   5.41   2.17 
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1 
     0   2.07   0.44 
     1   0.36   2.07 

   CPU     0      1 
     0   2.07   1.41 
     1   1.53   2.16 

To Reproduce

follow instruction in this repo (tag 580.95.05-p2p)

Bug Incidence

Always

nvidia-bug-report.log.gz

nvidia-bug-report.log.gz

More Info

Please let me know if you need any further info.

[Panchovix]

Hello, not repo creator here, but did you have this issue on 580.82.09? It seems 580.95.05 did multiple changes to P2P.

[R-omk]

@Panchovix

Initially, I encountered the problem with 580.82.09 version, but then I saw that a new version had appeared and I didn’t report it with the previous version, but the problem same for both versions.

--

It would be great if someone could confirm that VLLM was able to run without errors or hangs.

I moved the sys drive to a different motherboard and tested 2x 3090 cards. There are no assertions, but VLLM still hangs. (smi topo shows different connection type)

[Panchovix]

Can you give me a model example to test? That fits on 2x5090.

Maybe your same model and vLLM launch options to try to replicate it.

[Panchovix]

Okay for reference, I tested this model https://huggingface.co/TheHouseOfTheDude/L3.3-Animus-V10.0_Compressed-Tensors/tree/W4A16-ASYM and it works fine on 2x5090 and P2P driver.

[R-omk]

docker-compose.yaml

services:
  vllm-qwen3-next:
    command:
      - --model
      - cpatonn/Qwen3-Next-80B-A3B-Instruct-AWQ-4bit
      - --tensor-parallel-size
      - "1"
      - --pipeline-parallel-size
      - "2"
      - --gpu-memory-utilization
      - "0.92"
      - --max-model-len=200000
      - --dtype
      - float16
    deploy:
      resources:
        reservations:
          devices:
            - capabilities:
                - gpu
              driver: nvidia
              device_ids: ["1", "2"] # zero -  3090
    environment:
      CUDA_DEVICE_ORDER: PCI_BUS_ID
      NCCL_DEBUG: TRACE
      NCCL_P2P_DISABLE: "1" # only works with this option, without it an assert will occur at some point.
      VLLM_ATTENTION_BACKEND: FLASHINFER
      VLLM_LOGGING_LEVEL: DEBUG
    image: vllm/vllm-openai:v0.11.0
    ipc: host
    ports:
      - "41000:8000"
    restart: unless-stopped
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
      - ./cache:/root/.cache/vllm/

If I don't add NCCL_P2P_DISABLE option, an assertion will occur and one of the GPUs will endlessly show 100% utilization, in which case only a full host restart helps.

At the same time, if tensor-parallel-size == 2, it doesn't work even with the NCCL_P2P_DISABLE option (haven't checked with the patch from this repository yet)

[Panchovix]

Hmm I can't replicate it. I did install vLLM like this

pip install torch torchvision --index-url https://download.pytorch.org/whl/cu129

pip install packaging wheel ninja
pip install flash-attn --no-build-isolation
    
pip install flashinfer-python

pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129

And for that model it works out of the box, loading it with:

vllm serve /LLM_Models_nogg/L3.3-Animux-V10.0_Compressed-Tensors   --trust-remote-code   --quantization compressed-tensors   --tensor-parallel-size 2   --max-model-len 16384   --gpu-memory-utilization 0.80   --disable-custom-all-reduce

[aikitoria]

It seems to be working completely fine for me on a system with 8 5090 and 1 RTX 6000. I no longer have access to a system with older generations to test functionality there...

[R-omk]

I tried the new driver version.

NVIDIA-Linux-x86_64-580.105.08.run

Only 4 GPUs (5090) are installed. I'm observing similar issues.

2025-11-12T08:16:18.303635+00:00 aiserver kernel: NVRM: nvAssertFailedNoLog: Assertion failed: ((base & RM_PAGE_MASK) == 0) @ kern_bus.c:295
2025-11-12T08:16:18.303652+00:00 aiserver kernel: NVRM: nvAssertFailedNoLog: Assertion failed: remoteWMBoxLocalAddr != ~0ULL @ kern_bus_gm200.c:89
2025-11-12T08:16:18.303653+00:00 aiserver kernel: NVRM: nvAssertFailedNoLog: Assertion failed: ((base & RM_PAGE_MASK) == 0) @ kern_bus.c:295
2025-11-12T08:16:18.303654+00:00 aiserver kernel: NVRM: nvAssertFailedNoLog: Assertion failed: remoteWMBoxLocalAddr != ~0ULL @ kern_bus_gm200.c:89

nvidia-smi 
Wed Nov 12 08:21:31 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.105.08             Driver Version: 580.105.08     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5090        Off |   00000000:41:00.0 Off |                  N/A |
|  0%   33C    P0             72W /  600W |       0MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 5090        Off |   00000000:A1:00.0 Off |                  N/A |
|  0%   33C    P0             69W /  600W |       0MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GeForce RTX 5090        Off |   00000000:C1:00.0 Off |                  N/A |
|  0%   35C    P0             74W /  600W |       0MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GeForce RTX 5090        Off |   00000000:E1:00.0 Off |                  N/A |
|  0%   36C    P0             68W /  575W |       0MiB /  32607MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+



/p2pBandwidthLatencyTest 
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 5090, pciBusID: 41, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 5090, pciBusID: a1, pciDeviceID: 0, pciDomainID:0
Device: 2, NVIDIA GeForce RTX 5090, pciBusID: c1, pciDeviceID: 0, pciDomainID:0
Device: 3, NVIDIA GeForce RTX 5090, pciBusID: e1, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CAN Access Peer Device=2
Device=0 CAN Access Peer Device=3
Device=1 CAN Access Peer Device=0
Device=1 CAN Access Peer Device=2
Device=1 CAN Access Peer Device=3
Device=2 CAN Access Peer Device=0
Device=2 CAN Access Peer Device=1
Device=2 CAN Access Peer Device=3
Device=3 CAN Access Peer Device=0
Device=3 CAN Access Peer Device=1
Device=3 CAN Access Peer Device=2

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2     3
     0       1     1     1     1
     1       1     1     1     1
     2       1     1     1     1
     3       1     1     1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 1525.88  44.26  43.56  44.06 
     1  43.96 1559.38  44.28  44.19 
     2  43.95  44.35 1562.50  43.81 
     3  44.16  43.97  43.71 1559.43 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2      3 
     0 1522.95  12.17  12.18  12.14 
     1  12.06 1547.03  12.18  12.18 
     2  12.18  12.18 1540.97  12.17 
     3  12.18  12.18  12.18 1543.97 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 1528.05  57.21  57.44  57.48 
     1  57.09 1541.64  57.13  57.34 
     2  57.10  57.17 1540.88  57.00 
     3  56.99  56.78  57.21 1539.36 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3 
     0 1529.57  24.34  24.34  24.34 
     1  24.34 1542.38  24.33  24.34 
     2  24.34  24.34 1540.12  24.33 
     3  24.34  24.34  24.33 1539.34 
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2      3 
     0   2.07  13.88  14.30  14.30 
     1  14.25   2.07  14.32  14.31 
     2  12.52  14.33   2.07  12.53 
     3  14.32  13.81  14.32   2.07 

   CPU     0      1      2      3 
     0   2.07   5.20   5.51   5.16 
     1   5.10   2.03   5.42   5.06 
     2   5.32   5.35   2.15   5.30 
     3   5.10   5.10   5.46   2.03 
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2      3 
     0   2.07   0.36   0.36   0.36 
     1   0.44   2.07   0.37   0.36 
     2   0.37   0.36   2.07   0.36 
     3   0.44   0.36   0.36   2.07 

   CPU     0      1      2      3 
     0   2.05   1.41   1.39   1.39 
     1   1.49   2.04   1.38   1.43 
     2   1.56   1.53   2.15   1.53 
     3   1.43   1.40   1.44   2.07 

I suspect this may be related to reBAR, as my motherboard BIOS lacks the ability to change this setting.

[aikitoria]

If rebar is not supported, it's likely you can't use the driver. Run sudo lspci -vvv | grep -i -A40 nvidia and see if bar1 size is reported as 32G

[R-omk]

BAR 1: current size: 256MB

lspci -vvv (one sample)

a1:00.0 VGA compatible controller: NVIDIA Corporation Device 2b85 (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Palit Microsystems Inc. Device f318
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 122
        NUMA node: 0
        Region 0: Memory at bc000000 (32-bit, non-prefetchable) [size=64M]
        Region 1: Memory at 40060000000 (64-bit, prefetchable) [size=256M]
        Region 3: Memory at 40072000000 (64-bit, prefetchable) [size=32M]
        Region 5: I/O ports at c000 [size=128]
        Expansion ROM at c0000000 [virtual] [disabled] [size=512K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/16 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [60] Express (v2) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 unlimited
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s (downgraded), Width x16
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range AB, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Via message, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-
                         AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-32GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
                LnkCtl2: Target Link Speed: 32GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [9c] Vendor Specific Information: Len=14 <?>
        Capabilities: [b0] MSI-X: Enable+ Count=9 Masked-
                Vector table: BAR=0 offset=00b90000
                PBA: BAR=0 offset=00ba0000
        Capabilities: [100 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [12c v1] Latency Tolerance Reporting
                Max snoop latency: 1048576ns
                Max no snoop latency: 1048576ns
        Capabilities: [134 v1] Physical Resizable BAR
                BAR 1: current size: 256MB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
        Capabilities: [140 v1] Virtual Resizable BAR
                BAR 2: current size: 256MB, supported: 64MB 128MB 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB 256TB 512TB 1PB 2PB 4PB 8PB 16PB 32PB 64PB 128PB 256PB 512PB 1EB 2EB 4EB 8EB
        Capabilities: [14c v1] Data Link Feature <?>
        Capabilities: [158 v1] Physical Layer 16.0 GT/s <?>
        Capabilities: [188 v1] Extended Capability ID 0x2a
        Capabilities: [1b8 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO+ CmpltAbrt- UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout+ AdvNonFatalErr-
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn+ ECRCChkCap+ ECRCChkEn+
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [200 v1] Lane Margining at the Receiver <?>
        Capabilities: [248 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 1
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [250 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration- 10BitTagReq+ Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
                IOVSta: Migration-
                Initial VFs: 1, Total VFs: 1, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 2, stride: 1, Device ID: 2b85
                Supported Page Size: 00000573, System Page Size: 00000001
                Region 0: Memory at 0000040074000000 (64-bit, prefetchable)
                Region 2: Memory at 0000040050000000 (64-bit, prefetchable)
                Region 4: Memory at 0000040070000000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [290 v2] L1 PM Substates
                L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
                          PortCommonModeRestoreTime=75us PortTPowerOnTime=10us
                L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
                           T_CommonMode=0us LTR1.2_Threshold=32768ns
                L1SubCtl2: T_PwrOn=10us
        Capabilities: [2a4 v1] Vendor Specific Information: ID=0001 Rev=1 Len=014 <?>
        Capabilities: [2bc v1] Power Budgeting <?>
        Capabilities: [2f4 v1] Device Serial Number 8e-7e-2b-a0-3b-2d-b0-48
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

---

I see that the messages in the logs

NVRM: nvAssertFailedNoLog: Assertion failed: remoteWMBoxLocalAddr != ~0ULL @ kern_bus_gm200.c:89

are the same as those mentioned in ticket #6 , and bandwidth is also decreasing when p2p is enabled.

[aikitoria]

Unfortunately 5090s don't default to 32G bar if available like 4090s do, the driver must resize them first, and this can only be done if the system supports rebar.