Xid 43 GPU crash on RTX 5090 (Blackwell) after b8680 — flash_attn_stream_k_fixup kernel

[lance0]

Summary

After upgrading from b8679 to b8685, llama-server crashes repeatedly with NVIDIA Xid 43 errors on RTX 5090 (Blackwell, sm_120a). Rolling back to b8679 resolves the issue completely. The only CUDA-related change in this range is #21159 (flash_attn_stream_k_fixup kernel optimization, merged as b8680).

Environment

• GPU: NVIDIA RTX 5090 (GB202, sm_120a, 32GB GDDR7)
• CUDA Toolkit: 12.8 (/usr/local/cuda-12.8)
• Driver: nvidia-driver-580-open (580.126.20)
• Build flags: -DCMAKE_CUDA_ARCHITECTURES="120" (auto-converts to 120a), -DGGML_CUDA_FA_ALL_QUANTS=ON, -DGGML_NATIVE=ON
• OS: Ubuntu 24.04, kernel 6.17.0-20-generic
• Model: Qwen3.5-27B UD-Q6_K_XL (25.7GB), 131K context, q4_0 KV cache, flash attention enabled

Reproduction

1. Build llama.cpp at b8685 (or any tag >= b8680) with CUDA 12.8 + sm_120a
2. Start llama-server with flash attention enabled and any model
3. Send a chat completion request
4. Server crashes with Xid 43

With b8679, the same configuration runs indefinitely without errors.

dmesg output

2026-04-07T10:11:08,080132-04:00 NVRM: Xid (PCI:0000:41:00): 43, pid=1049772, name=llama-server, channel 0x00000002
2026-04-07T10:11:29,913774-04:00 NVRM: Xid (PCI:0000:41:00): 43, pid=1108679, name=llama-server, channel 0x00000002
2026-04-07T10:12:37,588699-04:00 NVRM: Xid (PCI:0000:41:00): 43, pid=1108875, name=llama-server, channel 0x00000002
2026-04-07T10:13:21,041572-04:00 NVRM: Xid (PCI:0000:41:00): 43, pid=1109236, name=llama-server, channel 0x00000002
2026-04-07T10:13:41,759143-04:00 NVRM: Xid (PCI:0000:41:00): 43, pid=1109611, name=llama-server, channel 0x00000002
2026-04-07T10:14:48,131891-04:00 NVRM: Xid (PCI:0000:41:00): 43, pid=1109788, name=llama-server, channel 0x00000002
2026-04-07T10:19:15,071737-04:00 NVRM: Xid (PCI:0000:41:00): 43, pid=1110167, name=llama-server, channel 0x00000002
2026-04-07T10:19:35,554712-04:00 NVRM: Xid (PCI:0000:41:00): 43, pid=1112450, name=llama-server, channel 0x00000002

Each PID is a different llama-server instance (process manager auto-restarts after crash). 8 crashes in ~10 minutes.

Bisect

b8679 (94ca829b6) — STABLE, no crashes
b8680 (15f786e65) — first tag with #21159 [CUDA] flash_attn_stream_k_fixup
b8685 (0988accf8) — crashes immediately

Commits between b8679 and b8685:

• 0988accf8 [SYCL] Q8_0 reorder ([SYCL] Add Q8_0 reorder optimization for Intel GPUs (~3x token generation speedup) #21527) — not CUDA
• 0033f53a0 docs typo fix
• d0a6dfeb2 WebGPU MUL_MAT_ID (ggml-webgpu: Add the support of MUL_MAT_ID #21147) — not CUDA
• 2e1f0a889 Q1_0 1-bit quant CPU (ggml: add Q1_0 1-bit quantization support (CPU) #21273) — not CUDA
• 506200cf8 CLI newline fix (console: fix stripping of \n in multiline input #21485) — not CUDA
• 15f786e65 [CUDA] flash_attn_stream_k_fixup ([CUDA ] Write an optimized flash_attn_stream_k_fixup kernel #21159) — only CUDA change

Notes

• Xid 43 = illegal memory access on the GPU (not OOM, not driver timeout)
• The Xid 8 errors from the previous day (different PIDs, different channel) were a separate issue with Gemma 4 mmproj, unrelated
• CUDA 12.8 builds with native sm_120a — no PTX JIT, no FORCE_CUBLAS
• This GPU has been stable on b8665–b8679 for weeks with the same model and configuration

[am17an]

Please elaborate on the steps to reproduce, the exact chat completion request with the exact llama-server command (model + parameters) to reproduce

[lance0]

Reproduction

Model: unsloth/Qwen3.5-27B-GGUF — Qwen3.5-27B-UD-Q6_K_XL.gguf (25.7GB)

Server command:

llama-server \
  --model Qwen3.5-27B-UD-Q6_K_XL.gguf \
  --ctx-size 131072 \
  --threads 4 \
  --threads-batch 24 \
  --batch-size 4096 \
  --ubatch-size 1024 \
  --gpu-layers -1 \
  --cache-type-k q4_0 \
  --cache-type-v q4_0 \
  --flash-attn \
  --jinja \
  --reasoning-budget 0 \
  --temp 0.7 \
  --top-p 0.8 \
  --top-k 20 \
  --port 8081

Request (crashes on first or second request):

curl http://localhost:8081/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"test","messages":[{"role":"user","content":"Hello, how are you?"}],"max_tokens":50}'

The server starts fine and passes warmup. On the first or second chat completion request, it crashes with Xid 43 in dmesg. The process manager restarts it, and the next request crashes it again — this repeats indefinitely.

Build:

cmake -B build \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES="120" \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DGGML_NATIVE=ON \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.8/bin/nvcc

cmake --build build -j$(nproc)

Identical config on b8679 runs indefinitely without any Xid errors. The only CUDA change between b8679 and b8680 is this PR (#21159).

[am17an]

Cc @gaugarg-nv @JohannesGaessler

[gaugarg-nv]

Working on a repro. Will report back soon.

[gaugarg-nv]

@lance0 I can't repro this issue on my RTX 5090 setup.

The main difference on my system is that I am using CUDA driver 595.58.03 and CTK 13.0.2. Can you try upgrading these?

[lance0]

Will do and report back.

[simmonsharper247-oss]

Thanks for the detailed report and bisect — this is very helpful.

…

On Tue, Apr 7, 2026, 9:27 AM lance ***@***.***> wrote: *lance0* created an issue (ggml-org/llama.cpp#21564) <#21564> Summary After upgrading from b8679 to b8685, llama-server crashes repeatedly with NVIDIA Xid 43 errors on RTX 5090 (Blackwell, sm_120a). Rolling back to b8679 resolves the issue completely. The only CUDA-related change in this range is #21159 <#21159> ( flash_attn_stream_k_fixup kernel optimization, merged as b8680). Environment - *GPU*: NVIDIA RTX 5090 (GB202, sm_120a, 32GB GDDR7) - *CUDA Toolkit*: 12.8 (/usr/local/cuda-12.8) - *Driver*: nvidia-driver-580-open (580.126.20) - *Build flags*: -DCMAKE_CUDA_ARCHITECTURES="120" (auto-converts to 120a), -DGGML_CUDA_FA_ALL_QUANTS=ON, -DGGML_NATIVE=ON - *OS*: Ubuntu 24.04, kernel 6.17.0-20-generic - *Model*: Qwen3.5-27B UD-Q6_K_XL (25.7GB), 131K context, q4_0 KV cache, flash attention enabled Reproduction 1. Build llama.cpp at b8685 (or any tag >= b8680) with CUDA 12.8 + sm_120a 2. Start llama-server with flash attention enabled and any model 3. Send a chat completion request 4. Server crashes with Xid 43 With b8679, the same configuration runs indefinitely without errors. dmesg output 2026-04-07T10:11:08,080132-04:00 NVRM: Xid (PCI:0000:41:00): 43, pid=1049772, name=llama-server, channel 0x00000002 2026-04-07T10:11:29,913774-04:00 NVRM: Xid (PCI:0000:41:00): 43, pid=1108679, name=llama-server, channel 0x00000002 2026-04-07T10:12:37,588699-04:00 NVRM: Xid (PCI:0000:41:00): 43, pid=1108875, name=llama-server, channel 0x00000002 2026-04-07T10:13:21,041572-04:00 NVRM: Xid (PCI:0000:41:00): 43, pid=1109236, name=llama-server, channel 0x00000002 2026-04-07T10:13:41,759143-04:00 NVRM: Xid (PCI:0000:41:00): 43, pid=1109611, name=llama-server, channel 0x00000002 2026-04-07T10:14:48,131891-04:00 NVRM: Xid (PCI:0000:41:00): 43, pid=1109788, name=llama-server, channel 0x00000002 2026-04-07T10:19:15,071737-04:00 NVRM: Xid (PCI:0000:41:00): 43, pid=1110167, name=llama-server, channel 0x00000002 2026-04-07T10:19:35,554712-04:00 NVRM: Xid (PCI:0000:41:00): 43, pid=1112450, name=llama-server, channel 0x00000002 Each PID is a different llama-server instance (process manager auto-restarts after crash). 8 crashes in ~10 minutes. Bisect b8679 (94ca829) — STABLE, no crashes b8680 (15f786e) — first tag with #21159 [CUDA] flash_attn_stream_k_fixup b8685 (0988acc) — crashes immediately Commits between b8679 and b8685: - 0988acc [SYCL] Q8_0 reorder (#21527 <#21527>) — not CUDA - 0033f53 docs typo fix - d0a6dfe WebGPU MUL_MAT_ID (#21147 <#21147>) — not CUDA - 2e1f0a8 Q1_0 1-bit quant CPU (#21273 <#21273>) — not CUDA - 506200c CLI newline fix (#21485 <#21485>) — not CUDA - *15f786e65 [CUDA] flash_attn_stream_k_fixup (#21159 <#21159>) — only CUDA change* Notes - Xid 43 = illegal memory access on the GPU (not OOM, not driver timeout) - The Xid 8 errors from the previous day (different PIDs, different channel) were a separate issue with Gemma 4 mmproj, unrelated - CUDA 12.8 builds with native sm_120a — no PTX JIT, no FORCE_CUBLAS - This GPU has been stable on b8665–b8679 for weeks with the same model and configuration — Reply to this email directly, view it on GitHub <#21564>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/B7VN24M2PHZUOVHCFWK7S534UUF4BAVCNFSM6AAAAACXPQ2UZSVHI2DSMVQWIX3LMV43ASLTON2WKOZUGIYTQMZWGA3TSMA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[lance0]

@gaugarg-nv Thanks for looking into this. Tested with driver 590.48.01 (up from 580) + CUDA 12.8 + b8708 — still crashes under load.

Could you share more about your setup? Specifically:

• Are you building with -DCMAKE_CUDA_ARCHITECTURES="120" or something else?
• Which model/quant are you testing with? (We hit it with K-quants like Q6_K and Q4_K on Qwen3.5-27B)
• Are you using flash attention (-fa on) with quantized KV cache (-ctk q4_0/q8_0)?
• Any specific llama-server flags or just defaults?

Want to make sure we're testing the same code path. The crash only happens under the flash_attn_stream_k_fixup kernel from PR #21159.

[gaugarg-nv]

Can you try 595.58.03 and CTK 13.0.2?

I have used the same compilation flag, runtime options, and model as #21564 (comment). The difference was I passed --flash-attn 1. I think option 1 or on was missing from your CLI options.

I tried running the curl command one after another sequentially ~10-20 times.

[JohannesGaessler]

The OP looks very much like it was generated by a language model, if no one else can reproduce this issue it may just be an incorrect report.

[lance0]

@gaugarg-nv will give it a go and report back as soon as I can. Thanks!

[lance0]

Full test matrix on RTX 5090, b8708, Qwen3.5-27B UD-Q6_K_XL, -fa on -ctk q4_0 -ctv q4_0, 131K context:

Driver	CUDA Toolkit	Result
580.126.20	12.8	Crash (Xid 43)
590.48.01	12.8	Crash
590.48.01	13.0 (nvcc 13.0.88)	Crash (worse — instant, first request)

All tests crash on b8708, all stable on b8679 with same config. Rolled back to b8679 + CUDA 12.8 each time.

595.58.03 isn't available via apt for Ubuntu 24.04 yet — no packages in the NVIDIA CUDA repo or Ubuntu repos. Is there a clean way to get it besides the .run installer? Or could you confirm whether you're testing with quantized KV cache (-ctk q4_0)?