Eval bug: SYCL: Qwen3.5 spitting garbage on the second prompt

[WizardlyBump17]

Name and Version

root@cb8f619a68ce:/app# ./llama-cli --version
load_backend: loaded SYCL backend from /app/libggml-sycl.so
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
version: 8699 (4eb19514d)
built with IntelLLVM 2025.3.2 for Linux x86_64

Operating systems

Linux

GGML backends

SYCL

Hardware

Ryzen 7 5700X3D + B580

Models

Any Qwen3.5 that is not F16 or BF16

Problem description & steps to reproduce

Just send two prompts and watch the second one being responded with garbage

First Bad Commit

No response

Relevant log output

Logs

root@cb8f619a68ce:/app# ./llama-cli --model /models/Qwen3.5-35B-A3B-Q4_K_M.gguf
load_backend: loaded SYCL backend from /app/libggml-sycl.so
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so

Loading model...  


▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀

build      : b8699-4eb19514d
model      : Qwen3.5-35B-A3B-Q4_K_M.gguf
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read <file>        add a text file
  /glob <pattern>     add text files using globbing pattern


> hi

[Start thinking]
Thinking Process:

1.  **Analyze the Request:**
    *   Input: "hi"
    *   Intent: Greeting.
    *   Context: Start of a conversation.
    *   Tone: Friendly, helpful, casual.

2.  **Determine the appropriate response:**
    *   Acknowledge the greeting.
    *   Offer assistance.
    *   Keep it concise but warm.

3.  **Draft potential responses:**
    *   "Hello! How can I help you today?"
    *   "Hi there! What's on your mind?"
    *   "Hello! Nice to meet you. How are you doing?"
    *   "Hi! 👋 Need any help with something specific?"

4.  **Select the best option:** A combination of friendly greeting and offer of help is standard and effective. "Hello! How can I help you today?" is clear and professional yet friendly. Adding an emoji can make it warmer.

5.  **Final Polish:** "Hello! 👋 How can I help you today?" or "Hi there! How's it going? What can I do for you?" Let's go with a simple, welcoming response.

6.  **Final Output Generation:** (Matches the selected draft)
    "Hello! 👋 How can I help you today?" or similar.

    *Self-Correction during thought process:* Keep it natural. Don't overthink a simple "hi".

    *Final Decision:* "Hello! 👋 How can I help you today?" or "Hi there! How's it going?" Let's use a friendly greeting + offer help.

    "Hello! 👋 How can I assist you today?" (Professional yet friendly)
    "Hi! How's it going? Anything I can help you with?" (More casual)

    Let's go with: "Hello! 👋 How can I help you today?" (Balanced)

    *Wait, checking system instructions:* I am an AI assistant. I should be helpful.

    *Revised Plan:* Just say hi back and ask how to help.

    *Final Output:* "Hello! 👋 How can I help you today?" or "Hi there! What can I do for you?"

    Let's pick: "Hello! 👋 How can I help you today?" (Simple and effective).

    *Wait, one more consideration:* Sometimes users just want to chat. So "How are you?" is also good.
    "Hi there! How's it going? Is there something specific you'd like to talk about or work on?"

    Let's keep it open-ended but inviting.

    "Hello! 👋 How's it going? Is there anything I can help you with today?"

    Actually, brevity is often better for a "hi".
    "Hi there! 👋 How can I help you today?"

    Okay, ready to output.cw

[End thinking]

Hello! 👋 How's it going? Is there anything I can help you with today?

[ Prompt: 48.2 t/s | Generation: 28.1 t/s ]

> hi

[Start thinking]
-s55=G,G9+125H89-*C"2@'8+=H53CAG%&4H8E<>6'<<.%F."4-<"-B)G1E8/'8+%D*@/2!F7>*C1DD2,(4=)CF;*E;D<CA6AHGA;F(HH<+&$&0GF+4H@'(C0>!%G55:,$E2GB2$'6=9,:,#"/E3G':+$+BBE+5C7B24.4E+-:6$>/E)F,:D63)F(19>AGB(4:88;C/&$5H$68C+50&56:<:!(":<D2A*<0H1&<G4.1E*0HF!4@A9/(BFG1G>!/!7/A#'-*CB,7:60;H"72+;6';#$$(#C#1:4E-*5'#&E)+E!72B6*,$5.G.6$*-H#F!2=:E.E09=!;#$>ABA,2:2G'HE;</)@=-#=*&@C'5DHHE:!5&/+&+))6$F(@-G,/H7/1;%E>$EB(7>#0+D=-C;@<<E,B=7-8#"G2+9DG($<!<*0.<''@&$'5B76"B-9*>8H@CH0C1+H6>7)E9;<A6&<;8(83(!%@15B,8-FEB87=3CG,#4+8>$$&:7H+91&6!F2%DF8>9+8D-88)-)>*!.*6-$!D#4HC4>&>DH7FA:9:&-186-,9&%C&;='/4E>E!7.C3"5%5<'5CEC=*;"!H"'+6%%/<3/>,>2*7='5<*E+,!G@AD4@5H"B2<C,AGBH9G"(#%1E,$6*

[ Prompt: 51.1 t/s | Generation: 29.1 t/s ]

> 

Exiting...
llama_memory_breakdown_print: | memory breakdown [MiB]                     | total   free     self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - SYCL0 (Intel(R) Arc(TM) B580 Graphics) | 12216 = 1023 + (10922 = 10282 +     142 +     497) +         270 |
llama_memory_breakdown_print: |   - Host                                   |                 19830 = 19814 +       0 +      16                |
root@cb8f619a68ce:/app# 

[WizardlyBump17]

Reverting 0988acc fixes the issue

[sniperwhg]

Observing the same functional regression on an Arc B70.

[NeoZhangJianyu]

It's complex issue.
I can check it.

Thank you!

[PMZFX]

This is caused by my PR #21527 (Q8_0 reorder optimization). Root cause identified: the GEMM dequantization path in convert.cpp is missing a reorder-aware variant for Q8_0. After token generation reorders the weights, prompt processing reads them with the wrong layout.

Q4_0, Q4_K, and Q6_K already have _reorder dequantizers there. Q8_0 is missing them. Working on a fix now.

Workaround: GGML_SYCL_DISABLE_OPT=1 disables the reorder optimization entirely.

[PMZFX]

Fix submitted: #21638

[PMZFX]

@NeoZhangJianyu I have the fix ready in #21638 if you'd like to review it. Here's the full breakdown:

What happened: My PR #21527 added Q8_0 reorder support for the DMMV and MMVQ kernel paths (used during token generation, batch=1), but missed the GEMM dequantization path (used during prompt processing, batch>8). After the first token generation pass reorders the Q8_0 weights in-place, any subsequent prompt processing reads the reordered data through the standard dequantize_q8_0 function, which expects the original interleaved layout. Result: garbage output from the second prompt onward.

The existing reorder types (Q4_0, Q4_K, Q6_K) already had _reorder variants in both ggml_get_to_fp16_sycl() and ggml_get_to_fp32_sycl() in convert.cpp. Q8_0 was simply missing them.

The fix (3 files, ~60 lines) adds:

• dequantize_block_q8_0_reorder() in dequantize.hpp
• dequantize_row_q8_0_sycl_reorder() in convert.cpp
• Reorder-aware dispatch in both ggml_get_to_fp16_sycl() and ggml_get_to_fp32_sycl()

Testing on Arc Pro B70 (Xe2), oneAPI DPC++ 2025.3.3:

• 5 sequential prompts on Qwen3.5-9B Q8_0: all coherent
• 5 sequential prompts on Qwen3.5-35B-A3B Q4_K_M (mixed quant with Q8_0 tensors): all coherent
• 3 sequential prompts on Qwen2.5-1.5B Q8_0: all coherent
• llama-bench confirms the Q8_0 tg speedup is preserved (180 t/s)

Code was written with AI assistance (Claude), reviewed and tested by me on real hardware.

@WizardlyBump17 @sniperwhg @encodatamHirmer if you're able to test the fix branch before it merges, that would be appreciated.

[encodatamHirmer]

@PMZFX works for me now. Tested with

• unsloth/gemma-4-26B-A4B-it:UD-Q4_K_XL
• unsloth/Nemotron-3-Nano-30B-A3B:Q8_0
  Thank you!

[sniperwhg]

@PMZFX Functionality has been restored, but the performance increase I saw with the original fix seems to have been lost for unsloth/Qwen-3.5-27B-UD_Q6_K_XL.

[PMZFX]

@encodatamHirmer Great to hear, thanks for testing!

@sniperwhg Thanks for testing. On my B70, Q6_K and Q8_0 performance is preserved on the fix branch (Qwen3.5-27B: Q6_K 15.36 tg, Q8_0 15.31 tg, matching my pre-fix-branch reference numbers).

Could you share your specific before/after numbers? Also, what are you comparing against: master before #21527, or master with #21527 (the broken version)?

Q6_K_XL is a mixed quant format. The Q8_0 speedup would only apply to whichever tensors in the model are stored as Q8_0. If most tensors are Q6_K, the overall speedup from the Q8_0 reorder would be small.

[sniperwhg]

@PMZFX I previously saw (9.31 t/s - >14.4 t/s) (master pre-fix then post fix) , now back to around 9.6 t/s after building from your repo.