[cuda] initial Q1_0 backend

[khosravipasha]

DRAFT for running CIs and etc, main PR will go main llama.cpp.

[khosravipasha]

  Benchmark Results — RTX 5090, Q1_0 CUDA, Flash Attention

  Speed (llama-bench -fa 1)

  ┌─────────────┬──────────┬─────────────┬─────────────┐
  │    Model    │   Size   │ pp512 (t/s) │ tg128 (t/s) │
  ├─────────────┼──────────┼─────────────┼─────────────┤
  │ Bonsai-1.7B │ 231 MiB  │ 29,815      │ 627         │
  ├─────────────┼──────────┼─────────────┼─────────────┤
  │ Bonsai-4B   │ 540 MiB  │ 18,585      │ 486         │
  ├─────────────┼──────────┼─────────────┼─────────────┤
  │ Bonsai-8B   │ 1.07 GiB │ 12,238      │ 373         │
  └─────────────┴──────────┴─────────────┴─────────────┘

  KL Validation (Q1_0 vs dequantized F16, WikiText-2, 20 chunks)

  ┌─────────────┬──────────┬────────────┬────────┬────────┐
  │    Model    │ Mean KLD │ Same Top p │ RMS Δp │ Status │
  ├─────────────┼──────────┼────────────┼────────┼────────┤
  │ Bonsai-1.7B │ 0.000419 │ 98.94%     │ 0.555% │ Pass   │
  ├─────────────┼──────────┼────────────┼────────┼────────┤
  │ Bonsai-4B   │ 0.000429 │ 98.51%     │ 0.593% │ Pass   │
  ├─────────────┼──────────┼────────────┼────────┼────────┤
  │ Bonsai-8B   │ 0.000514 │ 98.71%     │ 0.635% │ Pass   │
  └─────────────┴──────────┴────────────┴────────┴────────┘

[Copilot]

Pull request overview

Adds initial CUDA backend support for the GGML_TYPE_Q1_0 quantization format, wiring it into CUDA dequantization/conversion paths and the quantized matmul kernels (MMVQ/MMQ).

Changes:

• Implemented Q1_0×Q8_1 dot-product support and hooked it into mul_mat_vec_q (MMVQ).
• Added MMQ (mul_mat_q) support for Q1_0, including MMA-tile loading and template instantiation generation.
• Enabled Q1_0 for CUDA get-rows and type conversion/dequantization helpers.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
ggml/src/ggml-cuda/vecdotq.cuh	Adds Q1_0 dot-product helpers (Q1_0 × Q8_1) and VDR constants.
ggml/src/ggml-cuda/template-instances/mmq-instance-q1_0.cu	Adds generated MMQ instantiation unit for Q1_0.
ggml/src/ggml-cuda/template-instances/generate_cu_files.py	Ensures MMQ instantiation files are generated for Q1_0.
ggml/src/ggml-cuda/mmvq.cu	Routes Q1_0 through MMVQ vector-dot and VDR selection, and adds type dispatch.
ggml/src/ggml-cuda/mmq.cuh	Adds Q1_0 MMQ traits + tile loader and ds-layout selection.
ggml/src/ggml-cuda/mmq.cu	Adds Q1_0 MMQ type dispatch and selection logic gating.
ggml/src/ggml-cuda/ggml-cuda.cu	Advertises Q1_0 support in CUDA backend op capability checks.
ggml/src/ggml-cuda/getrows.cu	Enables get_rows for Q1_0 via dequantize kernel.
ggml/src/ggml-cuda/dequantize.cuh	Adds device-side dequantization routine for Q1_0.
ggml/src/ggml-cuda/convert.cu	Enables Q1_0 conversions to FP16/FP32 (contiguous and non-contiguous).
ggml/src/ggml-cuda/common.cuh	Adds CUDA type traits for Q1_0 (qk/qr/qi).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[khosravipasha]

  ┌───────┬───────────┬───────────┬───────────┬───────────┐
  │ Model │ Old pp512 │ New pp512 │ Old tg128 │ New tg128 │
  ├───────┼───────────┼───────────┼───────────┼───────────┤
  │ 1.7B  │ 29,815    │ 29,250    │ 627       │ 626       │
  ├───────┼───────────┼───────────┼───────────┼───────────┤
  │ 4B    │ 18,585    │ 18,621    │ 486       │ 485       │
  ├───────┼───────────┼───────────┼───────────┼───────────┤
  │ 8B    │ 12,238    │ 12,287    │ 373       │ 374       │
  └───────┴───────────┴───────────┴───────────┴───────────┘
  
  
 ┌───────┬──────────┬────────────┬────────┐
  │ Model │ Mean KLD │ Same Top p │ RMS Δp │
  ├───────┼──────────┼────────────┼────────┤
  │ 1.7B  │ 0.000419 │ 98.94%     │ 0.555% │
  ├───────┼──────────┼────────────┼────────┤
  │ 4B    │ 0.000429 │ 98.51%     │ 0.593% │
  ├───────┼──────────┼────────────┼────────┤
  │ 8B    │ 0.000514 │ 98.71%     │ 0.635% │
  └───────┴──────────┴────────────┴────────┘