Skip to content

perf(cuda): reduce MMQ stream-k overhead#518

Open
dusterbloom wants to merge 1 commit into
Luce-Org:mainfrom
dusterbloom:perf/mmq-stream-k-overhead
Open

perf(cuda): reduce MMQ stream-k overhead#518
dusterbloom wants to merge 1 commit into
Luce-Org:mainfrom
dusterbloom:perf/mmq-stream-k-overhead

Conversation

@dusterbloom

@dusterbloom dusterbloom commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

What changed

  • Backport upstream llama.cpp #22298 (9725a313be0528214c4a02fed906ddaf7b3f712e) into the GGML snapshot vendored by Lucebox Hub.
  • Record the backport in server/deps/llama.cpp/VENDOR.md.
  • Add a CUDA-only IQ4_XS CPU-vs-CUDA MUL_MAT oracle covering the scheduler's high-efficiency no-fixup path, partial rows, stream-k fixup path, two-wave boundary, and a deeper K dimension.

The resulting mmq.cuh Git blob is b867d39653c19cf4efd3bb7809e8b43572a79a6e, matching the upstream patch output exactly.

Why

Lucebox Hub vendors GGML directly, and its vendored mmq.cuh did not yet include upstream's MMQ stream-k overhead reduction. Qwen3.6-27B IQ4_XS prefill spends enough time in this path for the scheduler and fixup overhead to be measurable.

This is intentionally a surgical upstream backport rather than a broad GGML synchronization.

Measured impact

Fresh A-B-B-A validation was collected on Hub main at 0e0023649131a23f45d58be71f2bfc60d6cd25a0, using the same RTX 3090, Qwen3.6-27B IQ4_XS model, CUDA 12.6/GCC 11 toolchain, runtime flags, salted prompts, and disabled prefix/prefill caches. Each phase used one warmup and five measured repetitions per context.

Context Clean pooled median Candidate pooled median Change
1,024 1300.68 tok/s 1365.68 tok/s +5.00%
8,192 1312.16 tok/s 1382.97 tok/s +5.40%

Content-addressed current-tip evidence:

  • Evidence ID: evidence-sha256-be8ec967d917b0a1110e55fc5571b23756f77228739a5a374b00c6898a565964
  • Evidence file SHA-256: 5805ded9110250a1b178f5952e125dd2bda6eec211074931f33017209009d143
  • Clean executable SHA-256: 8ad8a9fa53e54ef640f1503527c1990455811fa19bae8b8f0fc736004d693ad0
  • Candidate executable SHA-256: dddb7db95cd7c54d0a2d0c84b8ab0f3dc30cd5daf7f5153217d5b42c1669b232
  • Model SHA-256: 8a3365759dc1b33b52c4e7d91d5a67d5ee1418e8408aa54196f04a98da53e5dc

The earlier full-logit, repeat-determinism, and generation-canary campaign remains separately pinned to its original Hub revision in AutoLuce's quality report. It is supporting historical evidence, not relabeled as current-tip evidence.

Validation

  • dflash_server Release build, CUDA 12.6, GCC 11, sm_86, -j4: pass.
  • Focused test_mmq_streamk_iq4_xs on RTX 3090: 5/5 pass; all outputs finite and all NMSE values below GGML's 5e-4 threshold.
  • Candidate MMQ Git blob matches the upstream patch output.
  • git diff --check: pass.
  • AutoLuce harness/tests: 345 pass; Ruff: pass.
  • Adversarial GLM-5.2 review: no code defects or portability blockers found.

The test is built only for the CUDA backend. The backported kernel retains upstream's existing non-CDNA HIP and pre-Volta fallback behavior; no new HIP-specific implementation is introduced.

Review in cubic

@dusterbloom dusterbloom marked this pull request as ready for review July 13, 2026 12:57

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 4 files

Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant