perf(cuda): reduce MMQ stream-k overhead#518
Open
dusterbloom wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
9725a313be0528214c4a02fed906ddaf7b3f712e) into the GGML snapshot vendored by Lucebox Hub.server/deps/llama.cpp/VENDOR.md.MUL_MAToracle covering the scheduler's high-efficiency no-fixup path, partial rows, stream-k fixup path, two-wave boundary, and a deeper K dimension.The resulting
mmq.cuhGit blob isb867d39653c19cf4efd3bb7809e8b43572a79a6e, matching the upstream patch output exactly.Why
Lucebox Hub vendors GGML directly, and its vendored
mmq.cuhdid not yet include upstream's MMQ stream-k overhead reduction. Qwen3.6-27B IQ4_XS prefill spends enough time in this path for the scheduler and fixup overhead to be measurable.This is intentionally a surgical upstream backport rather than a broad GGML synchronization.
Measured impact
Fresh A-B-B-A validation was collected on Hub
mainat0e0023649131a23f45d58be71f2bfc60d6cd25a0, using the same RTX 3090, Qwen3.6-27B IQ4_XS model, CUDA 12.6/GCC 11 toolchain, runtime flags, salted prompts, and disabled prefix/prefill caches. Each phase used one warmup and five measured repetitions per context.Content-addressed current-tip evidence:
evidence-sha256-be8ec967d917b0a1110e55fc5571b23756f77228739a5a374b00c6898a5659645805ded9110250a1b178f5952e125dd2bda6eec211074931f33017209009d1438ad8a9fa53e54ef640f1503527c1990455811fa19bae8b8f0fc736004d693ad0dddb7db95cd7c54d0a2d0c84b8ab0f3dc30cd5daf7f5153217d5b42c1669b2328a3365759dc1b33b52c4e7d91d5a67d5ee1418e8408aa54196f04a98da53e5dcThe earlier full-logit, repeat-determinism, and generation-canary campaign remains separately pinned to its original Hub revision in AutoLuce's quality report. It is supporting historical evidence, not relabeled as current-tip evidence.
Validation
dflash_serverRelease build, CUDA 12.6, GCC 11, sm_86,-j4: pass.test_mmq_streamk_iq4_xson RTX 3090: 5/5 pass; all outputs finite and all NMSE values below GGML's5e-4threshold.git diff --check: pass.The test is built only for the CUDA backend. The backported kernel retains upstream's existing non-CDNA HIP and pre-Volta fallback behavior; no new HIP-specific implementation is introduced.