[AMD] MiniMax-M3 MXFP8 MI355X vLLM disagg: TP4 worker-ratio + serve sync / MiniMax-M3 MXFP8 MI355X vLLM 分离式：TP4 工作单元配比 + serve 参数对齐#2144
Conversation
hongxiayang
requested review from
1am9trash,
chunfangamd,
seungrokj and
yctseng0211
as code owners
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
LGTM — config-only retune of a benchmark sweep with test-sweep validation; only a stale-comment nit was found.
Extended reasoning...
Overview
This PR retunes the minimaxm3-fp8-mi355x-vllm-disagg benchmark recipe: bumps the vLLM ROCm image, syncs per-worker serve flags/env with the validated single-node MiniMax-M3 MXFP8 recipe (--moe-backend aiter, --linear-backend emulation, --max-num-batched-tokens 32768, VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1, VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT6), and replaces the mixed-TP search space with three all-TP4 P/D layouts tuning the prefill:decode worker ratio (1P/1D, 1P/2D, 2P/1D) that respect the 3-node mi355x-disagg pool constraint. Touches only 3 YAML files: benchmarks/multi_node/amd_utils/models_vllm.yaml, configs/amd-master.yaml, and perf-changelog.yaml.
Security risks
None. This is purely benchmark configuration data — no code paths, no auth/crypto/permissions changes, no new inputs from untrusted sources. The image bump moves to a specific SHA-pinned vLLM ROCm nightly (rather than a floating tag), which is the safer pattern already established elsewhere in this file.
Level of scrutiny
Low. This is a config-only PR in the benchmark sweep space, following established patterns in the file (e.g. the sister minimaxm3-fp4-mi355x-vllm-disagg recipe already uses the same TP4 P/D layout structure and worker-ratio approach). The author cites a validation sweep with both multi-node eval jobs green (GSM8K ~0.958) and provides per-GPU throughput numbers. All layouts respect the xP + yD <= 3 node budget for the 3-node mi355x-disagg pool. The comment block explaining the new topology in configs/amd-master.yaml is thorough and matches the actual search-space entries.
Other factors
The one finding from the bug hunt is a documentation nit (a parenthetical about 'TP8 and TP4 layouts' in models_vllm.yaml that is now stale since every worker in the sweep is TP4) — non-blocking, safe to address in a follow-up if desired. No CODEOWNER-critical code paths are touched. No prior reviewer comments to address.
| @@ -55,7 +55,13 @@ MiniMax-M3-MXFP8: | |||
| # from the master-config prefill/decode tp (the sweep mixes TP8 and TP4 layouts). | |||
There was a problem hiding this comment.
🟡 The comment on line 55 says '(the sweep mixes TP8 and TP4 layouts)', but this PR retunes minimaxm3-fp8-mi355x-vllm-disagg (the only consumer of the MiniMax-M3-MXFP8 model_vllm block) to use TP4 across every prefill and decode worker — only the prefill:decode worker ratio varies now. The placeholder-rewrite mechanism this comment justifies is still correct, but the parenthetical reason is stale; consider updating to e.g. '(the sweep uses TP4 workers only; server_vllm.sh still rewrites the placeholder from the master-config prefill/decode tp)'.
Extended reasoning...
What the bug is. The comment block preceding prefill_flags/decode_flags in the MiniMax-M3-MXFP8 entry (benchmarks/multi_node/amd_utils/models_vllm.yaml line 55) explains why --tensor-parallel-size 8 in the flag strings is a placeholder: server_vllm.sh sed-rewrites it to PREFILL_TP_SIZE/DECODE_TP_SIZE from the master-config prefill/decode tp, because '(the sweep mixes TP8 and TP4 layouts)'. After this PR, that parenthetical is no longer true.\n\nWhy the parenthetical is stale. The only sweep entry that consumes this MiniMax-M3-MXFP8 model_vllm block is minimaxm3-fp8-mi355x-vllm-disagg (the other minimaxm3-fp8-mi355x-*-disagg entry uses framework: atom-disagg and a different model, so it does not go through models_vllm.yaml). This PR retunes that sweep to three P/D layouts that are all TP4: 1P-TP4/1D-TP4, 1P-TP4/2D-TP4, and 2P-TP4/1D-TP4. Every removed layout was the TP8 or mixed-TP one; no TP8 worker remains anywhere in the sweep.\n\nImpact. Zero runtime impact — the sed-rewrite from PREFILL_TP_SIZE/DECODE_TP_SIZE continues to work correctly regardless of what mix the sweep uses. The stale justification is just documentation drift: a future reader will look for the TP8+TP4 mix the comment promises, not find one, and either doubt the placeholder mechanism or waste time reconciling it against the sweep.\n\nStep-by-step proof.\n1. Grep for consumers of the MiniMax-M3-MXFP8 model_vllm block: only minimaxm3-fp8-mi355x-vllm-disagg in configs/amd-master.yaml uses framework: vllm-disagg with model: MiniMaxAI/MiniMax-M3-MXFP8.\n2. Read that recipe after this PR: 1k1k has two search-space entries — {prefill: tp=4, decode: tp=4} and {prefill: tp=4, decode: tp=4, num-worker=2}; 8k1k has two — {prefill: tp=4, decode: tp=4} and {prefill: tp=4, num-worker=2, decode: tp=4}.\n3. So every worker in every layout is TP4 — the sweep does not mix TP8 and TP4 anymore. The rationale is now the worker ratio (xP:yD), which the master-config comment block correctly describes.\n\nHow to fix. Update the parenthetical to something like (the sweep uses TP4 workers only; tp is authoritative in the master-config) — the load-bearing part (that --tensor-parallel-size 8 is a placeholder rewritten from master-config prefill.tp/decode.tp) stays the same, only the reason line needs to catch up with the retuned sweep.\n\nSeverity: nit. Doc-only staleness introduced by this PR; not merge-blocking.
There was a problem hiding this comment.
…ync / MiniMax-M3 MXFP8 MI355X vLLM 分离式：TP4 工作单元配比 + serve 参数对齐 Bump minimaxm3-fp8-mi355x-vllm-disagg image to nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9, sync the per-worker vLLM serve flags/env with the single-node minimaxm3-fp8-mi355x-vllm recipe (--moe-backend aiter, --linear-backend emulation, --max-num-batched-tokens 32768, VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1, VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT6), and retune the P/D search space to TP4 across all workers, tuning the prefill:decode worker ratio instead of TP (decode-heavy 1P/2D for 1k1k, prefill-heavy 2P/1D for 8k1k; all layouts keep prefill+decode workers <= 3 for the 3-node pool). On-box MI355X per-GPU throughput improves ~2.9x (8k1k) / ~1.5x (1k1k) vs the original 6/24 config. 中文：将 minimaxm3-fp8-mi355x-vllm-disagg 镜像升级至 nightly-2afa3f7e...，把每个 worker 的 vLLM serve 参数与环境变量与单节点 minimaxm3-fp8-mi355x-vllm recipe 对齐（--moe-backend aiter、--linear-backend emulation、--max-num-batched-tokens 32768、AITER 共享专家融合、INT6 quick all-reduce），并将 P/D 搜索空间全部改为 TP4，改为调节 prefill:decode 的 worker 配比而非 TP（1k1k 采用 decode 侧加倍的 1P/2D，8k1k 采用 prefill 侧加倍的 2P/1D； 所有布局 prefill+decode worker 数 <= 3 以适配 3 节点池）。在 MI355X 实测每 GPU 吞吐量相较 6/24 初版分别提升约 2.9 倍(8k1k)/1.5 倍(1k1k)。 Co-authored-by: Cursor <cursoragent@cursor.com>
hongxiayang
force-pushed
the
hy/disagg-mm3-mxfp8
branch
from
ada3669 to
e5aa01d
Compare
…Max-M3-MXFP8 分离式配置注释中过时的 TP 描述 The parenthetical "(the sweep mixes TP8 and TP4 layouts)" is stale: the minimaxm3-fp8-mi355x-vllm-disagg sweep now uses TP4 across all workers and varies the prefill:decode worker ratio instead of TP. Comment-only change; the placeholder-rewrite behavior it documents is unchanged. 中文：注释中 "(the sweep mixes TP8 and TP4 layouts)" 已过时—— minimaxm3-fp8-mi355x-vllm-disagg 扫描现已全部使用 TP4，改为调节 prefill:decode 的 worker 配比而非 TP。仅注释改动，占位符改写逻辑不变。 Co-authored-by: Cursor <cursoragent@cursor.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29102814212 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29102963825 |
|
/reuse-sweep-run |
|
cc @seungrokj for helping review and sign-offf. Thanks. |
seungrokj
left a comment
seungrokj
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that the single-node recipes are similar to the official vLLM recipes and/or theSGLang cookbook:
- If they are not, I have verified that a PR has been opened in vLLM recipe repo or SGLang repo and linked it below in the additional detail section:
- If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- insert any additional info here
Signed: seungrokj
❌❌❌ REJECTED ❌❌❌@seungrokj — the sign-off uses a stale copy of ✅ Check 0 (CODEOWNER): PASS — |
There was a problem hiding this comment.
/reuse-sweep-run
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that the single-node recipes are similar to the official vLLM recipes and/or theSGLang cookbook:
- If they are not, I have verified that a PR has been opened in vLLM recipe repo or SGLang repo and linked it below in the additional detail section:
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- insert any additional info here
Signed: seungrokj
✅✅✅ Verdict: PASS ✅✅✅Supersedes the earlier rejection for this SHA: the sign-off was re-posted with the current checklist template, which was the only blocker. ✅ Check 0 (CODEOWNER): PASS — |
3 participants
Summary
Retune the
minimaxm3-fp8-mi355x-vllm-disaggconfig for MiniMax-M3 MXFP8 disaggregated serving on MI355X:vllm/vllm-openai-rocm:nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9.minimaxm3-fp8-mi355x-vllmrecipe (PR [AMD] MiniMax-M3 MXFP8 MI355X vLLM: nightly + AITER-on TP4 + emulatin linear / MiniMax-M3 MXFP8 MI355X vLLM：升级 nightly + 启用 AITER TP4 + emulation linear #2003):--moe-backend aiter,--linear-backend emulation,--max-num-batched-tokens 32768,VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1,VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT6(inbenchmarks/multi_node/amd_utils/models_vllm.yaml).1P-TP4/1D-TP4(conc 1-1024) + decode-heavy1P-TP4/2D-TP4(conc 256-2048); 8k1k = balanced1P-TP4/1D-TP4(conc 1-512) + prefill-heavy2P-TP4/1D-TP4(conc 128-512). All layouts keep prefill+decode workers ≤ 3 for the 3-nodemi355x-disaggpool.Co-authored by @chunfangamd / Thanks for collaboration.
Results
On-box MI355X per-GPU throughput vs the original 6/24 disagg config (PR #1762):
More optimization to follow
Validation
test-configsweep run 29063432435: both multi-node eval jobs GREEN (GSM8K ~0.958) and 3/4 throughput jobs green. The 4th (1k1k balanced) failed mid-sweep with the server going unresponsive at conc≥32 — a transient flake (the identical config passed in run 29039061173); re-running.nofileulimit fix already onmain(both server and router containers) to avoid FD exhaustion at high concurrency / during eval.Test plan
中文说明
重新调优 MI355X 上 MiniMax-M3 MXFP8 分离式(disagg)配置
minimaxm3-fp8-mi355x-vllm-disagg：vllm/vllm-openai-rocm:nightly-2afa3f7e...。minimaxm3-fp8-mi355x-vllmrecipe（PR [AMD] MiniMax-M3 MXFP8 MI355X vLLM: nightly + AITER-on TP4 + emulatin linear / MiniMax-M3 MXFP8 MI355X vLLM：升级 nightly + 启用 AITER TP4 + emulation linear #2003）：--moe-backend aiter、--linear-backend emulation、--max-num-batched-tokens 32768、VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1、VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT6。1P-TP4/1D-TP4（并发 1-1024）+ decode 侧加倍1P-TP4/2D-TP4（并发 256-2048）；8k1k 采用均衡1P-TP4/1D-TP4（并发 1-512）+ prefill 侧加倍2P-TP4/1D-TP4（并发 128-512）。所有布局 prefill+decode worker 数 ≤ 3 以适配 3 节点池。结果
MI355X 实测每 GPU 吞吐相较 6/24 初版分离式配置（PR #1762）：8k1k 峰值约 2085 → 6003 tok/s/gpu（约 2.9 倍），1k1k 峰值约 1736 → 2586 tok/s/gpu（约 1.5 倍）。
验证
手动
test-config扫描 run 29063432435：两个多节点 eval 任务全部通过（GSM8K ~0.958），4 个吞吐任务中 3 个通过；第 4 个（1k1k 均衡）在并发≥32 时服务端中途无响应——属偶发抖动（同一配置在 run 29039061173 中通过），正在重跑。依赖已合入main的容器nofileulimit 修复（服务端与路由器容器）以避免高并发/评估时的 FD 耗尽。Made with Cursor