From 82bcee1ef493c15b57c293cf6e87b596fb0d566b Mon Sep 17 00:00:00 2001 From: Hongxia Yang Date: Fri, 10 Jul 2026 02:20:07 +0000 Subject: [PATCH 1/3] =?UTF-8?q?[AMD]=20MiniMax-M3=20MXFP8=20MI355X=20vLLM?= =?UTF-8?q?=20disagg:=20TP4=20worker-ratio=20+=20serve=20sync=20/=20MiniMa?= =?UTF-8?q?x-M3=20MXFP8=20MI355X=20vLLM=20=E5=88=86=E7=A6=BB=E5=BC=8F?= =?UTF-8?q?=EF=BC=9ATP4=20=E5=B7=A5=E4=BD=9C=E5=8D=95=E5=85=83=E9=85=8D?= =?UTF-8?q?=E6=AF=94=20+=20serve=20=E5=8F=82=E6=95=B0=E5=AF=B9=E9=BD=90?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bump minimaxm3-fp8-mi355x-vllm-disagg image to nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9, sync the per-worker vLLM serve flags/env with the single-node minimaxm3-fp8-mi355x-vllm recipe (--moe-backend aiter, --linear-backend emulation, --max-num-batched-tokens 32768, VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1, VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT6), and retune the P/D search space to TP4 across all workers, tuning the prefill:decode worker ratio instead of TP (decode-heavy 1P/2D for 1k1k, prefill-heavy 2P/1D for 8k1k; all layouts keep prefill+decode workers <= 3 for the 3-node pool). On-box MI355X per-GPU throughput improves ~2.9x (8k1k) / ~1.5x (1k1k) vs the original 6/24 config. 中文：将 minimaxm3-fp8-mi355x-vllm-disagg 镜像升级至 nightly-2afa3f7e...，把每个 worker 的 vLLM serve 参数与环境变量与单节点 minimaxm3-fp8-mi355x-vllm recipe 对齐（--moe-backend aiter、--linear-backend emulation、--max-num-batched-tokens 32768、AITER 共享专家融合、INT6 quick all-reduce），并将 P/D 搜索空间全部改为 TP4，改为调节 prefill:decode 的 worker 配比而非 TP（1k1k 采用 decode 侧加倍的 1P/2D，8k1k 采用 prefill 侧加倍的 2P/1D； 所有布局 prefill+decode worker 数 <= 3 以适配 3 节点池）。在 MI355X 实测每 GPU 吞吐量相较 6/24 初版分别提升约 2.9 倍(8k1k)/1.5 倍(1k1k)。 Co-authored-by: Cursor --- .../multi_node/amd_utils/models_vllm.yaml | 12 +- configs/amd-master.yaml | 124 +++++------------- perf-changelog.yaml | 9 ++ 3 files changed, 51 insertions(+), 94 deletions(-) diff --git a/benchmarks/multi_node/amd_utils/models_vllm.yaml b/benchmarks/multi_node/amd_utils/models_vllm.yaml index 13456c2dcb..bf391cac00 100644 --- a/benchmarks/multi_node/amd_utils/models_vllm.yaml +++ b/benchmarks/multi_node/amd_utils/models_vllm.yaml @@ -55,7 +55,13 @@ MiniMax-M3-MXFP8: # from the master-config prefill/decode tp (the sweep mixes TP8 and TP4 layouts). # --block-size 128 is mandatory (MSA sparse/index cache); text-only benchmark # so --language-model-only frees the vision encoder. gfx950 uses FP8 KV cache. - prefill_flags: "--tensor-parallel-size 8 --block-size 128 --language-model-only --kv-cache-dtype fp8 --attention-backend TRITON_ATTN --no-enable-prefix-caching --gpu-memory-utilization 0.90 --tool-call-parser minimax_m3 --reasoning-parser minimax_m3 --enable-auto-tool-choice" - decode_flags: "--tensor-parallel-size 8 --block-size 128 --language-model-only --kv-cache-dtype fp8 --attention-backend TRITON_ATTN --no-enable-prefix-caching --gpu-memory-utilization 0.90 --tool-call-parser minimax_m3 --reasoning-parser minimax_m3 --enable-auto-tool-choice" - env: "VLLM_USE_V1=1 VLLM_ROCM_USE_AITER=1 VLLM_USE_BREAKABLE_CUDAGRAPH=0 VLLM_ENGINE_READY_TIMEOUT_S=3600" + # Serve flags + env kept in sync with the single-node recipe + # benchmarks/single_node/fixed_seq_len/minimaxm3_fp8_mi355x.sh: + # --moe-backend aiter (AITER fused MoE), --linear-backend emulation, and the + # larger --max-num-batched-tokens 32768 prefill budget; plus the AITER + # router-append shared-experts fusion (self-disables under EP; the sweep is + # EP1 so it stays active) and INT6 quick all-reduce quantization. + prefill_flags: "--tensor-parallel-size 8 --block-size 128 --language-model-only --kv-cache-dtype fp8 --attention-backend TRITON_ATTN --moe-backend aiter --linear-backend emulation --max-num-batched-tokens 32768 --no-enable-prefix-caching --gpu-memory-utilization 0.90 --tool-call-parser minimax_m3 --reasoning-parser minimax_m3 --enable-auto-tool-choice" + decode_flags: "--tensor-parallel-size 8 --block-size 128 --language-model-only --kv-cache-dtype fp8 --attention-backend TRITON_ATTN --moe-backend aiter --linear-backend emulation --max-num-batched-tokens 32768 --no-enable-prefix-caching --gpu-memory-utilization 0.90 --tool-call-parser minimax_m3 --reasoning-parser minimax_m3 --enable-auto-tool-choice" + env: "VLLM_USE_V1=1 VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT6 VLLM_USE_BREAKABLE_CUDAGRAPH=0 VLLM_ENGINE_READY_TIMEOUT_S=3600" hf_dir: "models--MiniMaxAI--MiniMax-M3-MXFP8" diff --git a/configs/amd-master.yaml b/configs/amd-master.yaml index 592800e6a0..0d19b60992 100644 --- a/configs/amd-master.yaml +++ b/configs/amd-master.yaml @@ -2855,14 +2855,21 @@ minimaxm3-fp8-mi325x-vllm-mtp: - { tp: 8, ep: 8, conc-start: 256, conc-end: 256, spec-decoding: mtp } - { tp: 8, ep: 8, dp-attn: true, conc-start: 256, conc-end: 256, spec-decoding: mtp } -# MiniMax-M3 MXFP8 MI355X vLLM disaggregated (prefill/decode) smoke test on the -# day-zero ROCm image. Minimal 1 prefill (TP8) + 1 decode (TP8) at conc 1 to -# validate the MoRI-IO KV-transfer disagg pipeline end-to-end for M3. Layered on -# the MoRI-patch-removal infra (#1585). No EP (TP8 only); MoE experts are -# TP-sharded as in the single-node M3 TP8 recipe. Per-worker serve flags live in +# MiniMax-M3 MXFP8 MI355X vLLM disaggregated (prefill/decode) sweep on the +# day-zero ROCm image, over the MoRI-IO KV-transfer pipeline (MoRI-patch-removal +# infra #1585). All workers are TP4, no EP: the single-node M3 MXFP8 recipe +# (minimaxm3-fp8-mi355x-vllm, PR #2003) found plain TP4 beats both TP8 and +# TP4/EP4 on tok/s/GPU for this model on gfx950, so prefill and decode both use +# TP4 and we tune the prefill:decode worker ratio (xP:yD) instead of TP. The +# mi355x-disagg pool has 3 nodes and the launcher places one worker per node +# (NUM_NODES = xP + yD), so every layout keeps xP + yD <= 3: +# - 1P-TP4 / 1D-TP4 (2 nodes): balanced, full concurrency curve. +# - 1P-TP4 / 2D-TP4 (3 nodes): decode-heavy, for the decode-bound 1k1k tail. +# - 2P-TP4 / 1D-TP4 (3 nodes): prefill-heavy, for the prefill-bound 8k1k tail. +# Per-worker serve flags live in # benchmarks/multi_node/amd_utils/models_vllm.yaml (MiniMax-M3-MXFP8). minimaxm3-fp8-mi355x-vllm-disagg: - image: vllm/vllm-openai-rocm:nightly-556bc4e3a089378e9df2482659898192da18db15 + image: vllm/vllm-openai-rocm:nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9 model: MiniMaxAI/MiniMax-M3-MXFP8 model-prefix: minimaxm3 runner: mi355x-disagg @@ -2872,29 +2879,15 @@ minimaxm3-fp8-mi355x-vllm-disagg: disagg: true scenarios: fixed-seq-len: + # 1k1k is decode-bound (1024-token prefill, then 1024 decode steps): pair the + # balanced layout with a decode-heavy 1P/2D layout to lift high-concurrency + # throughput. Evals do not run at 1k1k, so concurrency is free to run high. - isl: 1024 osl: 1024 search-space: + # Balanced 1P TP4 + 1D TP4 (2 nodes) across the full curve. - spec-decoding: "none" conc-list: [ 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024 ] - prefill: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "PREFILL_NODES=1" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "DECODE_NODES=1" - # Asymmetric 1P TP4 + 1D TP8 (smaller prefill, full-node decode) across - # conc 1,2,4,8,16,32,64,128,256. - - spec-decoding: "none" - conc-list: [ 1, 2, 4, 8, 16, 32, 64, 128, 256 ] prefill: num-worker: 1 tp: 4 @@ -2904,14 +2897,15 @@ minimaxm3-fp8-mi355x-vllm-disagg: - "PREFILL_NODES=1" decode: num-worker: 1 - tp: 8 + tp: 4 ep: 1 dp-attn: false additional-settings: - "DECODE_NODES=1" - # Balanced half-node 1P TP4 + 1D TP4 at high conc 64,128,256,512,1024. + # Decode-heavy 1P TP4 + 2D TP4 (3 nodes): double the decode engines to + # absorb the decode-bound 1k1k tail at high concurrency. - spec-decoding: "none" - conc-list: [ 64, 128, 256, 512, 1024 ] + conc-list: [ 256, 512, 1024, 2048 ] prefill: num-worker: 1 tp: 4 @@ -2920,75 +2914,23 @@ minimaxm3-fp8-mi355x-vllm-disagg: additional-settings: - "PREFILL_NODES=1" decode: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "DECODE_NODES=1" - # 2P TP4 + 1D TP8: two half-node TP4 prefill workers (PREFILL_NODES=2) - # feeding one full-node TP8 decode, at high conc 256,512,768,1024. - - spec-decoding: "none" - conc-list: [ 256, 512, 768, 1024 ] - prefill: num-worker: 2 tp: 4 ep: 1 dp-attn: false additional-settings: - - "PREFILL_NODES=2" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "DECODE_NODES=1" - # 8k1k disagg sweep across four P/D layouts (1P TP8 + 1D TP8 conc 1..1024; - # 1P TP4 + 1D TP8 conc 1..256; 1P TP4 + 1D TP4 conc 64..1024; 2P TP4 + 1D TP8 - # conc 256..1024). The multi-node eval policy (8k1k + conc >= 16) marks one - # lm-eval on the highest-max-conc layout (TP8+TP8, eval-conc=median=128) — - # validating the M3 MoRI-IO disagg pipeline's correctness end-to-end. + - "DECODE_NODES=2" + # 8k1k is prefill-bound (8192-token prompts vs 1024 decode steps): pair the + # balanced layout with a prefill-heavy 2P/1D layout. Concurrency is capped at + # 512 so the multi-node eval policy (8k1k + conc >= 16, highest eligible conc) + # marks lm-eval at conc 512 — matching the range NVIDIA's aggregated 8k1k + # sweep tops out at and keeping the lm-eval async client stable. - isl: 8192 osl: 1024 search-space: + # Balanced 1P TP4 + 1D TP4 (2 nodes) across the full curve. - spec-decoding: "none" - conc-list: [ 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024 ] - prefill: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "PREFILL_NODES=1" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "DECODE_NODES=1" - # Asymmetric 1P TP4 + 1D TP8 (smaller prefill, full-node decode) across - # conc 1,2,4,8,16,32,64,128,256. - - spec-decoding: "none" - conc-list: [ 1, 2, 4, 8, 16, 32, 64, 128, 256 ] - prefill: - num-worker: 1 - tp: 4 - ep: 1 - dp-attn: false - additional-settings: - - "PREFILL_NODES=1" - decode: - num-worker: 1 - tp: 8 - ep: 1 - dp-attn: false - additional-settings: - - "DECODE_NODES=1" - # Balanced half-node 1P TP4 + 1D TP4 at high conc 64,128,256,512,1024. - - spec-decoding: "none" - conc-list: [ 64, 128, 256, 512, 1024 ] + conc-list: [ 1, 2, 4, 8, 16, 32, 64, 128, 256, 512 ] prefill: num-worker: 1 tp: 4 @@ -3003,10 +2945,10 @@ minimaxm3-fp8-mi355x-vllm-disagg: dp-attn: false additional-settings: - "DECODE_NODES=1" - # 2P TP4 + 1D TP8: two half-node TP4 prefill workers (PREFILL_NODES=2) - # feeding one full-node TP8 decode, at high conc 256,512,768,1024. + # Prefill-heavy 2P TP4 + 1D TP4 (3 nodes): two half-node TP4 prefill workers + # keep the single TP4 decode engine fed for the prefill-bound 8k1k tail. - spec-decoding: "none" - conc-list: [ 256, 512, 768, 1024 ] + conc-list: [ 128, 256, 512 ] prefill: num-worker: 2 tp: 4 @@ -3016,7 +2958,7 @@ minimaxm3-fp8-mi355x-vllm-disagg: - "PREFILL_NODES=2" decode: num-worker: 1 - tp: 8 + tp: 4 ep: 1 dp-attn: false additional-settings: diff --git a/perf-changelog.yaml b/perf-changelog.yaml index f4a77dd6a1..d54d4e9cd5 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -4707,3 +4707,12 @@ - "Clean the export envs" - "Enable two batch overlap" pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2093 + +- config-keys: + - minimaxm3-fp8-mi355x-vllm-disagg + description: + - "Bump image to vllm/vllm-openai-rocm:nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9" + - "Sync per-worker vLLM serve flags with the single-node minimaxm3-fp8-mi355x-vllm recipe (PR #2003): add --moe-backend aiter, --linear-backend emulation, --max-num-batched-tokens 32768; export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 and VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT6" + - "Retune the P/D search space to TP4 across all workers (drop the TP8 and TP4/EP layouts, which regress tok/s/GPU on gfx950) and tune the prefill:decode worker ratio instead. 1k1k: balanced 1P-TP4/1D-TP4 (conc 1-1024) + decode-heavy 1P-TP4/2D-TP4 (conc 256-2048). 8k1k: balanced 1P-TP4/1D-TP4 (conc 1-512) + prefill-heavy 2P-TP4/1D-TP4 (conc 128-512). All layouts keep prefill+decode workers <= 3 for the 3-node mi355x-disagg pool" + - "On-box MI355X per-GPU throughput vs the original 6/24 disagg config: 8k1k peak ~2085 -> ~6003 tok/s/gpu (~2.9x), 1k1k peak ~1736 -> ~2586 tok/s/gpu (~1.5x)" + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX From e5aa01d714b3c3dd20d4c7c6d5005e185da065a2 Mon Sep 17 00:00:00 2001 From: Hongxia Yang Date: Fri, 10 Jul 2026 04:11:10 +0000 Subject: [PATCH 2/3] =?UTF-8?q?chore:=20set=20perf-changelog=20pr-link=20t?= =?UTF-8?q?o=20#2144=20/=20=E5=B0=86=20perf-changelog=20=E7=9A=84=20pr-lin?= =?UTF-8?q?k=20=E8=AE=BE=E4=B8=BA=20#2144?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 中文：把 minimaxm3-fp8-mi355x-vllm-disagg 变更条目的 pr-link 占位符替换为实际 PR #2144。 Co-authored-by: Cursor --- perf-changelog.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/perf-changelog.yaml b/perf-changelog.yaml index d54d4e9cd5..e4e9cb50bc 100644 --- a/perf-changelog.yaml +++ b/perf-changelog.yaml @@ -4715,4 +4715,4 @@ - "Sync per-worker vLLM serve flags with the single-node minimaxm3-fp8-mi355x-vllm recipe (PR #2003): add --moe-backend aiter, --linear-backend emulation, --max-num-batched-tokens 32768; export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 and VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT6" - "Retune the P/D search space to TP4 across all workers (drop the TP8 and TP4/EP layouts, which regress tok/s/GPU on gfx950) and tune the prefill:decode worker ratio instead. 1k1k: balanced 1P-TP4/1D-TP4 (conc 1-1024) + decode-heavy 1P-TP4/2D-TP4 (conc 256-2048). 8k1k: balanced 1P-TP4/1D-TP4 (conc 1-512) + prefill-heavy 2P-TP4/1D-TP4 (conc 128-512). All layouts keep prefill+decode workers <= 3 for the 3-node mi355x-disagg pool" - "On-box MI355X per-GPU throughput vs the original 6/24 disagg config: 8k1k peak ~2085 -> ~6003 tok/s/gpu (~2.9x), 1k1k peak ~1736 -> ~2586 tok/s/gpu (~1.5x)" - pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX + pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2144 From 85187a66a9673eba9dd403a58ed72e52605303cb Mon Sep 17 00:00:00 2001 From: Hongxia Yang Date: Fri, 10 Jul 2026 15:15:11 +0000 Subject: [PATCH 3/3] =?UTF-8?q?docs:=20fix=20stale=20TP=20comment=20in=20M?= =?UTF-8?q?iniMax-M3-MXFP8=20disagg=20block=20/=20=E4=BF=AE=E6=AD=A3=20Min?= =?UTF-8?q?iMax-M3-MXFP8=20=E5=88=86=E7=A6=BB=E5=BC=8F=E9=85=8D=E7=BD=AE?= =?UTF-8?q?=E6=B3=A8=E9=87=8A=E4=B8=AD=E8=BF=87=E6=97=B6=E7=9A=84=20TP=20?= =?UTF-8?q?=E6=8F=8F=E8=BF=B0?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The parenthetical "(the sweep mixes TP8 and TP4 layouts)" is stale: the minimaxm3-fp8-mi355x-vllm-disagg sweep now uses TP4 across all workers and varies the prefill:decode worker ratio instead of TP. Comment-only change; the placeholder-rewrite behavior it documents is unchanged. 中文：注释中 "(the sweep mixes TP8 and TP4 layouts)" 已过时—— minimaxm3-fp8-mi355x-vllm-disagg 扫描现已全部使用 TP4，改为调节 prefill:decode 的 worker 配比而非 TP。仅注释改动，占位符改写逻辑不变。 Co-authored-by: Cursor --- benchmarks/multi_node/amd_utils/models_vllm.yaml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/benchmarks/multi_node/amd_utils/models_vllm.yaml b/benchmarks/multi_node/amd_utils/models_vllm.yaml index bf391cac00..9c046b4cf8 100644 --- a/benchmarks/multi_node/amd_utils/models_vllm.yaml +++ b/benchmarks/multi_node/amd_utils/models_vllm.yaml @@ -52,7 +52,8 @@ gpt-oss-120b: MiniMax-M3-MXFP8: # MiniMax-M3 MXFP8 disagg, no EP. The --tensor-parallel-size 8 below is just a # placeholder: server_vllm.sh sed-rewrites it to PREFILL_TP_SIZE/DECODE_TP_SIZE - # from the master-config prefill/decode tp (the sweep mixes TP8 and TP4 layouts). + # from the master-config prefill/decode tp (the sweep uses TP4 workers only, + # varying the prefill:decode worker ratio rather than TP). # --block-size 128 is mandatory (MSA sparse/index cache); text-only benchmark # so --language-model-only frees the vision encoder. gfx950 uses FP8 KV cache. # Serve flags + env kept in sync with the single-node recipe