Skip to content

Refresh DeepSeek-V4-Pro GB200 Dynamo vLLM stack / 更新 DeepSeek-V4-Pro GB200 Dynamo vLLM 推理栈 #2139

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Ankur-singh wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Refresh DeepSeek-V4-Pro GB200 Dynamo vLLM stack / 更新 DeepSeek-V4-Pro GB200 Dynamo vLLM 推理栈 #2139

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ec0a5f3
perf(dsv4): refresh GB200 Dynamo vLLM stack
Ankur-singh Jul 9, 2026
eccc729
chore: link performance changelog to PR #2139
Ankur-singh Jul 9, 2026
94d15da
perf(dsv4): use MessagePack for Dynamo request plane
Ankur-singh Jul 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions ...rks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/8k1k/disagg-gb200-high-tpt-megamoe.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,12 @@ name: "svf-vllm-disagg-gb200-high-tpt-megamoe"
# absorb cold-cache /mnt/numa1 model loads.
model:
path: "deepseek-v4-pro"
container: "vllm/vllm-openai:v0.20.0-ubuntu2404"
container: "vllm/vllm-openai:dsv4-megamoe-mxfp4-arm64-cu130-4ba0a72"
precision: "fp4"

dynamo:
install: true
wheel: "1.2.0.dev20260426"
wheel: "1.3.0.dev1"

@claude claude Bot Jul 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 WARNING: Incomplete stack refresh — the identity: block at the bottom of this file (lines 147–155) still pins the old stack, while model.container and dynamo.wheel here were updated.

Why it matters: identity.container.image still says vllm/vllm-openai:v0.20.0-ubuntu2404 and identity.frameworks.dynamo still says 1.2.0.dev20260426, contradicting the stack that actually runs (dsv4-megamoe-mxfp4-arm64-cu130-4ba0a72 / 1.3.0.dev1). Every other dsv4 recipe keeps these fields in sync with the top-level ones (e.g. disagg-gb200-low-latency-mtp2.yaml lines 10/131), so anything consuming this block for provenance will attribute the new benchmark numbers to the old image and wheel.

Fix: Update the identity block to match:

identity:
  model:
    repo: "deepseek-ai/DeepSeek-V4-Pro"
    revision: "0366e4e064385807ea86b088a5c6c878ff23343b"
  container:
    image: "vllm/vllm-openai:dsv4-megamoe-mxfp4-arm64-cu130-4ba0a72"
  frameworks:
    dynamo: "1.3.0.dev1"
    vllm: "0.20.0"  # confirm the vLLM version shipped in the 4ba0a72 image

identity.frameworks.vllm: "0.20.0" is also likely stale for the new dev image — please confirm what vLLM version 4ba0a72 ships and update accordingly.

This same stale identity: block exists in all five modified recipes (same lines): disagg-gb200-high-tpt-megamoe.yaml, disagg-gb200-low-latency.yaml, disagg-gb200-low-middle-curve.yaml, disagg-gb200-max-tpt-megamoe.yaml, disagg-gb200-mid-curve-megamoe.yaml. The header comment at line 13 of each file ("model.container set to vllm/vllm-openai:v0.20.0-ubuntu2404 to match nvidia-master.yaml") is also stale — worth fixing in the same pass since it documents the master-config-match invariant this PR relies on.

Fix this →

Comment on lines 20 to +26

@claude claude Bot Jul 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Documentation drift across all 5 modified GB200 recipes: both the 'Local deltas vs upstream' header comment (lines ~13-16, referencing v0.20.0-ubuntu2404) and the identity block at the bottom (container.image: v0.20.0-ubuntu2404, frameworks.dynamo: 1.2.0.dev20260426, frameworks.vllm: 0.20.0) were not updated to reflect the new model.container (dsv4-megamoe-mxfp4-arm64-cu130-4ba0a72) and dynamo.wheel (1.3.0.dev1) values. No runtime impact — nothing consumes these fields today — but each recipe is now internally inconsistent (compare with sibling disagg-gb200-high-tpt-megamoe-mtp2.yaml where identity mirrors model.container). Please refresh the header comment and identity block in all 5 files (disagg-gb200-{high-tpt-megamoe,low-latency,low-middle-curve,max-tpt-megamoe,mid-curve-megamoe}.yaml).

Extended reasoning...

What is stale

Each of the 5 modified recipes has two documentation surfaces that the PR should have kept in sync with the model.container and dynamo.wheel bumps but didn't:

1. Header comment (Local deltas vs upstream, lines ~13-16)

* model.container set to vllm/vllm-openai:v0.20.0-ubuntu2404 to
  match nvidia-master.yaml image (which the launch script registers as
  the alias key in srtslurm.yaml). Upstream variants ship either the
  non-dynamo floating tag or a sha256 pin.

This comment is the load-bearing documentation of a local-delta invariant (AGENTS.md: model.container must equal the image: entry in nvidia-master.yaml). configs/nvidia-master.yaml line 8590 was bumped in this same PR to dsv4-megamoe-mxfp4-arm64-cu130-4ba0a72, so the invariant still holds — but the tag literal in the comment now contradicts the value on the very next config line.

2. identity block at the bottom of each file

identity:
  container:
    image: "vllm/vllm-openai:v0.20.0-ubuntu2404"
  frameworks:
    dynamo: "1.2.0.dev20260426"
    vllm: "0.20.0"

Sibling recipe disagg-gb200-high-tpt-megamoe-mtp2.yaml demonstrates the intended convention: its identity.container.image mirrors its model.container. This PR breaks that pattern in all 5 modified files.

Step-by-step proof (using disagg-gb200-high-tpt-megamoe.yaml)

  1. Line 15 of the header comment says model.container set to vllm/vllm-openai:v0.20.0-ubuntu2404.
  2. Line 21 (model.container) now reads vllm/vllm-openai:dsv4-megamoe-mxfp4-arm64-cu130-4ba0a72. The comment and the value it describes disagree.
  3. Line ~152 (identity.container.image) still says vllm/vllm-openai:v0.20.0-ubuntu2404 — same old value the PR removed above.
  4. Line ~154 (identity.frameworks.dynamo) still says 1.2.0.dev20260426, while line 26 (dynamo.wheel) is now 1.3.0.dev1.
  5. The same three-way mismatch reproduces in each of the other 4 files (low-latency, low-middle-curve, max-tpt-megamoe, mid-curve-megamoe).

Impact

No runtime effect — the benchmark reads from model.container and dynamo.wheel, both of which are correct. A grep across the repo finds no Python consumer of the identity: block or of frameworks.dynamo/frameworks.vllm; identity is provenance metadata for humans and downstream tooling. So this is documentation drift only.

The concern is that any reviewer or downstream tool that trusts identity to describe "what actually ran" will now see the pre-bump image and wheel, not the new ones. And the header comment — which explicitly documents why the local pin exists — reads as if the pin is still the old tag.

How to fix

Mechanical text replacement in the 5 affected files:

  • Header comment: change v0.20.0-ubuntu2404 -> dsv4-megamoe-mxfp4-arm64-cu130-4ba0a72 in the Local deltas vs upstream block.
  • identity.container.image: change to vllm/vllm-openai:dsv4-megamoe-mxfp4-arm64-cu130-4ba0a72.
  • identity.frameworks.dynamo: change to 1.3.0.dev1.
  • identity.frameworks.vllm: the new image tag doesn't embed a clear upstream vLLM version, so pick whatever the vendored wheel inside the new image reports (or drop the field if unknown — matching sibling conventions).

Severity is nit because nothing breaks at runtime, but the fix is cheap and worth doing before merge since 5 files now carry contradictory self-description.


setup_script: vllm-container-deps.sh

Expand All @@ -48,11 +48,14 @@ infra:

frontend:
type: dynamo
enable_multiple_frontends: false
enable_multiple_frontends: true
env:
DYN_REQUEST_PLANE_CODEC: "msgpack"
backend:
type: vllm
connector: null
prefill_environment:
DYN_REQUEST_PLANE_CODEC: "msgpack"
TILELANG_CLEANUP_TEMP_FILES: "1"
VLLM_USE_NCCL_SYMM_MEM: "1"
TORCH_SYMMMEM: "NVSHMEM"
Expand All @@ -70,6 +73,7 @@ backend:
UCX_CUDA_IPC_ENABLE_MNNVL: "y"
NCCL_P2P_LEVEL: NVL
decode_environment:
DYN_REQUEST_PLANE_CODEC: "msgpack"
TILELANG_CLEANUP_TEMP_FILES: "1"
VLLM_USE_NCCL_SYMM_MEM: "1"
TORCH_SYMMMEM: "NVSHMEM"
Expand Down
10 changes: 7 additions & 3 deletions benchmarks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/8k1k/disagg-gb200-low-latency.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,12 @@ name: "svf-vllm-disagg-gb200-low-latency"
# absorb cold-cache /mnt/numa1 model loads.
model:
path: "deepseek-v4-pro"
container: "vllm/vllm-openai:v0.20.0-ubuntu2404"
container: "vllm/vllm-openai:dsv4-megamoe-mxfp4-arm64-cu130-4ba0a72"
precision: "fp4"

dynamo:
install: true
wheel: "1.2.0.dev20260426"
wheel: "1.3.0.dev1"

setup_script: vllm-container-deps.sh

Expand All @@ -47,11 +47,14 @@ infra:

frontend:
type: dynamo
enable_multiple_frontends: false
enable_multiple_frontends: true
env:
DYN_REQUEST_PLANE_CODEC: "msgpack"
backend:
type: vllm
connector: null
prefill_environment:
DYN_REQUEST_PLANE_CODEC: "msgpack"
TILELANG_CLEANUP_TEMP_FILES: "1"
VLLM_USE_NCCL_SYMM_MEM: "1"
NCCL_CUMEM_ENABLE: "1"
Expand All @@ -68,6 +71,7 @@ backend:
UCX_CUDA_IPC_ENABLE_MNNVL: "y"
NCCL_P2P_LEVEL: NVL
decode_environment:
DYN_REQUEST_PLANE_CODEC: "msgpack"
TILELANG_CLEANUP_TEMP_FILES: "1"
VLLM_USE_NCCL_SYMM_MEM: "1"
NCCL_CUMEM_ENABLE: "1"
Expand Down
10 changes: 7 additions & 3 deletions ...rks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/8k1k/disagg-gb200-low-middle-curve.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,12 @@ name: "svf-vllm-disagg-gb200-low-middle-curve"
# absorb cold-cache /mnt/numa1 model loads.
model:
path: "deepseek-v4-pro"
container: "vllm/vllm-openai:v0.20.0-ubuntu2404"
container: "vllm/vllm-openai:dsv4-megamoe-mxfp4-arm64-cu130-4ba0a72"
precision: "fp4"

dynamo:
install: true
wheel: "1.2.0.dev20260426"
wheel: "1.3.0.dev1"

setup_script: vllm-container-deps.sh

Expand All @@ -48,11 +48,14 @@ infra:

frontend:
type: dynamo
enable_multiple_frontends: false
enable_multiple_frontends: true
env:
DYN_REQUEST_PLANE_CODEC: "msgpack"
backend:
type: vllm
connector: null
prefill_environment:
DYN_REQUEST_PLANE_CODEC: "msgpack"
TILELANG_CLEANUP_TEMP_FILES: "1"
VLLM_USE_NCCL_SYMM_MEM: "1"
NCCL_CUMEM_ENABLE: "1"
Expand All @@ -69,6 +72,7 @@ backend:
UCX_CUDA_IPC_ENABLE_MNNVL: "y"
NCCL_P2P_LEVEL: NVL
decode_environment:
DYN_REQUEST_PLANE_CODEC: "msgpack"
TILELANG_CLEANUP_TEMP_FILES: "1"
VLLM_USE_NCCL_SYMM_MEM: "1"
NCCL_CUMEM_ENABLE: "1"
Expand Down
10 changes: 7 additions & 3 deletions ...arks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/8k1k/disagg-gb200-max-tpt-megamoe.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,12 @@ name: "svf-vllm-disagg-gb200-max-tpt-megamoe"
# absorb cold-cache /mnt/numa1 model loads.
model:
path: "deepseek-v4-pro"
container: "vllm/vllm-openai:v0.20.0-ubuntu2404"
container: "vllm/vllm-openai:dsv4-megamoe-mxfp4-arm64-cu130-4ba0a72"
precision: "fp4"

dynamo:
install: true
wheel: "1.2.0.dev20260426"
wheel: "1.3.0.dev1"

setup_script: vllm-container-deps.sh

Expand All @@ -48,11 +48,14 @@ infra:

frontend:
type: dynamo
enable_multiple_frontends: false
enable_multiple_frontends: true
env:
DYN_REQUEST_PLANE_CODEC: "msgpack"
backend:
type: vllm
connector: null
prefill_environment:
DYN_REQUEST_PLANE_CODEC: "msgpack"
TILELANG_CLEANUP_TEMP_FILES: "1"
VLLM_USE_NCCL_SYMM_MEM: "1"
TORCH_SYMMMEM: "NVSHMEM"
Expand All @@ -70,6 +73,7 @@ backend:
UCX_CUDA_IPC_ENABLE_MNNVL: "y"
NCCL_P2P_LEVEL: NVL
decode_environment:
DYN_REQUEST_PLANE_CODEC: "msgpack"
TILELANG_CLEANUP_TEMP_FILES: "1"
VLLM_USE_NCCL_SYMM_MEM: "1"
TORCH_SYMMMEM: "NVSHMEM"
Expand Down
10 changes: 7 additions & 3 deletions ...ks/multi_node/srt-slurm-recipes/vllm/deepseek-v4/8k1k/disagg-gb200-mid-curve-megamoe.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,12 @@ name: "svf-vllm-disagg-gb200-mid-curve-megamoe"
# absorb cold-cache /mnt/numa1 model loads.
model:
path: "deepseek-v4-pro"
container: "vllm/vllm-openai:v0.20.0-ubuntu2404"
container: "vllm/vllm-openai:dsv4-megamoe-mxfp4-arm64-cu130-4ba0a72"
precision: "fp4"

dynamo:
install: true
wheel: "1.2.0.dev20260426"
wheel: "1.3.0.dev1"

setup_script: vllm-container-deps.sh

Expand All @@ -48,11 +48,14 @@ infra:

frontend:
type: dynamo
enable_multiple_frontends: false
enable_multiple_frontends: true
env:
DYN_REQUEST_PLANE_CODEC: "msgpack"
backend:
type: vllm
connector: null
prefill_environment:
DYN_REQUEST_PLANE_CODEC: "msgpack"
TILELANG_CLEANUP_TEMP_FILES: "1"
VLLM_USE_NCCL_SYMM_MEM: "1"
TORCH_SYMMMEM: "NVSHMEM"
Expand All @@ -70,6 +73,7 @@ backend:
UCX_CUDA_IPC_ENABLE_MNNVL: "y"
NCCL_P2P_LEVEL: NVL
decode_environment:
DYN_REQUEST_PLANE_CODEC: "msgpack"
TILELANG_CLEANUP_TEMP_FILES: "1"
VLLM_USE_NCCL_SYMM_MEM: "1"
TORCH_SYMMMEM: "NVSHMEM"
Expand Down
2 changes: 1 addition & 1 deletion configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8587,7 +8587,7 @@ dsv4-fp4-b200-dynamo-vllm:
dp-attn: true

dsv4-fp4-gb200-dynamo-vllm:
image: vllm/vllm-openai:v0.20.0-ubuntu2404
image: vllm/vllm-openai:dsv4-megamoe-mxfp4-arm64-cu130-4ba0a72
model: deepseek-ai/DeepSeek-V4-Pro
model-prefix: dsv4
runner: gb200
Expand Down
9 changes: 9 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4699,3 +4699,12 @@
- "Clean the export envs"
- "Enable two batch overlap"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2093

- config-keys:
- dsv4-fp4-gb200-dynamo-vllm
description:
- "Update the Dynamo wheel from 1.2.0.dev20260426 to 1.3.0.dev1"
- "Enable multiple Dynamo frontends across all DeepSeek-V4-Pro FP4 GB200 topologies"
- "Use MessagePack encoding for the Dynamo request plane on frontends and workers"
- "Update the vLLM image from v0.20.0-ubuntu2404 to dsv4-megamoe-mxfp4-arm64-cu130-4ba0a72"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2139
Loading