[AMD][AgentX] DeepSeek V4 multi-node Agentic framework and recipe / DeepSeek V4 多节点 Agentic 框架与配方#2127
Conversation
ichbinblau
requested review from
billishyahao,
chunfangamd,
seungrokj and
yctseng0211
as code owners
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
seungrokj
changed the title
| - { tp: 8, ep: 8, dp-attn: true, kv-offloading: dram, kv-offload-backend: native, conc-list: [24, 32, 36, 40, 44, 48, 52, 56, 60, 64, 72, 80, 96] } | ||
|
|
||
| dsv4-fp4-mi355x-sglang-disagg-agentic-hicache: | ||
| image: rocm/sgl-dev:v0.5.14-rocm720-mi35x-20260706 |
There was a problem hiding this comment.
We have bumped the image to lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260706
There was a problem hiding this comment.
Yes we will switch to lmsysorg/sglang-rocm:v0.5.14-rocm720-mi35x-20260710 where included all 4 patches
|
The patches in this PR are derived from the following upstream SGLang PRs. |
ichbinblau
force-pushed
the
amd/agentx-v1.0-rebase
branch
from
3f0a1f9 to
da1084d
Compare
cquil11
added
the
agentx
ChatGPT Codex Connector
cquil11
force-pushed
the
amd/agentx-v1.0-rebase
branch
2 times, most recently
from
86da4cc to
a34c27e
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29052264020 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29052670710 |
cquil11
force-pushed
the
amd/agentx-v1.0-rebase
branch
from
a34c27e to
b0088c8
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29054064553 |
cquil11
force-pushed
the
amd/agentx-v1.0-rebase
branch
from
b0088c8 to
4e557d0
Compare
cquil11
changed the title
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29056644716 |
cquil11
force-pushed
the
amd/agentx-v1.0-rebase
branch
from
4e557d0 to
fce18af
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29057082376 |
cquil11
force-pushed
the
amd/agentx-v1.0-rebase
branch
from
fce18af to
b6a9877
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29057275526 |
| ) | ||
| kv_manager._staging_handler = self.staging_handler | ||
|
|
||
| + def _agree_and_order_queue(self) -> Tuple[List["DecodeRequest"], List["DecodeRequest"]]: |
There was a problem hiding this comment.
We will not accept any patches. I have removed all patches that have since been merged into upstream sglang main. Please update nightly image tomorrow.
However, this patch remains. Please remove it / open upstream pr.
There was a problem hiding this comment.
We will not accept any patches. I have removed all patches that have since been merged into upstream sglang main. Please update nightly image tomorrow.
However, this patch remains. Please remove it / open upstream pr.
This patch file has not been used as its caller func has been commented out. I have remove it.
|
@ichbinblau Thank you for this awesome work. Please see the changes I requested. Also, please re run full-sweep-enabled after making these changes. +viz @chunfangamd |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29065983132 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29066428461 |
ichbinblau
force-pushed
the
amd/agentx-v1.0-rebase
branch
from
e9aa46a to
55a7983
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29066543853 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29066685174 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29067272731 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29078166951 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29078166951 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29089300938 |
| # which is consistent across ranks and is the only set finalize_bootstrap acts on. | ||
| # --------------------------------------------------------------------------- | ||
| patch_gluon_pa_mqa_logits_instr_shape() { | ||
| patch_disagg_prefill_bootstrap_desync() { |
There was a problem hiding this comment.
I'm going to allow this for now as we are early days of agentX and sglang and vllm also have some scheduling patches, but I want to open an issue for you guys to track this and get an upstream fix in sglang. @billishyahao @ichbinblau
cc @ichbinblau
cquil11
force-pushed
the
amd/agentx-v1.0-rebase
branch
from
e988a79 to
f409c36
Compare
chunfangamd
left a comment
chunfangamd
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse. Approve for day 0
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29089300938
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/29089300938
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that the single-node recipes are similar to the official vLLM recipes and/or theSGLang cookbook:
- If they are not, I have verified that a PR has been opened in vLLM recipe repo or SGLang repo and linked it below in the additional detail section:
- If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- insert any additional info here
We will need a SGLang cookbook for the recipe. I approve it for the day 0 support for the AgentX
Signed: @chunfangamd
❌❌❌ REJECTED ❌❌❌@chunfangamd — blocking: no green, executed sweep/eval exists on any commit currently in this PR, so the reuse gate ( ❌ Check 1 (sweep+evals on in-PR commit): FAIL — the PR's only commit |
cquil11
force-pushed
the
amd/agentx-v1.0-rebase
branch
from
f409c36 to
59989c2
Compare
…ntry Resolve the perf-changelog.yaml tail conflict with #2127 per KLAUD_DEBUG.md §1.1: take main's changelog wholesale and re-append this PR's entry at the end, with pr-link updated from the XXX placeholder to #2153. 中文：将 origin/main 合并到 lmcache/agentix，按 KLAUD_DEBUG.md §1.1 处理 perf-changelog.yaml 末尾冲突：整体采用 main 的版本并在末尾重新追加本 PR 的 条目，同时把 pr-link 占位符更新为 #2153。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reverts #2127. It live-patches SGLang (disaggregation/prefill.py resolve_waiting_queue_bootstrap) during setup, which #2155 forbids without a docs/waiver/ entry. 中文：回滚 #2127，因其在 setup 阶段对 SGLang（disaggregation/prefill.py 的 resolve_waiting_queue_bootstrap）进行运行时打补丁，违反 #2155（未经 docs/waiver/ 豁免不得修改推理引擎）。
7 participants
Summary
Add end-to-end agentic benchmarking for DeepSeek-V4-Pro FP4 on MI355X using multi-node disaggregated SGLang and HiCache host-DRAM KV offloading.
This PR now focuses on the recipe and runtime implementation:
Dependency
Depends on #2141, which separately enables
kv-offloadingandkv-offload-backendpropagation for multi-node Agentic sweep matrices. The sharedutils/matrix_logicchanges are intentionally not included in this PR.Until #2141 is merged, the configuration in this PR declares
kv-offloading: dramandkv-offload-backend: hicache, but currentmainstill forces multi-node Agentic matrix entries tokv-offloading: none.Patch policy
Patch and overlay code corresponding to merged upstream SGLang changes has been removed. The remaining patch artifacts are limited to code without a merged upstream equivalent, including the disaggregated prefill bootstrap desynchronization fix and the reference-only decode TP queue agreement patch.
Validation
bash -n.configs/amd-master.yamlandperf-changelog.yamlparse successfully.PYTHONPATH=. pytest -q utils/agentic/aggregation/test_process_agentic_result.py— 26 passed.git diff --checkpasses.中文说明
本 PR 为 MI355X 上的 DeepSeek-V4-Pro FP4 增加端到端 Agentic 基准测试能力，采用多节点 SGLang 分离式推理，并通过 HiCache 使用主机 DRAM 进行 KV 缓存卸载。
本 PR 当前仅聚焦配方和运行时实现：
依赖关系
本 PR 依赖 #2141。#2141 单独为多节点 Agentic 扫描矩阵增加
kv-offloading和kv-offload-backend的透传能力。本 PR 已移除所有共享的utils/matrix_logic改动。在 #2141 合并之前，本 PR 的配置虽然声明了
kv-offloading: dram和kv-offload-backend: hicache，但当前main仍会把多节点 Agentic 矩阵项强制转换为kv-offloading: none。补丁策略
与已合并 SGLang 上游改动对应的补丁和 overlay 代码均已移除。当前仅保留尚无已合并上游等价实现的本地补丁，包括分离式 prefill bootstrap 不同步修复，以及仅作参考、默认不启用的 decode TP queue agreement 补丁。
验证
Shell 和 Slurm 脚本均通过
bash -n；configs/amd-master.yaml与perf-changelog.yaml可正常解析；Agentic 聚合测试 26 项全部通过；相关 Python 文件编译检查和git diff --check均通过。