[Agentx][sglang] config update

[Oasis-Git]

No description provided.

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[claude]

🟡 B200 non-DP branch now sets MEM_FRACTION_STATIC=0.90 unconditionally, but the accompanying comment justifies this as "the low-latency TP-only path (conc <= 16) runs GPU-only with no HiCache spill" — the gate is only DP_ATTENTION=false, with no CONC or KV_OFFLOADING check. The paired configs/nvidia-master.yaml still contains the non-DP HiCache row { tp: 8, kv-offloading: dram, kv-offload-backend: hicache, conc-list: [8, 16, 32] }, so that whole row (including conc=32, above the stated <=16 window) will now run at 0.90 while HiCache spills, contradicting the retained "leave the same HBM headroom … so a nearly full GPU KV cache does not OOM while HiCache is spilling" comment three lines up. Either mirror the B300 gate (elif { [ "$TP" = "8" ] || [ "$TP" = "4" ]; } && [ "${CONC:-999}" -le 16 ]; then … MEM_FRACTION_STATIC=0.90 at dsv4_fp4_b300_sglang.sh:156), gate on KV_OFFLOADING=none, or drop the non-DP HiCache row from the config.

Extended reasoning...

What the bug is

In benchmarks/single_node/agentic/dsv4_fp4_b200_sglang.sh (post-PR lines ~149–161), the MEM_FRACTION_STATIC selector now reads:

# DeepGEMM's DSv4 indexer needs a multi-GiB temporary allocation at long
# contexts. Leave the same HBM headroom used by the B300 recipe so a nearly
# full GPU KV cache does not OOM while HiCache is spilling to host memory.
# The low-latency TP-only path (conc <= 16) runs GPU-only with no HiCache
# spill, so it can take a larger static fraction for more KV headroom.
if [ "$DP_ATTENTION" = "true" ]; then
    if [ "$DEP_HIGH_CONC" = "true" ]; then
        MEM_FRACTION_STATIC=0.835
    else
        MEM_FRACTION_STATIC=0.88
    fi
else
    MEM_FRACTION_STATIC=0.90
fi

The new comment scopes the 0.90 raise to "the low-latency TP-only path (conc <= 16) runs GPU-only with no HiCache spill". But the actual gate is only DP_ATTENTION=false — it does not check CONC and does not check KV_OFFLOADING. Every non-DP invocation gets 0.90, including HiCache-enabled and conc>16 cases.

Why the retained comment matters

The comment immediately above the new one is the previous author's explicit invariant:

DeepGEMM's DSv4 indexer needs a multi-GiB temporary allocation at long contexts. Leave the same HBM headroom used by the B300 recipe so a nearly full GPU KV cache does not OOM while HiCache is spilling to host memory.

That is exactly why 0.88 was chosen. Raising to 0.90 for the non-DP + HiCache case removes ~2% of 192 GB ≈ ~3.8 GB of HBM headroom the previous author intentionally reserved for the DSv4 indexer while HiCache spills.

Step-by-step proof from the paired config

The PR also modifies configs/nvidia-master.yaml (line ~14088) but keeps a non-DP HiCache row:

- { tp: 8, kv-offloading: dram, kv-offload-backend: hicache, conc-list: [8, 16, 32] }

Trace the runner for conc=32 from that row:

1. Runner sets DP_ATTENTION=false, KV_OFFLOADING=dram, KV_OFFLOAD_BACKEND=hicache, CONC=32, TP=8.
2. DP_ATTENTION=true branch → false; falls into the else (line ~160): MEM_FRACTION_STATIC=0.90.
3. require_agentic_kv_offload_backend hicache is true → --enable-hierarchical-cache is passed, HiCache is active and will spill.
4. Result: non-DP + HiCache spilling + MEM_FRACTION_STATIC=0.90, which is exactly the case the retained comment says must run at 0.88. Additionally conc=32 exceeds the "conc <= 16" bound the new comment claims. Same failure mode for conc=8 and conc=16 in that row (still HiCache-enabled).

Pre-PR that same run used 0.88; post-PR it uses 0.90.

Cross-check against the sibling script

The companion benchmarks/single_node/agentic/dsv4_fp4_b300_sglang.sh (line ~156) does this correctly in the same PR:

elif { [ "$TP" = "8" ] || [ "$TP" = "4" ]; } && [ "${CONC:-999}" -le 16 ]; then
    …
    MEM_FRACTION_STATIC=0.90

So the author knew the correct pattern — gate 0.90 on the low-latency window and keep 0.88 for the general non-DP path — it's just missing on B200.

Fix

Any one of these resolves the code/comment/config three-way inconsistency:

1. Mirror the B300 gate on the B200 non-DP branch: only raise to 0.90 when { TP=8 || TP=4 } && CONC <= 16 (or equivalently gate on KV_OFFLOADING=none); keep 0.88 for non-DP + HiCache.
2. Drop the non-DP HiCache row { tp: 8, kv-offloading: dram, kv-offload-backend: hicache, conc-list: [8, 16, 32] } from configs/nvidia-master.yaml, so the 0.90 non-DP path never coincides with HiCache spill.
3. If 0.90 has been empirically validated safe for non-DP + HiCache on B200 (with the DSv4 indexer's multi-GiB indexer alloc), delete the retained "leave the same HBM headroom … while HiCache is spilling" comment and reword the new one so it no longer claims "conc <= 16 … no HiCache spill" as the justification.

Severity note

Marking nit because the ~3.8 GB delta may or may not trigger an actual OOM at long-context HiCache spill in practice — full-sweep-fail-fast validation will catch it if it does. But the code violates its own documented invariant, and the B300 mirror in the same PR shows the intended shape; the fix is a two-line gate change.

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29081167653
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29081167653

[github-actions]

see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29081167653
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=29081167653