[AMD][AgentX] Kimi-K2.7-Code FP4 MI355X agentX vLLM#2126
Conversation
Adds the kimik2.7 fp4 agentic trace-replay recipe (kimik2.7_fp4_mi355x.sh, KV_OFFLOADING none + dram/lmcache) and the kimik2.7-fp4-mi355x-vllm-agentic config key on cluster:mi355x-amds (tp4 conc32, none + lmcache). Recipe clones LMCache to /opt/lmcache-src so the CI checkout never trips over root-owned build artifacts.
The none case previously passed --no-enable-prefix-caching, which disables the on-GPU prefix cache entirely, not just DRAM offload. That crippled the no-offload baseline (no reuse even within HBM) and made lmcache look artificially good. Leave prefix caching at vLLM's default (ON) so none is an honest GPU-only-KV baseline, apples-to-apples vs lmcache. Matches the kimik2.5 / dsv4 agentic recipes.
Set the kimik2.7-fp4-mi355x-vllm-agentic sweep to conc [1,4,8,16,32,48] for both none and dram/LMCache at TP4, dram-utilization 0.80. Add the corresponding perf-changelog entry (pr-link placeholder to fill on PR open).
Previous commit accidentally dropped perf-changelog.yaml (blob too large for inline arg). Restore it with the kimik2.7-fp4-mi355x-vllm-agentic entry appended.
hyukjlee
requested review from
1am9trash,
billishyahao,
chunfangamd,
seungrokj and
yctseng0211
as code owners
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "Add Kimi-K2.7-Code-MXFP4 agentic-coding config on MI355X (cluster:gbt350docker->mi355x-amds) via vLLM v0.24.0, TP4" | ||
| - "Sweep KV-offloading none vs dram/LMCache at conc [1,4,8,16,32,42]; LMCache MP server + LMCacheMPConnector, dram-utilization 0.80" | ||
| - "Recipe benchmarks/single_node/agentic/kimik2.7_fp4_mi355x.sh clones LMCache to container-local /opt/lmcache-src (pinned) so CI checkout never trips over root-owned build artifacts" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2126 |
There was a problem hiding this comment.
🔴 The new changelog entry's pr-link is literally .../pull/PLACEHOLDER (perf-changelog.yaml:4644). utils/validate_perf_changelog.py only accepts the canonical link (.../pull/2126 for this PR) or the XXX sentinel — PLACEHOLDER is neither, so the changelog-gate CI step in .github/workflows/run-sweep.yml will raise ChangelogValidationError and fail this PR. Fix: replace PLACEHOLDER with 2126 (or XXX) before merge.
Extended reasoning...
The new changelog entry ends with pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PLACEHOLDER (perf-changelog.yaml:4644). Every surrounding entry uses a real PR number (2113, 2114, 2115), so this is unambiguously a leftover placeholder that was never rewritten with the real PR id.
This is not just cosmetic. utils/validate_perf_changelog.py defines the only accepted placeholder set:
PR_LINK_PLACEHOLDERS = {"XXX", "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX"}and validate_added_pr_link (line 144) enforces link in PR_LINK_PLACEHOLDERS or link == expected where expected == f"https://github.com/SemiAnalysisAI/InferenceX/pull/{pr_number}". The literal string "PLACEHOLDER" (or the full .../pull/PLACEHOLDER URL) is in neither set, so the validator will raise ChangelogValidationError on this PR.
That validator is wired into CI via .github/workflows/run-sweep.yml:102 (python3 utils/validate_perf_changelog.py), so the changelog-gate step will actively fail this PR until the link is fixed. utils/prepare_perf_changelog_merge.py performs the same check on the reuse-merge path, so even if CI were bypassed, the merge-with-reuse tooling would refuse the entry.
Step-by-step proof:
- On PR-2126 CI, the changelog gate invokes
validate_perf_changelog.pywith--pr-number 2126. - It parses head
perf-changelog.yaml, finds the newly appendedkimik2.7-fp4-mi355x-vllm-agenticentry. - It calls
validate_added_pr_link("https://github.com/SemiAnalysisAI/InferenceX/pull/PLACEHOLDER", 2126). expected = "https://github.com/SemiAnalysisAI/InferenceX/pull/2126". The link is neitherexpectednor a member ofPR_LINK_PLACEHOLDERS(only"XXX"/".../pull/XXX").raise ChangelogValidationError(...)→ the CI job exits non-zero → the PR check goes red.- If it somehow merged, the on-site changelog would contain a permanently dead hyperlink pointing at
/pull/PLACEHOLDER, breaking the pr-link traceability contract every surrounding entry follows.
Fix: change the value at perf-changelog.yaml:4644 to https://github.com/SemiAnalysisAI/InferenceX/pull/2126 (or the accepted XXX placeholder if you'd rather punt until squash-merge time). One-token change.
| # Variant of kimik2.7_fp4_mi355x.sh that supports TWO KV configs: | ||
| # KV_OFFLOADING=none -> GPU KV only | ||
| # KV_OFFLOADING=dram KV_OFFLOAD_BACKEND=lmcache -> LMCache MP server + connector |
There was a problem hiding this comment.
🟡 Header comment at line 7 reads "Variant of kimik2.7_fp4_mi355x.sh that supports TWO KV configs" — but this file is kimik2.7_fp4_mi355x.sh. The intended reference is almost certainly kimik2.5_fp4_mi355x.sh (the sibling this recipe is derived from — same env-var contract, install flow, and vLLM args, only the LMCache branch is new). Zero runtime impact; one-line doc fix.
Extended reasoning...
What the bug is. The recipe's header docstring at benchmarks/single_node/agentic/kimik2.7_fp4_mi355x.sh:7 reads:\n\n\n# Variant of kimik2.7_fp4_mi355x.sh that supports TWO KV configs:\n\n\nThat filename is the same file the comment lives in — a script cannot be a variant of itself. This is a copy-paste error from the sibling recipe.\n\nWhich file it should reference. The obvious parent is benchmarks/single_node/agentic/kimik2.5_fp4_mi355x.sh: it shares the same env-var contract (MODEL, TP, CONC, KV_OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR, DURATION, EP_SIZE), the same benchmark_lib.sh sourcing, the same hf download / amd-quark install flow, and an almost-identical vLLM launch invocation. The only structural addition in the 2.7 script is the lmcache branch of the case "$OFFLOAD_MODE" dispatch — which is exactly what the "supports TWO KV configs" clarifier is describing. So the intended comment is "Variant of kimik2.5_fp4_mi355x.sh".\n\nStep-by-step proof. (1) The path in the diff header is benchmarks/single_node/agentic/kimik2.7_fp4_mi355x.sh. (2) The comment at line 7 says "Variant of kimik2.7_fp4_mi355x.sh". (3) Those two strings are byte-identical, so the comment is a self-reference. (4) The sibling kimik2.5_fp4_mi355x.sh exists on main and only supports the none offload mode (its search-space includes kv-offload-backend: native but the recipe itself pre-dates LMCache), which matches the "supports TWO KV configs" clarifier exactly (2.7 adds the lmcache branch as a second KV config).\n\nImpact. None at runtime — this is a comment. It's a minor readability defect for future maintainers landing on this file who'd otherwise trace back to the kimik2.5 recipe to understand why the code is structured this way.\n\nSuggested fix. One-line change: Variant of kimik2.7_fp4_mi355x.sh → Variant of kimik2.5_fp4_mi355x.sh. Alternately, drop the "Variant of …" preamble entirely, since the two-KV-configs description stands on its own.\n\nSeverity. Nit. Pure documentation, no behavior change, not blocking merge — worth a one-line touch-up while the author is in the file.
| - config-keys: | ||
| - kimik2.7-fp4-mi355x-vllm-agentic | ||
| description: | ||
| - "Add Kimi-K2.7-Code-MXFP4 agentic-coding config on MI355X (cluster:gbt350docker->mi355x-amds) via vLLM v0.24.0, TP4" |
There was a problem hiding this comment.
🟡 The perf-changelog description references cluster:gbt350docker->mi355x-amds, but gbt350docker doesn't exist anywhere else in the repo — the actual runner declared in configs/amd-master.yaml is simply cluster:mi355x-amds. Looks like stale copy-paste; suggest dropping the parenthetical or rewriting it as (runner: cluster:mi355x-amds).
Extended reasoning...
What
Line 4641 of perf-changelog.yaml reads:
Add Kimi-K2.7-Code-MXFP4 agentic-coding config on MI355X (cluster:gbt350docker->mi355x-amds) via vLLM v0.24.0, TP4
The token gbt350docker is not a known cluster/runner identifier in this repo. A grep across the tree shows it appears only on this new changelog line — nowhere in configs/runners.yaml, configs/amd-master.yaml, any launcher, or any other doc.
Why it's wrong
The actual runner declared for kimik2.7-fp4-mi355x-vllm-agentic in configs/amd-master.yaml is:
runner: cluster:mi355x-amdsNo gbt350docker migration ever happened for this recipe; the AMD MI355X agentic runner is cluster:mi355x-amds (same as kimik2.5-fp4-mi355x-vllm-agentic, dsv4-fp4-mi355x-vllm-agentic, and qwen3.5-fp8-mi355x-sglang-agentic-hicache in the same file). The cluster:X->Y arrow format also appears nowhere else in perf-changelog.yaml — it isn't a project convention. This reads like stale template text carried over from an unrelated draft.
Impact
Zero runtime impact — perf-changelog.yaml is documentation, description strings are free-form, and the launcher/runner routing is driven by configs/amd-master.yaml (which is correct). The affected artifact is the public perf-changelog page: this string will render there and misrepresent the sweep provenance to any reader trying to trace where the numbers came from.
Suggested fix
Either drop the parenthetical entirely:
- "Add Kimi-K2.7-Code-MXFP4 agentic-coding config on MI355X via vLLM v0.24.0, TP4"or make it accurate:
- "Add Kimi-K2.7-Code-MXFP4 agentic-coding config on MI355X (runner: cluster:mi355x-amds) via vLLM v0.24.0, TP4"Proof (step-by-step)
configs/amd-master.yamlkimik2.7-fp4-mi355x-vllm-agentic:block declaresrunner: cluster:mi355x-amds.grep -rn gbt350docker .returns exactly one hit —perf-changelog.yaml:4641— and no other file in the repo defines or references that cluster name.- Every peer MI355X agentic recipe in the same file uses
runner: cluster:mi355x-amds; there is no fleet, cluster, or SKU by the namegbt350dockerinconfigs/. - Therefore the changelog's
cluster:gbt350docker->mi355x-amdsdescribes routing that doesn't exist — it is a copy-paste artifact.
seungrokj
changed the title
- Frontier-only sweep: none [1,4,8] (low conc, fits GPU HBM) + lmcache [16,32] (above the KV cliff). Drops dominated/cliff points per review. - Fix recipe header: 'Variant of kimik2.5_fp4_mi355x.sh' (was self-referential) - perf-changelog: correct runner (cluster:mi355x-amds) and conc lists
# Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29003473063 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=29030417289 |
cquil11
added
the
agentx
ChatGPT Codex Connector
|
/reuse-sweep-run |
There was a problem hiding this comment.
thanks for the contribution @seungrokj
can u please update recipes.vllm.ai with ur new configs?
vllm recipes doesn't currently support offloading tho the inferact guys said they are open to contributions to support mooncake store, lmcache, vllm native offloading support in vllm recipes
as in sglang cookbook recipes , they also have support for cpu offloading recipes
4 participants
MI355X using vLLM
benchmarks/single_node/agentic/kimik2.7_fp4_mi355x.shwithtwo KV offload backends:
none(GPU-only KV, prefix caching on) andlmcache(LMCache MP server + LMCacheMPConnector)kimik2.7-fp4-mi355x-vllm-agentic, TP4, conc sweep[1,4,8,16,32,42]acrossnoneandlmcache, dram-utilization 0.80vllm/vllm-openai-rocm:v0.24.0amd/Kimi-K2.7-Code-MXFP4