[AMD][AgentX] MINIMAX-M3 FP4 MI355X agentX vLLM #2129
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,206 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
| set -x | ||
| # Agentic trace replay benchmark for Minimax-M3 FP4 on MI355X using vLLM. | ||
| # | ||
| # Required env vars: | ||
| # MODEL, TP, CONC, OFFLOADING, TOTAL_CPU_DRAM_GB, RESULT_DIR | ||
| # | ||
| # OFFLOADING values: | ||
| # none - vLLM GPU KV only. | ||
| # cpu - vLLM native CPU offload. | ||
| # lmcache - LMCache MP server + vLLM LMCacheMPConnector. | ||
| source "$(dirname "$0")/../../benchmark_lib.sh" | ||
| check_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION | ||
| echo "MODEL=$MODEL TP=$TP CONC=$CONC KV_OFFLOADING=$KV_OFFLOADING TOTAL_CPU_DRAM_GB=$TOTAL_CPU_DRAM_GB RESULT_DIR=$RESULT_DIR DURATION=$DURATION EP_SIZE=$EP_SIZE DP_ATTENTION=$DP_ATTENTION" | ||
| PORT=${PORT:-8888} | ||
| DURATION=${DURATION:-1800} | ||
| EP_SIZE=${EP_SIZE:-1} | ||
| if [[ -n "${SLURM_JOB_ID:-}" ]]; then | ||
| echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}" | ||
| fi | ||
| # ROCR/HIP visibility for vLLM 0.14+ | ||
| if [ -n "${ROCR_VISIBLE_DEVICES:-}" ]; then | ||
| export HIP_VISIBLE_DEVICES="$ROCR_VISIBLE_DEVICES" | ||
| fi | ||
| if [[ "$MODEL" != /* ]]; then hf download "$MODEL"; fi | ||
| rocm-smi || true | ||
| amd-smi || true | ||
| # `hf download` creates the target dir if missing and is itself idempotent. | ||
| # When MODEL_PATH is unset (stand-alone runs), fall back to the HF_HUB_CACHE | ||
| # Either way, MODEL_PATH is what the server is launched with. | ||
| if [[ -n "${MODEL_PATH:-}" ]]; then | ||
| if [[ ! -d "$MODEL_PATH" || -z "$(ls -A "$MODEL_PATH" 2>/dev/null)" ]]; then | ||
| hf download "$MODEL" --local-dir "$MODEL_PATH" | ||
| fi | ||
| else | ||
| hf download "$MODEL" | ||
| export MODEL_PATH="$MODEL" | ||
| fi | ||
| resolve_trace_source | ||
| install_agentic_deps | ||
| # ---- Server config ---------------------------------------------------------- | ||
| SERVER_LOG="$RESULT_DIR/server.log" | ||
| LMCACHE_LOG="$RESULT_DIR/lmcache_server.log" | ||
| mkdir -p "$RESULT_DIR" | ||
| OFFLOAD_ARGS=(--no-enable-prefix-caching) | ||
| # ---- Lmcache config ---------------------------------------------------------- | ||
| LMCACHE_PID="" | ||
| cleanup_lmcache_server() { | ||
| if [[ -n "$LMCACHE_PID" ]] && kill -0 "$LMCACHE_PID" 2>/dev/null; then | ||
| kill "$LMCACHE_PID" 2>/dev/null || true | ||
| wait "$LMCACHE_PID" 2>/dev/null || true | ||
| fi | ||
| } | ||
| trap cleanup_lmcache_server EXIT | ||
| wait_for_lmcache_ready() { | ||
| { set +x; } 2>/dev/null | ||
| local attempts="${LMCACHE_READY_ATTEMPTS:-120}" | ||
| local tail_pid="" | ||
| while [ ! -f "$LMCACHE_LOG" ]; do | ||
| if [[ -n "$LMCACHE_PID" ]] && ! kill -0 "$LMCACHE_PID" 2>/dev/null; then | ||
| echo "LMCache server died before creating log file. Exiting." >&2 | ||
| exit 1 | ||
| fi | ||
| sleep 1 | ||
| done | ||
| tail -f -n +1 "$LMCACHE_LOG" & | ||
| tail_pid=$! | ||
| for ((i = 1; i <= attempts; i++)); do | ||
| if curl --output /dev/null --silent --fail "http://127.0.0.1:${LMCACHE_HTTP_PORT}/healthcheck"; then | ||
| kill "$tail_pid" 2>/dev/null || true | ||
| wait "$tail_pid" 2>/dev/null || true | ||
| return 0 | ||
| fi | ||
| if [[ -n "$LMCACHE_PID" ]] && ! kill -0 "$LMCACHE_PID" 2>/dev/null; then | ||
| echo "LMCache server died before becoming healthy. Log follows:" >&2 | ||
| kill "$tail_pid" 2>/dev/null || true | ||
| wait "$tail_pid" 2>/dev/null || true | ||
| cat "$LMCACHE_LOG" >&2 || true | ||
| exit 1 | ||
| fi | ||
| sleep 1 | ||
| done | ||
| echo "Timed out waiting for LMCache server healthcheck. Log follows:" >&2 | ||
| kill "$tail_pid" 2>/dev/null || true | ||
| wait "$tail_pid" 2>/dev/null || true | ||
| cat "$LMCACHE_LOG" >&2 || true | ||
| exit 1 | ||
| } | ||
| case "$KV_OFFLOAD_BACKEND" in | ||
| lmcache) | ||
| unset VLLM_USE_SIMPLE_KV_OFFLOAD | ||
| git clone https://github.com/LMCache/LMCache.git | ||
| cd LMCache | ||
| pip install -r requirements/build.txt | ||
| CXX=hipcc BUILD_WITH_HIP=1 pip install -e . --no-build-isolation | ||
| cd .. | ||
|
Comment on lines
+91
to
+97
Contributor
There was a problem hiding this comment. 🟡 The LMCache install at minimaxm3_fp4_mi355x.sh:93 clones Extended reasoning...What the bug isAt lmcache)
unset VLLM_USE_SIMPLE_KV_OFFLOAD
git clone https://github.com/LMCache/LMCache.git
cd LMCache
pip install -r requirements/build.txt
CXX=hipcc BUILD_WITH_HIP=1 pip install -e . --no-build-isolation
cd ..There is no Impact — reproducibilityThe entire purpose of a recipe in this repo is to produce numerically comparable perf points that get ingested into - { tp: 4, kv-offloading: dram, kv-offload-backend: lmcache, conc-list: [1, 4, 8, 16, 32] }i.e. every configuration this recipe measures goes through the Impact — idempotencyThe script begins with Regression from the pre-existing patternThe sibling NVIDIA recipe agentic_pip_install --quiet --no-cache-dir lmcacheThis was the established pattern for installing LMCache in an agentic recipe. Switching to an unpinned HEAD clone is a strict regression. (The MI355X build does need a HIP source build — no ROCm wheel on PyPI — so falling back to a Step-by-step proof of the non-idempotency failure
(Whether the workspace persists across configs depends on the launcher; the current launch_mi355x-amds.sh mounts $GITHUB_WORKSPACE at /workspace, so in practice fresh CI containers usually paper over this — but any local re-run or workspace-reuse path trips the failure.) FixA one-line change, either agentic_pip_install --quiet --no-cache-dir lmcache==<version>or, keeping the HIP source build: rm -rf LMCache
git clone --depth 1 --branch <tag-or-sha> https://github.com/LMCache/LMCache.gitSeverityNit — a clean container run in CI passes, the numerical result is some valid measurement, and there is no user-visible crash on the green path. But this is directly identified as an anti-pattern relative to the sibling recipes in the same directory and it silently erodes benchmark comparability over time, so worth calling out before merge. |
||
| python3 -c "import lmcache.integration.vllm.lmcache_mp_connector" >/dev/null | ||
| # Let the external MP server own the full CPU KV pool so vLLM does not | ||
| # split --kv-offloading-size across TP ranks through the integrated | ||
| # LMCache backend. | ||
| TOTAL_CPU_DRAM_GB="${TOTAL_CPU_DRAM_GB:-3000}" | ||
| TOTAL_CPU_DRAM_PARTITION_GB="${TOTAL_CPU_DRAM_PARTITION_GB:-${TOTAL_CPU_DRAM_GB}}" | ||
| LMCACHE_HOST="${LMCACHE_HOST:-127.0.0.1}" | ||
| LMCACHE_PORT="${LMCACHE_PORT:-5555}" | ||
| LMCACHE_HTTP_PORT="${LMCACHE_HTTP_PORT:-8080}" | ||
| # LMCacheMPConnector concatenates lmcache.mp.host and port into the | ||
| # ZMQ endpoint. Bind the server to a raw host, but pass the connector a | ||
| # ZMQ-style host string. | ||
| LMCACHE_CONNECT_HOST="${LMCACHE_CONNECT_HOST:-tcp://$LMCACHE_HOST}" | ||
| LMCACHE_L1_SIZE_GB="${LMCACHE_L1_SIZE_GB:-$((TOTAL_CPU_DRAM_PARTITION_GB))}" | ||
| LMCACHE_L1_INIT_SIZE_GB="${LMCACHE_L1_INIT_SIZE_GB:-20}" | ||
| # LMCache read locks are leases on chunks that lookup has promised | ||
| # vLLM can retrieve. The default 300s TTL is too short for this | ||
| # long-context agentic queue: TP8/conc32 can spend >300s between | ||
| # lookup and retrieve while GPU KV is saturated, which leaves the | ||
| # object present in L1 but no longer readable. Keep the 2.5 TB pool | ||
| # size unchanged and only extend the lookup-to-retrieve lease. | ||
| LMCACHE_L1_READ_TTL_SECONDS="${LMCACHE_L1_READ_TTL_SECONDS:-7200}" | ||
| # (srok) check 256 vs 32 | ||
| #LMCACHE_CHUNK_SIZE="${LMCACHE_CHUNK_SIZE:-32}" | ||
| LMCACHE_CHUNK_SIZE="${LMCACHE_CHUNK_SIZE:-256}" | ||
| LMCACHE_MAX_WORKERS="${LMCACHE_MAX_WORKERS:-$((TP * 2))}" | ||
| export PYTHONHASHSEED="${PYTHONHASHSEED:-0}" | ||
| export LMCACHE_BLOCKING_TIMEOUT_SECS=60 | ||
| echo "Starting LMCache MP server..." | ||
| LMCACHE_CMD=( | ||
| lmcache server | ||
| --host "$LMCACHE_HOST" | ||
| --port "$LMCACHE_PORT" | ||
| --http-host "$LMCACHE_HOST" | ||
| --http-port "$LMCACHE_HTTP_PORT" | ||
| --l1-size-gb "$LMCACHE_L1_SIZE_GB" | ||
| --l1-init-size-gb "$LMCACHE_L1_INIT_SIZE_GB" | ||
| --l1-read-ttl-seconds "$LMCACHE_L1_READ_TTL_SECONDS" | ||
| --chunk-size "$LMCACHE_CHUNK_SIZE" | ||
| --max-workers "$LMCACHE_MAX_WORKERS" | ||
| --eviction-policy LRU | ||
| ) | ||
| printf '%q ' "${LMCACHE_CMD[@]}" > "$RESULT_DIR/lmcache_command.txt" | ||
| printf '\n' >> "$RESULT_DIR/lmcache_command.txt" | ||
| "${LMCACHE_CMD[@]}" > "$LMCACHE_LOG" 2>&1 & | ||
| LMCACHE_PID=$! | ||
| echo "LMCache server PID: $LMCACHE_PID" | ||
| wait_for_lmcache_ready | ||
| # Remove --disable-hybrid-kv-cache-manager and enable hybrid kv cache manager (default) | ||
| # This gives extra cache hit than disabling hybrid kv cache manager | ||
| OFFLOAD_ARGS=( | ||
| --kv-transfer-config | ||
| "{\"kv_connector\":\"LMCacheMPConnector\",\"kv_connector_module_path\":\"lmcache.integration.vllm.lmcache_mp_connector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"lmcache.mp.host\":\"$LMCACHE_CONNECT_HOST\",\"lmcache.mp.port\":$LMCACHE_PORT}}" | ||
| ) | ||
| ;; | ||
| esac | ||
| # ---- LLM server config ---------------------------------------------------------- | ||
| PARALLEL_ARGS=(--tensor-parallel-size "$TP") | ||
| if [ "${DP_ATTENTION}" = "true" ]; then | ||
| PARALLEL_ARGS=( | ||
| --tensor-parallel-size 1 | ||
| --data-parallel-size "$TP" | ||
| --enable-expert-parallel | ||
| ) | ||
| elif [ "$EP_SIZE" -gt 1 ]; then | ||
| PARALLEL_ARGS+=(--enable-expert-parallel) | ||
| fi | ||
| echo "Starting vllm server..." | ||
| export PYTHONNOUSERSITE=1 | ||
| export VLLM_ENGINE_READY_TIMEOUT_S=3600 | ||
| export VLLM_USE_BREAKABLE_CUDAGRAPH=0 | ||
| export VLLM_ROCM_USE_AITER=1 | ||
| export VLLM_ROCM_USE_AITER_MOE=1 | ||
| export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1 | ||
| # INT4 quantized all-reduce for the (~1.5 MB) decode all-reduces, which are the | ||
| # single biggest decode kernel at high concurrency. The MIN_SIZE_KB override is | ||
| # required: vLLM's default INT4 quick-reduce size gate for (bf16, TP4) is 16 MB, | ||
| # so it never fires for decode-sized tensors without it. | ||
| export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 | ||
| export VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0 | ||
| export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256 | ||
| VLLM_CMD=( | ||
| vllm serve "$MODEL_PATH" | ||
| --served-model-name "$MODEL" | ||
| --host 0.0.0.0 | ||
| --port "$PORT" | ||
| "${PARALLEL_ARGS[@]}" | ||
| --trust-remote-code | ||
| --block-size 128 | ||
| --gpu-memory-utilization 0.85 | ||
| --language-model-only | ||
| --attention-backend TRITON_ATTN | ||
| --moe-backend aiter | ||
| --kv-cache-dtype fp8 | ||
| --tool-call-parser minimax_m3 | ||
| --enable-auto-tool-choice | ||
| --reasoning-parser minimax_m3 | ||
| --max-num-seqs "$CONC" | ||
| "${OFFLOAD_ARGS[@]}" | ||
| ) | ||
| printf '%q ' "${VLLM_CMD[@]}" | tee "$RESULT_DIR/vllm_command.txt" | ||
| printf '\n' | tee -a "$RESULT_DIR/vllm_command.txt" | ||
| "${VLLM_CMD[@]}" > "$SERVER_LOG" 2>&1 & | ||
| SERVER_PID=$! | ||
| echo "Server PID: $SERVER_PID" | ||
| wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID" | ||
| # ---- Run benchmark ---------------------------------------------------------- | ||
| build_replay_cmd "$RESULT_DIR" | ||
| run_agentic_replay_and_write_outputs "$RESULT_DIR" | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 The docstring, check_env_vars call, and variable defaults in the new script are internally inconsistent: (1) the header lists
OFFLOADINGbut check_env_vars actually requiresKV_OFFLOADING; (2) the header omitsDURATION,EP_SIZE,DP_ATTENTION, andKV_OFFLOAD_BACKEND, all of which check_env_vars enforces; (3) theDURATION=${DURATION:-1800}andEP_SIZE=${EP_SIZE:-1}defaults at lines 19-20 are dead code (check_env_vars has already exited if either was empty); and (4) the header enumeratesnone / cpu / lmcacheas OFFLOADING values but the case statement switches on$KV_OFFLOAD_BACKENDand only implements thelmcachearm —cpuandnonesilently fall through to--no-enable-prefix-caching. The launcher passes the correct vars so CI is unaffected, but the header will actively mislead anyone reading or running this stand-alone. Fix by syncing the docstring with the actual required vars and either dropping the unreachable defaults or removingDURATION/EP_SIZEfrom check_env_vars.Extended reasoning...
The script header at lines 5-20 sets up a runtime interface that contradicts itself on four related points. This is doc/dead-code inconsistency in a brand-new file, so nothing breaks in the CI-driven flow — but this recipe is very likely to be copy-pasted (it is the first vLLM+LMCache MI355X agentic entry), and future readers will be misled.
(1) Wrong variable name in the required-vars list. Line 7 documents
OFFLOADING, but line 14 callscheck_env_vars MODEL TP CONC KV_OFFLOADING TOTAL_CPU_DRAM_GB RESULT_DIR DURATION EP_SIZE DP_ATTENTION. Sibling recipes (dsv4_fp4_b300_sglang.sh:7,kimik2.5_fp4_b300.sh:10,qwen3.5_fp8_h100.sh:15,minimaxm3_fp8_mi325x.sh) all consistently use theKV_OFFLOADINGname — the header here is out of sync with the codebase convention.(2) Missing required vars. The header lists 6 vars;
check_env_varsenforces 9.DURATION,EP_SIZE, andDP_ATTENTIONare all required at line 14 but never mentioned in the header. A user reading only the header, exporting the 6 documented vars, and running the script will hitError: The following required environment variables are not set: DURATION EP_SIZE DP_ATTENTION.(3) Dead-code defaults. Lines 19-20 set
DURATION=${DURATION:-1800}andEP_SIZE=${EP_SIZE:-1}. Butcheck_env_vars(benchmarks/benchmark_lib.sh:234) uses[[ -z "${!var_name:-}" ]]to detect unset OR empty and exits 1 if any listed var is missing. So by the time execution reaches line 19,DURATIONandEP_SIZEare guaranteed non-empty — the:-fallbacks are unreachable. Compareminimaxm3_fp8_mi325x.sh:7, which requires the same vars incheck_env_varsand correctly does NOT repeat the defaults.(4) Values list documents a third variable name. Lines 9-12 enumerate
none / cpu / lmcacheasOFFLOADINGvalues. The case statement at line 90 switches on$KV_OFFLOAD_BACKEND— a third, undocumented name — and only implements thelmcachearm. SettingKV_OFFLOAD_BACKEND=cpu(or=none) falls through to the initialOFFLOAD_ARGS=(--no-enable-prefix-caching)from line 47, which is neither vLLM native CPU offload nor GPU-KV-only in any meaningful sense.Step-by-step proof. Take the header at face value and run:
MODEL=amd/MiniMax-M3-MXFP4 TP=4 CONC=8 OFFLOADING=lmcache TOTAL_CPU_DRAM_GB=3000 RESULT_DIR=/tmp/x ./minimaxm3_fp4_mi355x.sh.set -uis on, so line 14 checksKV_OFFLOADING(which the user did not set because the header saidOFFLOADING);check_env_varsreports 4 missing vars:KV_OFFLOADING,DURATION,EP_SIZE,DP_ATTENTION. Even if the user then also exportsKV_OFFLOADING=lmcache DURATION=1800 EP_SIZE=1 DP_ATTENTION=false, the case at line 90 needsKV_OFFLOAD_BACKEND=lmcache— still-undocumented — otherwise LMCache never starts and vLLM runs with--no-enable-prefix-cachingonly.Fix. Two clean options: (a) sync the header — replace
OFFLOADINGwithKV_OFFLOADING, addKV_OFFLOAD_BACKEND, DURATION, EP_SIZE, DP_ATTENTIONto the required list, drop the inventedcpu/nonevalues (or implement them), and delete lines 19-20; or (b) treatDURATION/EP_SIZEas optional-with-defaults — remove them fromcheck_env_vars, keep the defaults, and add them to the header as optional. The siblingminimaxm3_fp8_mi325x.shfollows option (a); either is fine so long as the docstring and code match. Non-blocking (nit) — CI passes becauseconfigs/amd-master.yamlsupplies the right variables via the launcher, so this only bites stand-alone users and future readers.