Skip to content

Optimize Kimi-K2.5-FP4 on AMD MI355X: Enable AITER and Expert Parallel #922

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ChuanLi1101 wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Optimize Kimi-K2.5-FP4 on AMD MI355X: Enable AITER and Expert Parallel #922

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6a21687
optimize kimi-k2.5-fp4 on amd mi355x gpu
lxgsbqylbk Mar 19, 2026
d404078
add expert parallel for kimik2.5-fp4-mi355x
lxgsbqylbk Mar 20, 2026
5410ce5
Clean up AITER env vars, switch quickreduce to INT4, add TP8 search s…
ChuanLi1101 Mar 22, 2026
4084eeb
Merge branch 'main' into dev/rocm
billishyahao Mar 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -350,14 +350,18 @@ kimik2.5-fp4-mi355x-vllm:
osl: 1024
search-space:
- { tp: 8, conc-start: 4, conc-end: 64 }
- { tp: 4, conc-start: 4, conc-end: 64 }
- isl: 1024
osl: 8192
search-space:
- { tp: 8, conc-start: 4, conc-end: 64 }
- { tp: 4, conc-start: 4, conc-end: 64 }
- { tp: 4, ep: 4, conc-start: 4, conc-end: 64 }
- isl: 8192
osl: 1024
search-space:
- { tp: 8, conc-start: 4, conc-end: 64 }
- { tp: 4, conc-start: 4, conc-end: 64 }

minimaxm2.5-fp8-mi355x-vllm:
image: vllm/vllm-openai-rocm:v0.15.1
Expand Down
27 changes: 21 additions & 6 deletions benchmarks/single_node/kimik2.5_fp4_mi355x.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,24 @@ fi
SERVER_LOG=/workspace/server.log
PORT=${PORT:-8888}

# do not enable aiter due to Aiter MLA not currently supporting num_heads=8
# https://github.com/vllm-project/vllm/issues/35641
# export VLLM_ROCM_USE_AITER=1
# If the machine runs a MEC FW older than 177, RCCL
# cannot reclaim some memory.
# Disable that features to avoid crashes.
# This is related to the changes in the driver at:
# https://rocm.docs.amd.com/en/docs-6.4.3/about/release-notes.html#amdgpu-driver-updates
version=`rocm-smi --showfw | grep MEC | head -n 1 | awk '{print $NF}'`
if [[ "$version" == "" || $version -lt 177 ]]; then
export HSA_NO_SCRATCH_RECLAIM=1
fi

export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4

if [ "$EP_SIZE" -gt 1 ]; then
EP=" --enable-expert-parallel"
else
EP=" "
fi

# following AMD andy luo's recipe
# https://x.com/linluo77/status/2017024513595301985
Expand All @@ -44,10 +59,10 @@ start_gpu_monitor
set -x
vllm serve $MODEL --port $PORT \
--tensor-parallel-size=$TP \
--gpu-memory-utilization 0.95 \
$EP \
--gpu-memory-utilization 0.90 \
--max-model-len $MAX_MODEL_LEN \
--block-size=64 \
--disable-log-requests \
--block-size=1 \
Copy link
Collaborator

@seungrokj seungrokj Mar 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--no-enable-prefix-caching \

@ChuanLi1101 now we need this

--trust-remote-code \
--mm-encoder-tp-mode data > $SERVER_LOG 2>&1 &

Expand Down