[plugin][OOT Benchmark] Refine OOT benchmark(manual trigger) to cover key models

[zejunchen-zejun]

Refine OOT benchmark with following points:

• only manual trigger OOT benchmark, no schedule, no nightly
• benchmark kimi(TP8 TP4), gptoss(TP1), qwen3.5(TP8), ds-fp8(TP8) and ds-mxfp4(TP8)
• benchmark concurrency 4 8 16 32 64
• benchmark 1k/1k, 8k/1k, 1k/8k
• default all False for all models
• 选择main branch发起OOT benchmark，默认直接拉取最新的OOT docker进行benchmark
• 选择main branch发起OOT benchmark，并且指定了某个OOT docker，直接拉取该docker进行benchmark
• 选择非main branch发起OOT benchmark，自动基于该branch构建OOT docker，随后进行benchmark，数据不上传到dashboard

[Copilot]

Pull request overview

This PR refines the manual OOT vLLM benchmark workflow to target a curated set of key models and benchmark parameter combinations, while switching from building a custom OOT image in-workflow to pulling a prebuilt “latest” image.

Changes:

• Make the OOT benchmark workflow manual-only with model toggles defaulting to false, and add an oot_image input to pull a prebuilt benchmark image.
• Change the benchmark execution from an in-job loop over param_lists to a full job matrix over (model × params) and generate per-config artifacts.
• Update the OOT model config list to adjust env vars and add Qwen3.5-397B-A17B-FP8.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
.github/workflows/atom-vllm-oot-benchmark.yaml	Switch to pulling a prebuilt OOT image; add param matrix expansion; default-disable models for manual selection.
.github/benchmark/oot_benchmark_models.json	Update env vars for existing models and add the Qwen3.5 FP8 model entry.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[wuhuikx]

1. for each concurrency, we need to re-launch the container, do you follow this instruction?
2. warmup num = 2x con
3. prompt num = 10x con

Ensure each model can be triggered separately. For example, when we have optimization on DS, we only need to refresh data on this model while keep the others silent to save hardware resource.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[zejunchen-zejun]

1. for each concurrency, we need to re-launch the container, do you follow this instruction?
2. warmup num = 2x con
3. prompt num = 10x con

Ensure each model can be triggered separately. For example, when we have optimization on DS, we only need to refresh data on this model while keep the others silent to save hardware resource.

1. followed in this PR
2. followed, set the warm up steps: --num-warmups=\"$(( 2 * CONC ))\" \
3. followed, set the promopt steps: --num-prompts=\"$(( CONC * 10 ))\" \

[wuhuikx]

One more thing, we need to make benchmark running on different atom branch for vllm upgrading. Do we have the function now for manually selecting the branch?

[zejunchen-zejun]

One more thing, we need to make benchmark running on different atom branch for vllm upgrading. Do we have the function now for manually selecting the branch?

make sense, we need it can also do acceptance test. Let me add it

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.