diff --git a/.github/codeowner-signoff-verify-prompt.md b/.github/codeowner-signoff-verify-prompt.md index 99e5236b0..ad0879c17 100644 --- a/.github/codeowner-signoff-verify-prompt.md +++ b/.github/codeowner-signoff-verify-prompt.md @@ -19,7 +19,7 @@ You are an automated merge-gate auditor for InferenceX. A CODEOWNER (`${SIGNOFF_AUTHOR}`) just posted the reviewer sign-off checklist (as a ${SIGNOFF_KIND}) that marks PR #${PR_NUMBER} as ready to merge. Your job is to -INDEPENDENTLY verify the checks below (0-9). Do not trust the reviewer's checkmarks +INDEPENDENTLY verify the checks below (0-10). Do not trust the reviewer's checkmarks — re-derive every conclusion from CODEOWNERS, CI runs, the PR diff, the master configs, and the linked recipe yourself. Be rigorous and specific. The checks encode the merge standard in `docs/PR_REVIEW_CHECKLIST.md` (read it in the checked-out @@ -291,8 +291,48 @@ Installing the benchmark harness and client-side deps (aiperf, eval tooling) is waiver not linked / waiver does not cover this patch). - N/A if the PR touches no benchmark scripts, images, or configs. +## Check 10 — Agentic spec-decode configs use the golden simulated acceptance length +APPLICABILITY: this check covers AGENTIC-workload benchmark changes that enable +speculative decoding. From the PR diff, identify configs that are BOTH: +- agentic — scripts under `benchmarks/single_node/agentic/**`, multi-node recipes + under an `agentic/` directory (e.g. `benchmarks/multi_node/srt-slurm-recipes/**/agentic/**`), + or master-config entries whose name/recipe path marks them agentic; AND +- speculative-decoding — MTP / EAGLE / draft-model flags such as + `--speculative-config`, `--speculative-algorithm`, `spec-decode`, draft-model + downloads, or config names containing `-mtp` / `eagle`. +Agentic replay does not reproduce real-world token-by-token traffic, so measured +acceptance there is not representative; per the AgentX fairness guidelines +(`golden_al_distribution/README.md` in the checked-out default branch), such configs +must instead SIMULATE acceptance at the committed golden acceptance length (AL). +Verify BOTH: +- (a) SIMULATED ACCEPTANCE ENABLED. The launch config must pin a simulated/synthetic + acceptance length: + - SGLang: env var `SGLANG_SIMULATE_ACC_LEN: ` (normally with + `SGLANG_SIMULATE_ACC_METHOD: match-expected` and + `SGLANG_SIMULATE_ACC_TOKEN_MODE: real-draft-token`) in the server/decode + environment (e.g. `aggregated_environment` / `decode_environment` in srt-slurm + YAMLs, or exported before launch in benchmark scripts). + - vLLM: the `--speculative-config` JSON contains BOTH + `"rejection_sample_method": "synthetic"` and `"synthetic_acceptance_length": `. + FAIL if an agentic spec-decode config runs real (unsimulated) acceptance — name the + config/script and line. +- (b) AL VALUE MATCHES THE GOLDEN CURVE. Read the committed golden AL YAML for the + model in `golden_al_distribution/` (default-branch checkout; e.g. `qwen3.5_mtp.yaml`, + `kimik2.5_eagle3.yaml`) and confirm the pinned AL equals the golden value for that + model, thinking mode, and the config's `num_speculative_tokens` / MTP level (e.g. + qwen3.5 thinking_on with 3 speculative tokens -> 3.39). A submission may choose any + supported draft length, but it may NOT substitute a different acceptance target. + FAIL on a mismatch — name the config, the pinned value, and the expected golden + value. If the model has no committed golden curve yet, do not guess: the sign-off's + additional detail section must state the source of the AL value (e.g. a pending + golden-curve collection run); FAIL if it does not. +- Also FAIL (as a benchmark hack) if simulated/synthetic-acceptance knobs appear on a + NON-agentic spec-decode config, where Check 8's real-traffic AL standard applies — + unless the sign-off documents a sanctioned exception. +- N/A if the PR has no agentic speculative-decoding changes (state that in one line). + ## Verdict and output -Decide PASS only if Checks 0-9 ALL pass (a check reported as `N/A` counts as a pass — +Decide PASS only if Checks 0-10 ALL pass (a check reported as `N/A` counts as a pass — keep the `N/A — ` row so the reviewer sees it was considered). Post EXACTLY ONE summary comment on PR #${PR_NUMBER} using `gh pr comment`. Start the comment with the hidden marker so reruns are identifiable: @@ -318,7 +358,7 @@ single terse line either. Rules: restating the checklist, no hedging ("if X then maybe Y" — make the call). Link the run/recipe instead of describing it. -- If everything is to standard: post the verdict header + the ten one-line rows +- If everything is to standard: post the verdict header + the eleven one-line rows (with the green run URL). Do NOT @-mention anyone on a pass. - If anything is NOT to standard: the verdict header must be immediately followed by a line that @-mentions the sign-off author as `@${SIGNOFF_AUTHOR}` diff --git a/.github/workflows/codeowner-signoff-verify.yml b/.github/workflows/codeowner-signoff-verify.yml index ab3f385a2..3f0faeb39 100644 --- a/.github/workflows/codeowner-signoff-verify.yml +++ b/.github/workflows/codeowner-signoff-verify.yml @@ -30,6 +30,9 @@ name: CODEOWNER Sign-off Verify # 9. No patches to the inference engine / serving stack (the pinned image # runs as shipped) — the only exception is a patch covered by a # filled-out docs/waiver/.md waiver. +# 10. Agentic spec-decode configs simulate acceptance at the committed golden +# AL from golden_al_distribution/ (SGLang SGLANG_SIMULATE_ACC_* env vars, +# vLLM synthetic rejection sampling in --speculative-config). # If any of those are not to standard, Claude posts a single comment that # @-mentions the reviewer who signed off and explains exactly what is wrong. # diff --git a/docs/PR_REVIEW_CHECKLIST.md b/docs/PR_REVIEW_CHECKLIST.md index 5eae119d4..c0963a3fb 100644 --- a/docs/PR_REVIEW_CHECKLIST.md +++ b/docs/PR_REVIEW_CHECKLIST.md @@ -20,6 +20,7 @@ As a PR reviewer and CODEOWNER, I have reviewed this and have: - [ ] Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. - [ ] Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. - [ ] Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world +- [ ] For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance — SGLang via `SGLANG_SIMULATE_ACC_LEN` (with `SGLANG_SIMULATE_ACC_METHOD: match-expected` and `SGLANG_SIMULATE_ACC_TOKEN_MODE: real-draft-token`), vLLM via `--speculative-config` with `"rejection_sample_method": "synthetic"` and `"synthetic_acceptance_length"` — and that the acceptance-length value exactly matches the committed golden AL curve in [golden_al_distribution/](https://github.com/SemiAnalysisAI/InferenceX/tree/main/golden_al_distribution) for that model, thinking mode, and `num_speculative_tokens`. A submission may choose any supported draft length, but it may not substitute a different acceptance target. - [ ] Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy - [ ] If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers - [ ] If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet. diff --git a/docs/PR_REVIEW_CHECKLIST_zh.md b/docs/PR_REVIEW_CHECKLIST_zh.md index 3895c03a2..81719deba 100644 --- a/docs/PR_REVIEW_CHECKLIST_zh.md +++ b/docs/PR_REVIEW_CHECKLIST_zh.md @@ -22,6 +22,7 @@ As a PR reviewer and CODEOWNER, I have reviewed this and have: - [ ] Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. - [ ] Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. - [ ] Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world +- [ ] For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance — SGLang via `SGLANG_SIMULATE_ACC_LEN` (with `SGLANG_SIMULATE_ACC_METHOD: match-expected` and `SGLANG_SIMULATE_ACC_TOKEN_MODE: real-draft-token`), vLLM via `--speculative-config` with `"rejection_sample_method": "synthetic"` and `"synthetic_acceptance_length"` — and that the acceptance-length value exactly matches the committed golden AL curve in [golden_al_distribution/](https://github.com/SemiAnalysisAI/InferenceX/tree/main/golden_al_distribution) for that model, thinking mode, and `num_speculative_tokens`. A submission may choose any supported draft length, but it may not substitute a different acceptance target. - [ ] Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy - [ ] If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers - [ ] If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet. @@ -42,12 +43,13 @@ Signed: `FILL_IN_GITHUB_USERNAME` 3. 已确认该 PR 通过了 PR 验证,并附上能证明这一点的 GitHub Action 工作流链接。 4. 已确认该 PR 通过了 evals(准确性评测),并附上能证明这一点的 GitHub Action 工作流链接。 5. 已确认投机解码(speculative decoding)PR 使用 chat template,使接受长度(AL)分布与真实场景对齐。 -6. 已确认模型架构未被基准测试 hack 更改 — 例如在不原生支持的模型上使用 `--hf-overrides` 每 x 层跳过 indexer。一般规则:不接受减少模型架构 FLOPs 的优化;让同样的计算跑得更快没有问题;更低精度的 FLOPs 也可以,前提是该配置通过私有 evals。北极星原则:只使用在意准确性的客户在生产中实际使用的优化。 -7. 如果公司声称在其硬件上将 vLLM/SGLang 作为一等 LLM 推理引擎支持,已确认相应 vLLM 提交使用上游 [vLLM docker 仓库](https://hub.docker.com/u/vllm)、SGLang 提交使用上游 [lmsysorg docker 仓库](https://hub.docker.com/u/lmsysorg)。唯一例外:新硬件(如 MI455X UALoE72、Vera Rubin NVL72、Rubin NVL8 等),以及经 vLLM/SGLang 社区维护者确认上游尚未从根本上支持的新模型架构。 -8. 如果公司声称在其硬件上将 vLLM/SGLang 作为一等上游 in-tree LLM 推理引擎支持,已确认相应 vLLM/SGLang 提交先于其他框架(TRT-LLM、ATOM 等)完成。例外情形同上。 -9. 已确认单节点 recipe 与官方 [vLLM recipes](https://recipes.vllm.ai/) 和/或 [SGLang cookbook](https://docs.sglang.io/cookbook/intro) 相似;如果不相似,已确认在 [vLLM recipe 仓库](https://github.com/vllm-project/recipes)或 [SGLang 仓库](https://github.com/sgl-project/sglang/tree/main/docs_new)开了 PR,并在下方 Additional detail section 中给出链接。 -10. 已确认该 PR 未对推理引擎或 serving 技术栈打补丁 —— 锁定的镜像必须原样运行。涵盖:.patch 文件 / `git apply` / `patch`、内嵌在基准测试脚本中的行内补丁(例如在启动服务前用 python3/sed heredoc 改写已安装的引擎源码)、就地编辑 site-packages、monkey-patch、覆盖容器文件、以及在锁定镜像之上安装 fork 或重新构建的引擎 wheel。唯一例外:该补丁已由 [docs/waiver/](https://github.com/SemiAnalysisAI/InferenceX/tree/main/docs/waiver)`.md`(以引入补丁的 PR 编号命名,并在同一 PR 中提交)中填写完整的豁免覆盖 —— 写明补丁内容、为何未修改的上游镜像无法运行该基准测试、上游 PR/issue 链接及移除计划 —— 并已在下方 Additional detail section 中给出链接。 -11. 如果上述任何条目无法合理满足,已在下方提供额外说明。 +6. 对 agentic 工作负载:已确认投机解码配置(EAGLE / MTP / draft 模型)启用了模拟合成接受(simulated synthetic acceptance)—— SGLang 通过 `SGLANG_SIMULATE_ACC_LEN`(配合 `SGLANG_SIMULATE_ACC_METHOD: match-expected` 与 `SGLANG_SIMULATE_ACC_TOKEN_MODE: real-draft-token`),vLLM 通过 `--speculative-config` 中的 `"rejection_sample_method": "synthetic"` 与 `"synthetic_acceptance_length"` —— 且接受长度(AL)取值与 [golden_al_distribution/](https://github.com/SemiAnalysisAI/InferenceX/tree/main/golden_al_distribution) 中该模型、thinking 模式与 `num_speculative_tokens` 对应的已提交黄金 AL 曲线完全一致。提交可选择任意受支持的 draft 长度,但不得替换为其他接受目标。 +7. 已确认模型架构未被基准测试 hack 更改 — 例如在不原生支持的模型上使用 `--hf-overrides` 每 x 层跳过 indexer。一般规则:不接受减少模型架构 FLOPs 的优化;让同样的计算跑得更快没有问题;更低精度的 FLOPs 也可以,前提是该配置通过私有 evals。北极星原则:只使用在意准确性的客户在生产中实际使用的优化。 +8. 如果公司声称在其硬件上将 vLLM/SGLang 作为一等 LLM 推理引擎支持,已确认相应 vLLM 提交使用上游 [vLLM docker 仓库](https://hub.docker.com/u/vllm)、SGLang 提交使用上游 [lmsysorg docker 仓库](https://hub.docker.com/u/lmsysorg)。唯一例外:新硬件(如 MI455X UALoE72、Vera Rubin NVL72、Rubin NVL8 等),以及经 vLLM/SGLang 社区维护者确认上游尚未从根本上支持的新模型架构。 +9. 如果公司声称在其硬件上将 vLLM/SGLang 作为一等上游 in-tree LLM 推理引擎支持,已确认相应 vLLM/SGLang 提交先于其他框架(TRT-LLM、ATOM 等)完成。例外情形同上。 +10. 已确认单节点 recipe 与官方 [vLLM recipes](https://recipes.vllm.ai/) 和/或 [SGLang cookbook](https://docs.sglang.io/cookbook/intro) 相似;如果不相似,已确认在 [vLLM recipe 仓库](https://github.com/vllm-project/recipes)或 [SGLang 仓库](https://github.com/sgl-project/sglang/tree/main/docs_new)开了 PR,并在下方 Additional detail section 中给出链接。 +11. 已确认该 PR 未对推理引擎或 serving 技术栈打补丁 —— 锁定的镜像必须原样运行。涵盖:.patch 文件 / `git apply` / `patch`、内嵌在基准测试脚本中的行内补丁(例如在启动服务前用 python3/sed heredoc 改写已安装的引擎源码)、就地编辑 site-packages、monkey-patch、覆盖容器文件、以及在锁定镜像之上安装 fork 或重新构建的引擎 wheel。唯一例外:该补丁已由 [docs/waiver/](https://github.com/SemiAnalysisAI/InferenceX/tree/main/docs/waiver)`.md`(以引入补丁的 PR 编号命名,并在同一 PR 中提交)中填写完整的豁免覆盖 —— 写明补丁内容、为何未修改的上游镜像无法运行该基准测试、上游 PR/issue 链接及移除计划 —— 并已在下方 Additional detail section 中给出链接。 +12. 如果上述任何条目无法合理满足,已在下方提供额外说明。 ## 示例