Skip to content

align act_max of experts for qwen3-vl and qwen3-next #1311

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
xin3he wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

align act_max of experts for qwen3-vl and qwen3-next #1311

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ef05ec0
align act_max of experts for qwen3-vl and qwen3-next
xin3he Jan 21, 2026
6f5ed28
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion auto_round/utils/model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -613,6 +613,8 @@ def is_moe_layer(module: torch.nn.Module) -> bool:
"DeepseekV3MoE".lower(),
"Qwen2MoeSparseMoeBlock".lower(),
"Qwen3MoeSparseMoeBlock".lower(),
"Qwen3NextSparseMoeBlock".lower(),
"Qwen3VLMoeTextSparseMoeBlock".lower(),
Copy link
Contributor

@WeiweiZhang1 WeiweiZhang1 Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this change is intended to support NVFP quantization for Qwen3VLMoE&Qwen3Next, please also verify VLLM generation, thanks!

Copy link
Contributor Author

@xin3he xin3he Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, this PR is necessary since some experts are not calibrated and act_max is missing.

]
)

Expand Down Expand Up @@ -722,7 +724,16 @@ def module_match_name_list(module, name_list):
return any(name.lower() in type(module).__name__.lower() for name in name_list)

if module_match_name_list(
module, ["Qwen2MoeSparseMoeBlock", "Qwen3MoeSparseMoeBlock", "DeepseekMoE", "DeepseekV2MoE", "DeepseekV3MoE"]
module,
[
"Qwen2MoeSparseMoeBlock",
"Qwen3MoeSparseMoeBlock",
"Qwen3NextSparseMoeBlock",
"Qwen3VLMoeTextSparseMoeBlock",
"DeepseekMoE",
"DeepseekV2MoE",
"DeepseekV3MoE",
],
):
return ["gate_proj", "down_proj", "up_proj"]
elif module_match_name_list(module, ["MixtralMoeSparseMoeBlock"]):
Expand Down