Add Gemma 4 support#492

Merged

jlamypoirier merged 24 commits intomainfrom

May 7, 2026

Commits on May 5, 2026

Add embedding_scale and final_logit_softcap for Gemma 4 support

jlamypoirier
and
claude
committed
Add QK norm and post-mixer/MLP normalization for Gemma 4 support

jlamypoirier
and
claude
committed
Expand test_attention with MQA/MHA/rotary/norm coverage

jlamypoirier
and
claude
committed
Add ProportionalRotary and rewrite test_rotary as parametrized suite

jlamypoirier
and
claude
committed
Add value_norm, shared_key_value, and FixedRMSNorm (Gemma 4 attention)

jlamypoirier
and
claude
committed
Fix in-place rotary corruption of query_norm backward context

jlamypoirier
and
claude
committed
Add HybridMoEMLP and independent pre-norm controls for decoder block

jlamypoirier
and
claude
committed
Add Gemma 4 HuggingFace checkpoint converter

jlamypoirier
and
claude
committed
Improve Gemma4 converter: shared_key_value, MoE weight shapes, roundtrip test

jlamypoirier
and
claude
committed
Fix test_rotary and test_mlp failures

jlamypoirier
and
claude
committed
Add output_scale to DecoderBlock for Gemma 4 layer_scalar

jlamypoirier
and
claude
committed
Fold output_scale into bias_dropout_add; fix subprocess interpreter

jlamypoirier
and
claude
committed
Move pre/post norm to MLPBaseConfig; add MoE router preprocessing

jlamypoirier
and
claude
committed
Add learnable per-expert scale to MoE router
jlamypoirier
committed
Exercise HybridMoE in gemma4 test fixture
jlamypoirier
committed
Guard converters against unsupported features
jlamypoirier
committed
Address PR review feedback
jlamypoirier
committed
Address second-round PR review

jlamypoirier
and
claude
committed

Commits on May 6, 2026

Commits on May 7, 2026

Merge branch 'main' into worktree-gemma
jlamypoirier
authored