Add qwen3 model

[pstjohn]

This PR includes changes from #1486, if we merge that the diff here will just be in qwen3.

Includes a temporary fix for NVIDIA/TransformerEngine#2718, which we can remove when that is merged and in the base image.

This adds the Qwen3 model (https://huggingface.co/Qwen/Qwen3-0.6B), specifically the dense variant, although MoE would be fairly easy to add with our Mixtral model recipe.

Key differences of Qwen3 vs. Llama3 --

• qk_norm layers, using TE's qk_norm_type and qk_norm_before_rope kwargs
• sliding window attention (SWA), using the window_size kwarg on certain layers:

window_size=(config.sliding_window, config.sliding_window) 
  if config.layer_types[layer_idx] == "sliding_attention" else None

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}