[Proposal] Add Nemotron-H hybrid Mamba2-Transformer adapter

[jlarson4]

Proposal

Add a TransformerBridge adapter for NemotronHForCausalLM (NVIDIA Nemotron-H), a hybrid that mixes Mamba-2 and attention layers.

Motivation

Nemotron-H keeps only a small fraction of attention layers (around 8%) and replaces the rest with Mamba-2. That makes the few attention layers a clean target for interpretability: researchers can ask what those layers do that the state-space layers cannot. The line has strong, ongoing NVIDIA releases and wide adoption, and it complements the existing Mamba and Mamba2 support.

We have a limited amount of support for Mamba layers, and working on this will open some new avenues to support possible work on those Mamba layers as well.

Gap scan (2026-06-18): 53 models, ~4.99M downloads, the highest-ranked hybrid state-space gap.

Pitch

Build on the existing Mamba2 components for the state-space layers and standard attention hooks for the interleaved attention layers. A tiny test checkpoint (trl-internal-testing/tiny-NemotronHForCausalLM-nano) keeps CI cheap.

• Claude Code users can scaffold with /add-model-support nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16.
• Register at the four sites listed in contributing.md.
• Verify smallest-first: trl-internal-testing/tiny-NemotronHForCausalLM-nano, then nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16.

Additional context

• Paper: https://arxiv.org/abs/2504.03624
• Checkpoint: https://huggingface.co/nvidia/Nemotron-H-4B-Base-8K
• Found via the hf_scraper architecture-gaps pass (2026-06-18).

Checklist

• I have checked that there is no similar issue in the repo (required)

[mukund1985]

I'd like to take this on. Working on the DeepSeek-V2 MLA adapter (#1408) currently — Nemotron-H's Mamba2-Transformer hybrid is a natural next step. Will comment on the issue before opening a PR.

[jlarson4]

@mukund1985 you are now assigned! let me know if you have any questions.

[mukund1985]

PR up at #1434. Closes this.

Key design decision worth flagging: NemotronH uses a single .mixer attribute for all four layer types (mamba/attention/moe/mlp), so I used SSMBlockBridge + SSM2MixerBridge as a pure passthrough — works for all types since it delegates to original_component(*args, **kwargs). Mamba-specific submodules (in_proj, conv1d, inner_norm, out_proj) are marked optional so component_setup skips them gracefully on attention/moe/mlp layers.

One minor note: GatedRMSNormBridge.__init__ doesn't accept optional= (unlike the GeneralizedComponent base), so I set it post-init. Happy to discuss if there's a cleaner approach there.

52 unit tests, all passing.

[jlarson4]

Thanks @mukund1985! I will dig into that PR first thing tomorrow