Skip to content

feat: add Qwen3.5 image-to-text support#152

Open
xsuler wants to merge 40 commits into
mainfrom
feat/qwen35-image-to-text
Open

feat: add Qwen3.5 image-to-text support#152
xsuler wants to merge 40 commits into
mainfrom
feat/qwen35-image-to-text

Conversation

@xsuler

@xsuler xsuler commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

This pull request adds Qwen3.5 image-to-text support across AReno's model, serving, training, and example paths. It enables Qwen3.5 checkpoints with vision components to accept image inputs, project visual features through the vision tower, and feed the resulting image embeddings into the language model with aligned image-token masks and multimodal position information.

Scope

The implementation covers the following areas:

  • Qwen3.5-VL model integration, including vision tower execution, image feature projection, multimodal feature injection, and mRoPE position handling.
  • Qwen3.5-MoE-VL adapter support, so checkpoints with a MoE language backbone and vision configuration are matched, initialized, and loaded through a dedicated multimodal wrapper instead of the dense VL or text-only MoE paths.
  • OpenAI-compatible serving support for image inputs, including base64 image payloads and compatibility with tool-call requests when the processor supports the combined format.
  • Training-path support for multimodal samples, including packed train batch handling for image features, image token masks, and multimodal position ids.
  • Agentic image-to-text example coverage through a visual tic-tac-toe task, including dataset generation, loader, reward logic, run agent entrypoint, and web UI integration.
  • Documentation and example commands for running Qwen3.5 image-to-text SFT and agentic GSPO workflows.

Compatibility

The change is intended to preserve existing text-only Qwen3.5 behavior while adding the multimodal path for checkpoints that include vision configuration and image token metadata. Existing text-only serving and training paths should continue to use their previous data flow.

Notes

The Qwen3.5-MoE-VL path reuses the same vision tower and multimodal feature injection flow as dense Qwen3.5-VL, while selecting the MoE language backbone from the checkpoint configuration. The vision merger shape follows the checkpoint layout where the merged visual hidden size is hidden_size * spatial_merge_size^2.

Closes #133
Closes #134
Closes #135
Closes #136

xsuler added 30 commits July 6, 2026 15:10
@xsuler xsuler requested a review from adohe July 7, 2026 08:34

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces comprehensive support for multimodal models, specifically Qwen3.5-VL, across the training, serving, and rollout pipelines, including new helper utilities for image encoding, token expansion, and MRoPE position alignment. It also adds a complete Tic-Tac-Toe image training example. The review feedback identifies two critical bugs in areno/engine/runtime/train_step.py and areno/models/qwen3_5/model.py where indexing image_token_mask with row_idx instead of 0 will cause an IndexError for batches containing multiple sequences.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread areno/engine/runtime/train_step.py
Comment thread areno/models/qwen3_5/model.py
@xsuler xsuler self-assigned this Jul 7, 2026
@xsuler xsuler added this to the v0.0.5 milestone Jul 7, 2026
@xsuler xsuler added kind/feature Categorizes issue or PR as related to a new feature area/models Issues or PRs related to model adapters (llama, qwen3, bailing, gemma4, etc.) area/serving Issues or PRs related to the inference server and continuous batching labels Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/models Issues or PRs related to model adapters (llama, qwen3, bailing, gemma4, etc.) area/serving Issues or PRs related to the inference server and continuous batching kind/feature Categorizes issue or PR as related to a new feature

Projects

None yet

1 participant