This is the official implementation of "From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion", accepted to ECCV 2026. [arXiv]
Abstract. Vision-Language Models (VLMs) create a severe visual feature bottleneck by using a crude, asymmetric connection that links only the output of the vision encoder to the input of the large language model (LLM). We introduce Cross-Layer Injection (CLI), a novel framework that forges a dynamic "many-to-many" bridge between the two modalities. CLI consists of two synergistic, parameter-efficient components: an Adaptive Multi-Projection (AMP) module that harmonizes features from diverse vision layers, and an Adaptive Gating Fusion (AGF) mechanism that empowers the LLM to selectively inject the most relevant visual information based on its real-time decoding context. We validate CLI on LLaVA-OneVision and LLaVA-1.5, demonstrating significant performance improvements across 28 diverse benchmarks.
- Many-to-Many Fusion: CLI replaces the conventional one-to-one vision-language bridge with a dynamic many-to-many architecture, enabling each LLM decoder layer to query the full visual hierarchy on demand.
- Adaptive Multi-Projection (AMP): Parameter-efficient LoRA-based projectors harmonize features from diverse vision encoder layers into a shared semantic space.
- Adaptive Gating Fusion (AGF): A query-based attention gate dynamically selects and injects the most relevant visual information based on the LLM's real-time decoding context.
- +19.06 Aggregate Score Gain: Significant improvements across 28 benchmarks on LLaVA-OV-7B, with up to +6.5 on LLaVA-in-the-Wild, +4.7 on OCR-Bench, and +3.3 on MME.
- Architecture-Agnostic: Validated on both LLaVA-OneVision (0.5B/7B) and LLaVA-1.5-7B, demonstrating broad applicability.
- Minimal Overhead: Only 1.3% inference memory increase with marginal latency impact.
We release the trained CLI checkpoints on Hugging Face:
| Model | Base VLM | Description | Checkpoint |
|---|---|---|---|
| CLI-7B | LLaVA-OneVision-7B | Full-scale CLI model (Qwen2-7B + SigLIP) | 🤗 codefuse-ai/CLI-7B |
| CLI-0.5B | LLaVA-OneVision-0.5B | Lightweight CLI model (Qwen2-0.5B + SigLIP) | 🤗 codefuse-ai/CLI-0.5B |
Vision Encoder (SigLIP / CLIP)
┌─────────────────────────────┐
│ Layer 1 ──────┐ │
│ Layer 5 ──────┤ │
│ Layer 9 ──────┤ Multi- │
│ Layer 13 ──────┤ Layer │
│ Layer 17 ──────┤ Features │
│ Layer 21 ──────┤ │
│ Layer 25 ──────┘ │
└─────────────────────────────┘
│
┌─────────────────┼─────────────────┐
│ │ │
┌────▼────┐ ┌────▼────┐ ┌────▼────┐
│ AMP │ │ AMP │ ... │ AMP │
│ (LoRA) │ │ (LoRA) │ │ (LoRA) │
└────┬────┘ └────┬────┘ └────┬────┘
│ │ │
▼ ▼ ▼
┌──────────────────────────────────────────────────┐
│ LLM Decoder │
│ Layer 1 ◄── AGF (gated fusion) │
│ Layer 5 ◄── AGF (gated fusion) │
│ Layer 9 ◄── AGF (gated fusion) │
│ Layer 13 ◄── AGF (gated fusion) │
│ Layer 17 ◄── AGF (gated fusion) │
│ Layer 21 ◄── AGF (gated fusion) │
└──────────────────────────────────────────────────┘
At each injection point, the AGF module uses multi-head attention with learnable queries to distill information from both the visual features and the LLM's current hidden state. A sigmoid-gated controller then selectively fuses the most relevant visual information, enabling dynamic criss-crossed connections between vision and language hierarchies.
| Model | AI2D | ChartQA | DocVQA | InfoVQA | LLaVA-W | MME | OCR | Sum |
|---|---|---|---|---|---|---|---|---|
| Baseline | 77.5 | 78.5 | 82.5 | 69.5 | 68.0 | 85.1 | 52.4 | 650.9 |
| w/ DeepStack | 76.0 | 63.8 | 72.6 | 62.2 | 70.3 | 84.1 | - | 611.8 |
| w/ SLI | 77.6 | 77.3 | 79.9 | 67.6 | 68.5 | 84.4 | - | 645.8 |
| w/ CLI (Ours) | 77.9 | 78.7 | 82.8 | 70.5 | 74.5 | 88.4 | 57.1 | 660.6 |
LLaVA-W ████████████████████████████████ +6.5
OCR ███████████████████████ +4.7
MME ████████████████ +3.3
MMStar ████████████ +2.4
InfoVQA █████ +1.0
OK-VQA ████ +0.7
MMMU ███ +0.6
RefCoCo+ ███ +0.5
RefCoCo ███ +0.5
RefCoCog ███ +0.5
git clone https://github.com/codefuse-ai/CLI.git
cd CLI
conda create -n cli python=3.10 -y
conda activate cli
pip install -e ".[train]"
pip install flash-attn --no-build-isolation| Package | Version |
|---|---|
| PyTorch | 2.1.2 |
| Transformers | 4.41.2 |
| DeepSpeed | 0.14.4 |
| Accelerate | 0.34.0 |
| PEFT | 0.4.0 |
Download the base model checkpoints:
| Model | LLM | Vision Encoder | Checkpoint |
|---|---|---|---|
| LLaVA-OV-0.5B | Qwen2-0.5B | SigLIP-so400m-patch14-384 | lmms-lab |
| LLaVA-OV-7B | Qwen2-7B-Instruct | SigLIP-so400m-patch14-384 | lmms-lab |
| LLaVA-1.5-7B | Vicuna-7B-v1.5 | CLIP-ViT-L-336px | liuhaotian |
Place the downloaded models:
pretrained_models/
├── lmms-lab/
│ ├── llava-onevision-qwen2-0.5b-mid-stage-a4/
│ └── llava-onevision-qwen2-7b-mid-stage-a4/
└── vision/
└── google/siglip-so400m-patch14-384-new/
We use ~1.4M samples sampled from the LLaVA-OneVision single-image instruction data. The dataset covers five categories:
| Category | Samples | Key Sources |
|---|---|---|
| General QA & Conversation | ~1.01M | ShareGPT4V/4o, Vision FLAN, Cambrian, ALLaVA |
| Language | ~180K | Magpie-Pro (L3, Qwen2) |
| Doc/Chart/Screen | ~77K | UReader, ChartQA, AI2D, InfographicVQA |
| Math & Reasoning | ~71K | Geo170K, MathQA, MathV360K |
| General OCR | ~6.5K | TextCaps, IAM, ST-VQA |
Place training images at:
dataset/LLaVA-OneVision-Images/
python run_train_cli.py \
--pretrain_vlm_model llava-onevision-qwen2-7b-mid-stage-a4 \
--vlm_exp_layers RANGE-1-28-4 \
--vision_exp_layers RANGE-1-28-4 \
--projector_name mlp2x_gelu \
--num_machines 1 \
--num_gpus 8 \
--output_path ./output/CLI/The layer selection is controlled via environment variables with a RANGE-start-end-step format:
| Parameter | Default | Description |
|---|---|---|
--vlm_exp_layers |
RANGE-1-24-4 |
LLM decoder layers for injection (e.g., layers 1, 5, 9, 13, 17, 21) |
--vision_exp_layers |
RANGE-1-28-4 |
Vision encoder layers to extract features (e.g., layers 1, 5, 9, 13, 17, 21, 25) |
--projector_name |
mlp2x_gelu |
Projector type |
--num_gpus |
8 |
GPUs per machine |
--num_machines |
1 |
Number of machines |
| Parameter | Value |
|---|---|
| Batch Size | 256 |
| Learning Rate (Projector & LLM) | 1e-5 |
| Learning Rate (Vision Encoder) | 2e-6 |
| LR Schedule | Cosine Decay |
| Warmup Ratio | 0.03 |
| Max Epochs | 5 (with early stopping) |
| Model Max Length | 32768 |
| Image Aspect Ratio | anyres_max_9 |
| DeepSpeed | ZeRO-2 |
| Precision | bf16 |
# Node 0
NUM_MACHINES=2 RANK=0 MASTER_ADDR=<master_ip> python run_train_cli.py ...
# Node 1
NUM_MACHINES=2 RANK=1 MASTER_ADDR=<master_ip> python run_train_cli.py ...python run_eval_cli.py \
--model_path ./output/CLI/finetune/<exp_name> \
--eval_tasks ai2d,chartqa,docvqa_val,mme,mmmu \
--vlm_exp_layers RANGE-1-28-4 \
--vision_exp_layers RANGE-1-28-4 \
--num_gpus 1We evaluate on 28 benchmarks across three categories using the LMMs-Eval framework:
Chart, Diagram, and Document Understanding
ai2d- AI2D Diagram Understandingchartqa- ChartQAdocvqa_val/docvqa_test- DocVQAinfovqa_val/infovqa_test- InfographicVQA
Perception and Multidisciplinary Reasoning
mme- MMEmmbench_en_dev- MMBenchmmvet- MM-Vetmmmu- MMMUmmstar- MM-Starmathvista_testmini- MathVistamathverse_testmini_vision_only/vision_dominant/vision_intensive- MathVersegqa- GQAok_vqa- OK-VQAscienceqa_img- ScienceQAseedbench- SEED-Benchpope- POPE
Real-world Understanding and Visual Chat
realworldqa- RealWorldQAllava_in_the_wild- LLaVA-in-the-Wild
python run_eval_cli.py \
--model_path ./output/CLI/finetune/<exp_name> \
--eval_tasks ai2d,chartqa,docvqa_val,docvqa_test,infovqa_val,infovqa_test,mme,mmbench_en_dev,mmvet,mmmu,mmstar,mathvista_testmini,gqa,ok_vqa,scienceqa_img,seedbench,pope,realworldqa,llava_in_the_wildCLI/
├── llava/
│ ├── model/
│ │ ├── llava_arch.py # Core CLI architecture (AMP + AGF)
│ │ ├── multimodal_encoder/ # Vision encoders (SigLIP, CLIP)
│ │ ├── multimodal_projector/ # Base projector implementations
│ │ ├── multimodal_resampler/ # Optional resamplers
│ │ ├── language_model/ # LLM backends (Qwen2, LLaMA)
│ │ └── builder.py # Model builder
│ ├── train/
│ │ ├── train_mem.py # Training entry point
│ │ └── llava_trainer.py # Custom trainer with CLI support
│ ├── eval/
│ │ └── model_lmms_eval.py # LMMs-Eval integration
│ └── utils.py # Utilities (layer parsing, etc.)
├── scripts/
│ ├── train/
│ │ ├── finetune_cli_max9.sh # Main training script
│ │ └── single_image_ov_cli.yaml # Dataset configuration
│ ├── eval/
│ │ └── eval_single.sh # Evaluation script
│ └── zero2.json # DeepSpeed ZeRO-2 config
├── run_train_cli.py # Training CLI entry point
├── run_eval_cli.py # Evaluation CLI entry point
└── pyproject.toml # Package configuration
Measured per sample on MMBench (LLaVA-OV-7B):
| Method | Inference Time | Inference Memory | Training Memory | Training FLOPs |
|---|---|---|---|---|
| Baseline | 0.66s | 241.2 MB | 251.3 MB | 5.46T |
| CLI | 0.77s | 244.4 MB (+1.3%) | 277.6 MB (+10.5%) | 7.42T |
@article{chen2026cli,
title={From One-to-One to Many-to-Many: Dynamic Cross-Layer Injection for Deep Vision-Language Fusion},
author={Chen, Cheng and Guo, Yuyu and Zeng, Pengpeng and Song, Jingkuan and Di, Peng and Yu, Hang and Gao, Lianli},
booktitle={European Conference on Computer Vision (ECCV)},
year={2026}
}This project builds upon the following excellent open-source projects:
- LLaVA-NeXT - The foundational VLM framework
- LLaVA-OneVision - Baseline model architecture
- LMMs-Eval - Evaluation framework
- DeepSpeed - Distributed training optimization
This project is released under the Apache 2.0 License.