[2026.07.08]Release of the dataset for SenseNova-Vision-Corpus-50M.[2026.07.08]Initial release of the weights for SenseNova-Vision-7B-MoT.[2026.07.08]Initial release of the inference code for SenseNova-Vision.[2026.07.08]Release of the Technical Report for SenseNova-Vision.
🚀 SenseNova-Vision formulate computer vision as unified multimodal generation, where heterogeneous visual tasks are expressed through the native text and image generation spaces of a unified multimodal model (UMM). Natural-language instructions and optional visual prompts specify the task, target regions or views, output schema, and decoding convention, while the model responds through native text, image, or mixed text-image generation.
Text generation expresses symbolic visual records such as categories, boxes, points, OCR strings, keypoints, and camera parameters. Image generation handles dense spatial targets such as segmentation masks, depth maps, surface normals, and multi-view point maps. Mixed responses support compositional tasks that combine symbolic and dense outputs. This shared formulation lets one model cover structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry while keeping outputs decodable for standard benchmarks.
To enable large-scale training, we convert heterogeneous computer-vision annotations into instruction-response examples and construct the SenseNova-Vision Corpus, spanning decodable text, image, and mixed text-image targets. Starting from an off-the-shelf pretrained UMM, SenseNova-Vision is trained primarily on this corpus, with auxiliary multimodal data used to preserve general understanding and generation capability, and requires no task-specific prediction heads, decoders, or architectural branches.
- 🔗 We introduce a unified multimodal generation formulation that casts heterogeneous computer vision tasks into the native input-output spaces of UMMs.
- 🧩 We construct the SenseNova-Vision Corpus, a large-scale computer-vision instruction-response corpus with decodable text, image, and mixed text-image targets.
- ✨ We train SenseNova-Vision and show strong results across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry, while supporting language-defined task variants beyond fixed benchmark schemas.
This repository provides one entrypoint for examples, single-image inference,
interactive inference, and benchmark inference. For the full runtime guide, see
docs/EVAL.md.
Create the environment from the repository root:
git clone https://github.com/OpenSenseNova/SenseNova-Vision.git
cd SenseNova-Vision
bash setup.sh sensenova-vision
conda activate sensenova-visionRun the curated example:
bash scripts/run_sensenova_vision.sh exampleRun one inference request:
bash scripts/run_sensenova_vision.sh inference \
binary_seg \
"person" \
examples/images/2.jpgLaunch the web demo. The wrapper prints the local URL before starting Gradio. Recommended: 1 x 80GB GPU for the full web demo.
MODEL_PATH=/path/to/SenseNova-Vision-7B-MoT \
bash scripts/run_sensenova_vision.sh demoRun the full benchmark after preparing datas/ and jsonl_generate/ according
to docs/data_prepare.md.
Recommended: at least one 8 x 80GB GPU machine for the full benchmark.
bash scripts/run_sensenova_vision.sh benchmark all \
--num_gpus 8 \
--tasks_per_gpu 2SenseNova-Vision is evaluated across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry. All tasks are formulated with natural-language instructions: textual outputs are parsed into benchmark-specific structures such as boxes, points, recognized text, keypoints, and camera parameters, while image outputs are decoded into masks, depth maps, normal maps, or 3D point maps.
Structured visual understanding evaluates tasks whose outputs can be represented as structured textual predictions, such as bounding boxes, points, recognized text, and keypoint coordinates.
| Method | Object Detection | OCR | GUI | Keypoint | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| COCO-Com. | HR/RefCOCOg V/T | LVIS | Dense200 | VisDrone | HierText | ICDAR15 | ScreenSpot-V2 | COCO-Kpt. | ||
| bbox | bbox | bbox | bbox | bbox | point | bbox | bbox | bbox | point | |
| Grounding DINO-Swin-T | 56.6 | 25.2 / 45.9 / 46.8 | 38.8 | 33.1 | 38.5 | -- | -- | -- | -- | -- |
| Bagel | 50.2 | 74.6 / 76.4 / 77.8 | 46.8 | 42.4 | 23.0 | 36.9 | 7.1 | 15.8 | 81.1 | -- |
| Qwen3-VL-8B-Instruct | 46.6 | 70.4 / 72.3 / 72.6 | 43.2 | 13.5 | 28.7 | 35.7 | 22.4 | 25.4 | 90.5 | -- |
| Qwen3.5-9B | 49.3 | 71.7 / 72.1 / 72.6 | 43.2 | 27.5 | 26.8 | 41.7 | 19.6 | 11.4 | 92.2 | -- |
| LocateAnything | 54.7 | 78.7 / 76.7 / 77.6 | 50.7 | 58.7 | 39.9 | 60.4 | 29.1 | 26.4 | 85.5 | -- |
| Rex-Omni | 52.9 | 79.9 / 73.6 / 74.3 | 46.9 | 58.3 | 35.8 | 58.9 | 28.0 | 28.1 | 88.4 | 32.6 |
| SenseNova-Vision | 56.6 | 80.2 / 79.6 / 80.5 | 54.8 | 66.8 | 43.3 | 62.9 | 31.2 | 49.5 | 85.9 | 34.6 |
Dense geometric prediction evaluates pixel-aligned geometric outputs, including monocular depth estimation and surface normal estimation.
| Method | Depth | Normal | ||||||
|---|---|---|---|---|---|---|---|---|
| NYUv2 | KITTI | ETH3D | ScanNet | DIODE | ScanNet | iBims-1 | NYUv2 | |
| AbsRel↓ / δ1↑ | Mean↓ / 11.25°↑ | |||||||
| DSINE | -- | -- | -- | -- | -- | 16.2 / 61.0 | 17.1 / 67.4 | 16.4 / 59.6 |
| DepthAnything | 4.3 / 98.1 | 7.6 / 94.7 | 12.7 / 88.2 | 4.3 / 98.1 | 26.0 / 75.9 | -- | -- | -- |
| DepthAnything V2 | 4.5 / 97.9 | 7.4 / 94.6 | 13.1 / 86.5 | 4.2 / 97.8 | 26.5 / 73.4 | -- | -- | -- |
| *MoGe-2 | 3.5 / 98.0 | 5.5 / 97.7 | 3.4 / 98.8 | 3.4 / 98.3 | 23.0 / 82.3 | 12.8 / 68.4 | 14.7 / 70.4 | 14.7 / 62.3 |
| Marigold | 5.5 / 96.4 | 9.9 / 91.6 | 6.5 / 95.9 | 6.4 / 95.2 | 30.8 / 77.3 | 21.3 / 45.6 | 18.5 / 64.7 | 20.9 / 50.5 |
| DICEPTION | 6.1 / 96.0 | 6.9 / 94.9 | 5.0 / 97.5 | 7.2 / 94.4 | 28.9 / 72.2 | 18.8 / 53.6 | -- | 18.3 / 52.9 |
| FE2E | 4.1 / 97.7 | 6.6 / 96.0 | 3.8 / 98.7 | 4.4 / 97.5 | 22.8 / 81.2 | 13.8 / 67.2 | 15.1 / 70.6 | 16.2 / 59.6 |
| Lotus-2 | 4.1 / 97.6 | 6.7 / 94.5 | 4.6 / 98.1 | 4.2 / 97.6 | 22.1 / 75.2 | 14.2 / 66.8 | 15.4 / 70.4 | 16.9 / 59.0 |
| SenseNova-Vision | 4.0 / 98.1 | 5.9 / 95.9 | 4.3 / 97.4 | 3.9 / 98.0 | 20.6 / 76.4 | 12.8 / 68.9 | 15.4 / 69.1 | 14.4 / 62.7 |
Segmentation evaluates mask prediction under semantic, referring, reasoning, grounded, and interactive guidance.
| Method | Gen. Seg. | Ref. Seg. | Rea. Seg. | GCG Seg. | Inter. Seg. |
|---|---|---|---|---|---|
| Pan. / Sem. | RefCOCO / + / g | Val / Test | Val / Test | Point / Box | |
| LISA-7B | -- | 74.9 / 65.1 / 67.9 | 52.9 / 47.3 | 62.0 / 61.7 | -- |
| PSALM | 55.9 / 66.6 | 83.6 / 72.9 / 73.8 | -- | -- | 64.3 / 67.3 |
| Text4Seg | -- | 79.2 / 72.8 / 74.0 | 59.1 / 57.1 | -- | -- |
| LENS | -- | 84.2 / 79.4 / 81.2 | 62.1 / 57.2 | -- | -- |
| ConverSeg | -- | 79.4 / 74.3 / 74.9 | 61.9 / 57.0 | -- | -- |
| X-SAM | 54.7 / 66.5 | 85.1 / 78.0 / 83.8 | 56.6 / 57.8 | 69.4 / 69.0 | 65.4 / 70.0 |
| SenseNova-Vision | 48.8 / 64.0 | 81.3 / 76.0 / 80.3 | 63.2 / 60.7 | 65.7 / 66.2 | 60.9 / 73.9 |
Multi-view visual geometry evaluates geometric prediction from multiple input images, including multi-view point map reconstruction and camera pose estimation.
| Method | Multi-View Reconstruction | Camera Pose | ||
|---|---|---|---|---|
| Acc.↓ / Comp.↓ / F1↑ | RRA@30↑ / RTA@30↑ / AUC@30↑ | |||
| 7Scenes | ETH3D | Re10K | CO3Dv2 | |
| DUSt3R | 0.026 / 0.034 / 87.1 | 0.359 / 0.531 / 66.6 | 99.8 / 84.9 / 67.6 | 97.7 / 93.4 / 78.3 |
| DepthAnything3 | 0.020 / 0.026 / 90.5 | 0.228 / 0.212 / 76.6 | 100.0 / 96.4 / 89.6 | 99.3 / 98.0 / 91.8 |
| VGGT | 0.023 / 0.032 / 88.4 | 0.177 / 0.155 / 80.9 | 100.0 / 93.5 / 79.3 | 98.3 / 96.6 / 89.2 |
| MoRe | 0.038 / 0.039 / 77.1 | 0.348 / 0.318 / 62.7 | 100.0 / 94.0 / 79.1 | 98.4 / 96.3 / 83.0 |
| MapAnything | 0.027 / 0.029 / 87.8 | 0.400 / 0.524 / 67.0 | 100.0 / 93.5 / 80.7 | 95.5 / 91.6 / 70.9 |
| G2VLM | 0.084 / 0.056 / 59.2 | 0.784 / 0.553 / 36.7 | 99.8 / 77.5 / 51.8 | 96.3 / 92.0 / 55.2 |
| SenseNova-Vision | 0.028 / 0.026 / 87.9 | 0.301 / 0.175 / 72.2 | 99.8 / 94.2 / 77.3 | 97.4 / 95.4 / 80.1 |
We further compare SenseNova-Vision with recent generalist visual models that span multiple visual capabilities.
| Method | Detection | Sem. Seg. | Ref. Seg. | Depth |
|---|---|---|---|---|
| mAP | mIoU | cIoU | δ1 | |
| COCO | Cityscapes | RefCOCO / + / g | NYUv2 | |
| Youtu-VL | 47.1 | 70.4 | 80.7 / 76.2 / 76.5 | 90.4 |
| SenseNova-Vision | 53.7 | 71.2 | 81.3 / 76.0 / 80.3 | 98.1 |
| Method | Sem. Seg. | Ref. Seg. | Rea. Seg. | Depth | Normal | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| mIoU | cIoU | gIoU | δ1 | Mean Error↓ | ||||||
| Cityscapes | RefCOCOg | ReasonSeg | KITTI | NYUv2 | DIODE | ETH3D | NYUv2 | ScanNet | DIODE | |
| Vision Banana | 69.9 | 73.8 | 79.3 | 91.5 | 94.8 | 91.7 | 93.5 | 17.8 | 15.1 | 13.8 |
| SenseNova-Vision | 71.2 | 80.3 | 63.2 | 95.9 | 98.1 | 76.4 | 97.4 | 14.4 | 12.8 | 15.3 |
Object Detection
| COCO-Com. | LVIS | Dense200 | VisDrone |
|---|---|---|---|
 |
 |
 |
 |
Referring Detection
 |
 |
OCR
| Textline Level | |
|---|---|
 |
 |
| Word Level | |
 |
 |
Visual Prompting
 |
 |
 |
 |
Layout Grounding
 |
 |
Keypoint Detection
| Human | |
|---|---|
 |
 |
| Animal | |
 |
 |
GUI Grounding
 |
 |
Dense Geometric Prediction
 |
Panoptic Segmentation
 |
 |
 |
 |
Semantic Segmentation
 |
 |
 |
 |
Referring Segmentation
 |
 |
 |
 |
Reasoning Segmentation
 |
 |
 |
 |
Grounded Conversation Generation Segmentation
 |
 |
 |
 |
Interactive Segmentation
| Point Prompt | |
|---|---|
 |
 |
| Box Prompt | Scribble Prompt |
 |
 |
SenseNova-Vision converts heterogeneous computer vision annotations into a common instruction-response schema. Each sample contains one or more visual inputs, a natural-language instruction that defines the task and output convention, and a decodable target represented as text, an image, or a mixed text-image response.
If you find SenseNova-Vision useful, please cite our technical report:
@misc{sensenova2026sensenovavision,
title={Vision as Unified Multimodal Generation},
author={Xiaoyang Han and Jianhua Li and Kewang Deng and Zukai Chen and Xuanke Shi and Sihan Wang and Boxuan Li and Linyan Wang and Siyi Xie and Xin You and Jinsheng Quan and Zhongang Cai and Haiwen Diao and Ziwei Liu and Lei Yang and Dahua Lin and Quan Wang},
year={2026},
eprint={2607.06560},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2607.06560},
}This project is released under the Apache 2.0 License.