A curated list of visual agents that perceive, reason, use tools, generate, and act.
This repository tracks high-signal work on agent systems that use visual observations to reason, call tools, create content, operate software, or take actions. It is intentionally selective: entries should help readers understand, build, evaluate, or deploy systems with an explicit agent loop.
- 2026-07-01 Major agent-only restructure: pruned non-agent VLM, generation, detection, segmentation, OCR, chart, and generic benchmark rows.
- 2026-07-01 Verified and expanded 2025-2026 GUI/CUA, visual tool-use, agentic AIGC, visual memory, and benchmark coverage from primary sources.
- 2026-07-01 Removed the old additions block and the standalone peripheral block; papers, benchmarks/datasets, and tools are kept in separate sections.
- About This List
- 1. Scope and Taxonomy
- 2. Agent Systems
- 3. Agentic AIGC and Creative Agents
- 4. Agentic Visual Perception and Grounding
- 5. Benchmarks and Evaluation
- 6. Tools and Engineering Resources
- 7. Workflow Stacks
- Related Lists
- Contributing
- Maintenance Policy
- Citation
- License
This list is a research and builder map, not a raw paper dump. It favors works that define a useful capability, benchmark, architecture, dataset, tool, or failure mode for visual agents.
| Dimension | Coverage |
|---|---|
| Primary scope | Visual agents, GUI/CUA agents, visual tool-use agents, agentic AIGC, visual memory, and agent-oriented grounding |
| Core audience | Researchers, students, builders, benchmark maintainers, and applied multimodal engineers |
| Source preference | Papers, official project pages, code repositories, datasets, benchmark pages, and technical reports |
| Inclusion bar | Clear contribution, inspectable source, and relevance to an agent loop: observe, reason, plan, use tools, act, remember, or self-correct |
| Exclusion bar | Generic model releases, pure foundation models, duplicate rows, marketing-only posts, unverifiable arXiv IDs, and one-shot demos with no reusable agent insight |
| Principle | Rule |
|---|---|
| Precision over volume | Prefer fewer strong entries over long unreviewed lists. |
| Primary sources first | Link to papers, official code, project pages, docs, or benchmark pages. |
| Separate roles | Keep papers, benchmarks, and engineering tools in distinct sections. |
| Benchmark hygiene | Put benchmarks, datasets, and evaluation suites in Benchmarks and Evaluation, not in model tables. |
| Agent relevance | Agent entries should involve observation, reasoning, planning, tool use, memory, control, or closed-loop evaluation. |
| AIGC relevance | AIGC entries should involve agentic planning, tool orchestration, iterative editing, critique, feedback, memory, or search. |
| Visual understanding relevance | Understanding entries should directly support GUI grounding, screen perception, tool-use reasoning, or action selection. |
Each paper table is designed for scanning: Work names the paper or system, Year gives the first public paper year, Links prioritizes primary sources, and Why it matters states the concrete contribution.
Included:
- GUI, web, desktop, and mobile agents that act from visual observations.
- Multimodal agents that use visual tools, memory, code, search, critique, or planning.
- AIGC systems only when they are agentic: multi-turn, tool-using, feedback-driven, self-correcting, memory-based, or search/planning based.
- Visual understanding methods only when they support grounding, screen perception, action selection, or agent control.
- Benchmarks, datasets, simulators, runtimes, and evaluation tools used to build or assess these systems.
Excluded by default:
- Plain image classification, OCR, captioning, VQA, chart QA, document parsing, detection, segmentation, or retrieval papers with no agent loop.
- Generic LLM/VLM releases without GUI control, tool use, action grounding, or agent evaluation.
- Pure text-to-image, text-to-video, image-editing, or controllable-generation papers unless the system itself is an agent.
Visual Agent Ecosystem
|-- 2. Agent Systems
| |-- GUI and computer-use agents
| |-- Web, desktop, and mobile environments
| |-- Visual tool-use and programmatic reasoning
|-- 3. Agentic AIGC and Creative Agents
| |-- Agentic planning, critique, and creative tool orchestration
|-- 4. Agentic Visual Perception and Grounding
| |-- GUI grounding and screen parsing
| |-- Agentic visual memory and active perception
| |-- Risk-aware and active visual perception for agents
|-- 5. Benchmarks and Evaluation
|-- 6. Tools and Engineering Resources
| Work | Year | Links | Why it matters |
|---|---|---|---|
| Large Multimodal Agents: A Survey | 2024 | paper | Broad entry point for multimodal agents, tools, memory, planning, and evaluation. |
| GUI Agents: A Survey | 2024 | paper | Focused overview of GUI-agent architectures, grounding, benchmarks, and failure modes. |
| A Comprehensive Survey of Agents for Computer Use | 2025 | paper | Maps the CUA stack from perception and planning to action execution and evaluation. |
| GUI Agents with Reinforcement Learning | 2026 | paper | Taxonomy and roadmap for offline, online, and hybrid RL in GUI-agent training. |
| GUI Agent Autonomy Levels | 2026 | paper | Six-level framework for comparing GUI-agent autonomy, task control, and deployment responsibility. |
| Agent S | 2024 | paper, code | Practical open framework for computer use with hierarchical planning and experience retrieval. |
| UI-TARS | 2025 | paper, desktop | Native GUI-agent model trained for screenshot perception, grounding, reasoning, and actions. |
| OpenCUA | 2025 | paper, code | Open data, annotation, training, and model stack for computer-use agents. |
| Work | Year | Links | Why it matters |
|---|---|---|---|
| MM-Navigator | 2023 | paper | Early GPT-4V-style smartphone navigation agent. |
| AppAgent | 2023 | paper, code | Mobile app agent that learns from exploration and demonstrations. |
| SeeAct | 2024 | paper, project | Web agent showing the importance of visual grounding in live websites. |
| SeeClick | 2024 | paper, code | Defines GUI grounding as a core bottleneck for screenshot-only GUI agents. |
| WebVoyager | 2024 | paper, code | End-to-end multimodal web agent evaluated on live websites. |
| Mobile-Agent | 2024 | paper, code | Vision-centric mobile agent with planning and visual perception tools. |
| OS-ATLAS | 2024 | paper | Foundation GUI action model with large-scale cross-platform grounding data. |
| ShowUI | 2024 | paper, code | Lightweight GUI action model for screenshot grounding and navigation. |
| Aguvis | 2024 | paper, project | Unified pure-vision GUI agent for autonomous cross-platform interaction. |
| Agent S2 | 2025 | paper, code | Generalist-specialist CUA framework with mixture-of-grounding and hierarchical planning. |
| LiteCUA | 2025 | paper, code | Treats the computer as an MCP server so agents can reason over structured state and actions. |
| UI-Evol | 2025 | paper | Plug-in knowledge-evolution module that retraces GUI trajectories and critiques external instructions. |
| MAPLE | 2025 | paper | Mobile GUI agent with persistent finite-state memory for planning, verification, rollback, and recovery. |
| Surfer-H / Holo1 | 2025 | paper | Cost-efficient open-weight web agent pairing Surfer-H with Holo1 UI-specialized VLMs. |
| DPO CUA | 2025 | paper | Uses LLM-as-judge trajectory signals to train smaller privacy-preserving local CUAs. |
| Mirage-1 | 2025 | paper, project | Cross-platform GUI agent with hierarchical multimodal skills and skill-augmented tree search. |
| GUIRoboTron-Speech | 2025 | paper, code | Speech-instruction GUI agent trained for screenshot grounding, planning, and action prediction. |
| UI-AGILE | 2025 | paper | Training and inference framework for GUI agents with RL rewards and decomposed high-resolution grounding. |
| CoAct-1 | 2025 | paper | Multi-agent CUA that dynamically routes subtasks between GUI control and code execution. |
| SEAgent | 2025 | paper | Self-evolving CUA that learns unfamiliar software through exploration, curriculum tasks, and experience. |
| ComputerRL / AutoGLM-OS | 2025 | paper | Online RL infrastructure for desktop agents with API-GUI actions and distributed virtual environments. |
| Mobile-Agent-v3 / GUI-Owl | 2025 | paper, code | Foundational GUI agent family spanning mobile, desktop, browser, planning, grounding, and environment RL. |
| WebSight | 2025 | paper | Vision-first autonomous web agent with planning, reasoning, vision-action, verification, and episodic memory. |
| UItron | 2025 | paper | Foundational GUI agent with data engineering, interactive infrastructure, perception, grounding, and planning. |
| UI-TARS-2 | 2025 | paper | Native GUI-centered agent model with data flywheel, stabilized multi-turn RL, and hybrid GUI environments. |
| Instruction Agent | 2025 | paper | GUI agent that extracts executable instructions from demonstrations with verifier and backtracker modules. |
| Ferret-UI Lite | 2025 | paper | Compact on-device GUI agent with visual tool-use, CoT reasoning, and RL for mobile, web, and desktop control. |
| UltraCUA | 2025 | paper | Hybrid-action CUA that alternates between GUI control and programmatic tool calls. |
| MGA | 2025 | paper | Memory-driven GUI agent that observes first, updates structured memory, and then decides actions. |
| HAR-GUI | 2025 | paper | History-aware reasoning framework that teaches GUI agents to use short-term trajectory context and reflective corrections. |
| EDAgent | 2025 | paper | Reflection-based GUI agent for electronic-design automation workflows in professional CAD-style software. |
| MAI-UI | 2025 | paper | Foundation GUI-agent family with device-cloud collaboration, self-evolving data, and online RL. |
| WebGym | 2026 | paper | Large-scale realistic training environment and RL recipe for visual web agents. |
| ColorBrowserAgent | 2026 | paper | Web agent with progress summarization and human-in-the-loop knowledge adaptation for long-horizon tasks. |
| Agent Alpha | 2026 | paper | Step-level MCTS framework that unifies generation, exploration, and evaluation for computer-use agents. |
| Mobile-Agent-v3.5 / GUI-Owl-1.5 | 2026 | paper, code | Multi-platform GUI-agent family with tool/MCP use, memory, multi-agent adaptation, and RL scaling. |
| SecAgent | 2026 | paper | Efficient multilingual mobile GUI agent using semantic context summaries for long interaction history. |
| MolmoWeb | 2026 | paper | Open visual web-agent family trained from browser demonstrations and web-GUI perception data. |
| ClawGUI | 2026 | paper | Full-stack framework for training, evaluating, and deploying GUI agents across devices. |
| UI-Copilot | 2026 | paper, code | Long-horizon GUI automation with retriever/calculator tools and policy optimization. |
| AgentLens | 2026 | paper | Mobile GUI agent that adapts user-facing visual modalities during background execution. |
| uxCUA | 2026 | paper | Trains CUAs to execute interaction flows and assess GUI usability from visual evidence. |
| Faithful-Agent | 2026 | paper | Mobile GUI agent trained for evidence-grounded and internally consistent actions. |
| Weblica | 2026 | paper | Reproducible web-replica environments for scaling visual web-agent RL. |
| EAM | 2026 | paper | Executable agentic memory with state-aware graph construction and value-guided search for long-horizon GUI automation. |
| MementoGUI | 2026 | paper | Plug-in multimodal memory control for long-horizon GUI trajectories. |
| PANDO | 2026 | paper | Online skill-distillation web agent with reflection, routing, visual compression, and cache-aware prompting. |
| UI-KOBE | 2026 | paper | Lightweight mobile GUI agent guided by autonomously explored app knowledge graphs. |
| Multi-Agent Computer Use | 2026 | paper, project | Manager-subagent CUA architecture for decomposition, parallel execution, and replanning. |
| OpenWebRL | 2026 | paper | Open online multi-turn RL framework for visual web agents on live websites. |
| Demo2Tutorial | 2026 | paper, code | Converts screen recordings and interaction logs into structured multimodal tutorials that improve GUI-agent planning. |
| AsyncWebRL | 2026 | paper | Asynchronous multi-step RL system that improves training throughput for visual web agents. |
| StainFlow | 2026 | paper | Entity-stain process reward model that tracks visual evidence flow for GUI-agent RL credit assignment. |
| GUI-AC | 2026 | paper | Continual-learning method that stabilizes GUI-agent RFT with grounding certainty. |
| HiViG | 2026 | paper, code | History-aware visually grounded critic for pre-execution CUA action checking. |
| ComAct | 2026 | paper | COM-as-action paradigm and self-correcting ComActor for professional software control. |
| AGMem | 2026 | paper | Action-grounded visual memory for GUI agents using local image crops tied to successful actions and recoveries. |
| LLM-as-Code | 2026 | paper | Agentic-programming harness that moves loop control into code for stable long visual operation sequences. |
| PreAct | 2026 | paper | Compiles successful CUA trajectories into screen-checked state-machine programs for repeated tasks. |
| RL with Autonomous Evaluation | 2026 | paper | Reinforcement learning for CUAs using autonomous vision-language evaluation as scalable reward feedback. |
| PEEU | 2026 | paper | Autonomous experience exploration and hindsight utilization for GUI-agent task planning. |
| GUICrafter | 2026 | paper, code | Weakly supervised GUI agent trained from massive unannotated screenshots plus small calibrated RL data. |
| Learning from Failure | 2026 | paper | Inference-time self-improvement loop that diagnoses failed CUA trajectories and patches agent behavior. |
| Xiaomi-GUI-0 | 2026 | paper | Real-device closed-loop mobile GUI agent with failure-driven data flywheel and agentic RL. |
| ATMem | 2026 | paper | Active task-driving memory that maintains workflow state for long-horizon mobile GUI agents. |
| Work | Year | Links | Why it matters |
|---|---|---|---|
| VISPROG | 2022 | paper, project | Visual programming approach that composes vision modules through executable steps. |
| Visual ChatGPT | 2023 | paper, code | Early system connecting ChatGPT-style interaction with visual foundation models. |
| ViperGPT | 2023 | paper, code | Uses generated Python programs to compose visual tools for reasoning. |
| Set-of-Mark Prompting | 2023 | paper, code | Simple visual marking strategy widely reused for agent grounding. |
| LLaVA-Plus | 2023 | paper, project | Trains multimodal assistants to select and invoke visual tools. |
| Visual Agentic RFT | 2025 | paper | Reinforcement fine-tuning for multimodal agents that browse, code, and manipulate images. |
| DeepEyes | 2025 | paper, code | Reinforcement learning for active visual reasoning and tool-like visual inspection. |
| VTool-R1 | 2025 | paper, code | Trains VLMs to interleave text with visual tool operations for multimodal chain-of-thought reasoning. |
| ChartAgent | 2025 | paper | Tool-augmented chart agent that decomposes questions into visual subtasks and interacts with chart images. |
| Orion | 2025 | paper | Unified visual agent that orchestrates detection, OCR, segmentation, geometry, and other tools for multi-step workflows. |
| CodeV | 2025 | paper | Code-based visual agent trained with tool-aware policy optimization for faithful image-operation use. |
| CiQi-Agent | 2026 | paper, dataset | Cultural-heritage visual agent with vision-tool invocation and multimodal retrieval for porcelain reasoning. |
| MTA-Agent | 2026 | paper | Open recipe for multimodal deep-search agents with verified multi-hop vision-language tool trajectories. |
| LMM-Searcher | 2026 | paper, code | Long-horizon multimodal search agent with file-based visual memory and on-demand image fetching. |
| POINTS-Seeker | 2026 | paper | Multimodal agentic search model trained from scratch with agentic seeding and visual history compression. |
| ProMMSearchAgent | 2026 | paper | Multimodal search agent trained with process-oriented rewards and sim-to-real transfer. |
| S1-VL | 2026 | paper | Scientific multimodal agent that actively manipulates images with code for visual reasoning. |
| HierVA | 2026 | paper | Hierarchical visual agent for chart reasoning with manager-worker planning and zoomed visual context. |
| ODE | 2026 | paper | On-policy data-evolution harness with reusable image-bank references for visual-native multimodal search agents. |
| MMSkills | 2026 | paper | Multimodal skill packages with visual evidence for reusable agent procedures. |
| VisHarness | 2026 | paper | Trainable visual agent that learns to route multi-turn reasoning through heterogeneous visual expert tools. |
| Diversity Over Frequency | 2026 | paper, project | Studies and regularizes visual tool-use diversity in visual chain-of-thought agents. |
| VESTA | 2026 | paper | Visual statistical-tool agent that creates and reuses diagnostic visualizations, tests, and data transforms. |
| Astra | 2026 | paper | Agentic spatial-reasoning framework that invokes a world simulator for imagined visual evidence. |
| IAPO | 2026 | paper | Attribution-aware RL method for improving tool use in small multimodal agents without brittle exact-match rewards. |
| PERIA | 2026 | paper | Tool-augmented visual agent for spatial reasoning across maps, probing, and reconstruction. |
| Visual-Seeker | 2026 | paper, code | Visual-native multimodal search agent that actively gathers fine-grained visual evidence. |
| S-Agent | 2026 | paper | Spatial tool-use agent for continuous multi-view and video reasoning. |
| ProMSA | 2026 | paper, code | Progressive multimodal search agent that selects image search, text search, or stopping under tool budgets. |
| Dynamo | 2026 | paper | Training-free vision-language agent that evolves reusable reasoning skills and executable visual tools. |
| TACO | 2026 | paper | Credit-assignment objective for multimodal code-tool agents that rewards useful visual tool calls. |
| SimpleSearch-VL | 2026 | paper | Lightweight multimodal agentic deep-search recipe with evidence-verified reasoning and self-summary. |
This section keeps only generation and editing work with an explicit agent loop: planning, tool use, critique, feedback, memory, search, self-correction, or multi-turn execution.
| Work | Year | Links | Why it matters |
|---|---|---|---|
| LLMGA | 2023 | paper | MLLM-based generation assistant for prompt refinement and image editing. |
| LLaVA-Plus | 2023 | paper, project | Connects visual understanding, generation, retrieval, and tool composition. |
| GenArtist | 2024 | paper, project | MLLM agent for image generation and editing through decomposition, tool use, and self-correction. |
| PlotEdit | 2025 | paper | Multi-agent chart-image editing system with extraction, code retrieval, instruction decomposition, and multimodal feedback. |
| ComfyGPT | 2025 | paper | Self-optimizing multi-agent system for generating and refining ComfyUI image-generation workflows. |
| RefineEdit-Agent | 2025 | paper | Training-free closed-loop image-editing agent with instruction parsing, planning, tool selection, and visual feedback. |
| Street Design Multi-Agent Pipeline | 2025 | paper | Multi-agent street-view redesign workflow with localization, prompt optimization, generation, and automated evaluation. |
| MIRA | 2025 | paper | Plug-and-play iterative image-editing agent that predicts atomic edits from visual feedback. |
| JarvisEvo | 2025 | paper, project | Self-evolving photo-editing agent with tool selection, interleaved multimodal CoT, and editor-evaluator optimization. |
| I2E | 2026 | paper | Decompose-then-action image-editing agent that turns object layers into actionable interactive environments. |
| MiLDEAgent | 2026 | paper | Reasoning-based multi-layer design-document editing agent with layer-aware understanding and targeted modifications. |
| GenAgent | 2026 | paper | Agentic multimodal model that invokes generators as tools and refines images through reasoning and reflection. |
| Agent Banana | 2026 | paper, code | Planner-executor image-editing agent for high-definition multi-turn workflows. |
| SAGE | 2026 | paper, project | Agentic 3D scene generator with critics, tool selection, and iterative self-refinement. |
| VisionCreator | 2026 | paper | Native visual-generation agentic model with understanding, thinking, planning, and creation. |
| VisionCreator-R1 | 2026 | paper | Reflection-enhanced visual-generation agent with reflection-plan co-optimization. |
| MSRAMIE | 2026 | paper | Training-free structured-reasoning agent for multi-instruction image editing with state and reference graphs. |
| WorldAgents | 2026 | paper | Multi-agent 3D world-synthesis pipeline with director, generator, and verifier roles. |
| GEMS | 2026 | paper, project | Agent-native multimodal generation framework with loop, memory, and skills. |
| Gen-Searcher | 2026 | paper, code | Search-augmented image-generation agent trained with SFT and agentic RL for knowledge-intensive prompts. |
| IMAGAgent | 2026 | paper, code | Multi-turn image-editing agent with constraint-aware planning, tool orchestration, and collaborative reflection. |
| Unify-Agent | 2026 | paper | World-grounded image-synthesis agent with multimodal evidence search, recaptioning, and generation trajectories. |
| AnomalyAgent | 2026 | paper | Tool-augmented anomaly-synthesis agent with retrieval, masking, quality evaluation, and self-reflection. |
| Generation Navigator | 2026 | paper | State-aware multi-turn image-generation agent trained with trajectory-level RL. |
| Code-as-Room | 2026 | paper | Agentic code-synthesis framework that converts top-down room images into executable Blender scenes. |
| GenEvolve | 2026 | paper, project | Self-evolving image-generation agents via tool-orchestrated visual experience distillation. |
| GenClaw | 2026 | paper | Code-driven agentic image generation with executable sketches as controllable intermediates. |
| IEA | 2026 | paper | Conversational image-editing agent that operates explicit parameterized tools with transparent edit traces. |
| Data2Story | 2026 | paper, project | Multi-agent newsroom that turns data into evidence-grounded multimodal stories with generated visual assets. |
| InterleaveThinker | 2026 | paper | Multi-agent planner-critic pipeline for interleaved text-image generation. |
| RS-Gen | 2026 | paper | Training-free reasoning and search-augmented image-generation agent with a closed-loop questioning-and-solving process. |
| Qwen-Image-Agent | 2026 | paper | Context-centric image-generation agent that plans, reasons, searches, remembers, and incorporates feedback before generation. |
| ManimAgent | 2026 | paper, project | Self-evolving multimodal agent that writes Manim animation code and stores visual reflection memories across tasks. |
This section keeps visual-understanding work only when it is designed for agents, screen control, action grounding, or risk-aware perception.
| Work | Year | Links | Why it matters |
|---|---|---|---|
| OmniParser | 2024 | paper, code | Parses screenshots into interactable regions for pure-vision GUI agents. |
| UGround | 2024 | paper, code | Pure-vision GUI grounding baseline without accessibility trees. |
| Phi-Ground | 2025 | paper, project | GUI-grounding model family tuned for CUA click localization and ScreenSpot-Pro-style deployment settings. |
| GUI-AIMA | 2025 | paper, code | Aligns intrinsic multimodal attention with GUI grounding signals for efficient coordinate-free click localization. |
| MEGA-GUI | 2025 | paper, code | Multi-stage GUI grounding agents with ROI zoom and context-aware query rewriting. |
| SafeGround | 2026 | paper | Uncertainty calibration for deciding when GUI grounding predictions are safe to trust. |
| See, Point, Refine | 2026 | paper, code | Closed-loop GUI grounding agent that refines cursor localization through visual feedback. |
| UI-in-the-Loop / UILoop | 2026 | paper | Explicit UI-element comprehension loop for screen-to-action reasoning. |
| AutoFocus | 2026 | paper | Uncertainty-aware active visual search for high-resolution GUI grounding. |
| PAGER | 2026 | paper | Point-precise geometric GUI control for pixel-level agent actions. |
| AQuaUI | 2026 | paper | Adaptive-quadtree visual token reduction for GUI-agent screenshots. |
| Work | Year | Links | Why it matters |
|---|---|---|---|
| IPR-1 | 2025 | paper, project | Interactive physical reasoner that uses world-model rollouts and action code for visual game-to-unseen transfer. |
| MuSEAgent | 2026 | paper | Multimodal reasoning agent with a quality-filtered stateful-experience bank for adaptive retrieval. |
| Visual Agentic Memory | 2026 | paper, code | Training-free long-video memory with online indexing, hierarchical storage, and agentic retrieval. |
| CoSee | 2026 | paper | Audits shared-state collaboration failures in resource-constrained visual agents through read-write-verify traces. |
| VisualClaw | 2026 | paper | Self-evolving personalized multimodal agent for streaming visual evidence, memory, and skill-bank updates. |
| DualMem | 2026 | paper, code | Dual visual-verbal memory architecture for multimodal agents that must remember pixel-only cues. |
| TASKER | 2026 | paper, code | Task-driven keyframe selector for video-guided GUI agents and long-horizon procedural transfer. |
| Area | Benchmark | Links | Primary use |
|---|---|---|---|
| Web agents | Mind2Web | paper, project | Offline web action prediction and grounding. |
| Web agents | WebArena | paper, code | Realistic web tasks with execution-based grading. |
| Web agents | VisualWebArena | paper, code | Visually grounded web tasks. |
| Web agents | WebLINX | paper, project | Conversational web navigation demonstrations. |
| Enterprise agents | WorkArena | paper, code | Enterprise workflow automation. |
| Desktop/web agents | OmniACT | paper | Benchmark for executable automation scripts across desktop and web tasks. |
| Web agents | WebClick | paper | Web UI localization benchmark introduced with Surfer-H and Holo1. |
| Visual web agents | WebGym | paper | Large-scale realistic web-agent training and evaluation environment with rubric-based tasks. |
| Web agents | WebChain / WebChainBench | paper | Human-annotated real-world web interaction traces with aligned visual, structural, and action data. |
| Web agents | Ego2Web | paper | Connects egocentric video understanding with online web-agent execution tasks. |
| Desktop agents | OSWorld | paper, code | Open-ended desktop tasks in real OS environments. |
| Desktop agents | OSWorld 2.0 | paper | Long-horizon real-world computer-use workflows with partial scoring and execution auditing. |
| Desktop agents | Windows Agent Arena | paper, code | Windows-specific OS-agent evaluation. |
| Desktop agents | MacArena | paper | Online macOS benchmark running on Apple Silicon virtualization. |
| Desktop agents | GUI-360 | paper, dataset | Large-scale Windows CUA dataset and benchmark for grounding, screen parsing, and action prediction. |
| Desktop agents | GUI vs. CLI | paper | Matched execution-layer benchmark comparing screen-only GUI agents with skill-mediated CLI agents. |
| Hybrid-interface agents | WeaveBench | paper | Long-horizon tasks requiring coordinated GUI, CLI, code, browser, and external-tool use. |
| Professional software agents | ComCADBench | paper | Industrial CAD benchmark for professional software agents using GUI and COM-style executable actions. |
| Professional software agents | GUI-EDA | paper | Electronic-design-automation GUI dataset and benchmark for CAD-style professional workflows. |
| Presentation agents | PPT-Eval | paper, project | PowerPoint creation and editing benchmark with rubric-based partial credit. |
| GUI design for agents | AUI-Gym | paper, code | Evaluates whether CUAs can judge and improve automatically generated GUI applications. |
| Desktop agents | OpenComputer | paper | Verifiable software worlds with state verifiers and auditable rewards. |
| Desktop agents | Workflow-GYM | paper | Long-horizon GUI workflows in professional software domains. |
| Desktop agents | MyPCBench | paper, project | Personal-computer tasks over seeded apps and user-specific context. |
| Scientific instruments | LabOSBench | paper | Web-based scientific-instrument control benchmark for multimodal GUI agents. |
| Desktop data | ProCUA-SFT | paper | Large-scale synthetic SFT data for desktop computer-use agents. |
| Mobile agents | Android in the Wild | paper | Android action traces from real-world use. |
| Mobile agents | AndroidWorld | paper, code | Dynamic Android tasks with reproducible rewards. |
| Mobile agents | AndroidLH | paper, project | Long-horizon Android benchmark introduced with Mirage-1 for online GUI-agent planning. |
| Enterprise agents | SaaS-Bench | paper, code | Deployable SaaS systems and professional workflow evaluation. |
| Enterprise readiness | UI-CUBE | paper | Enterprise-readiness benchmark beyond simple task success. |
| Living-screen GUI | LivingScreen | paper, code | Short-video-platform GUI tasks where the screen changes continuously between actions. |
| Omni-modal mobile GUI | OmniGUI | paper, project | Smartphone GUI-agent benchmark with static screenshots, audio cues, and temporal video dynamics. |
| Video-guided GUI | VG-GUIBench | paper, project | Tests whether GUI agents can learn procedures from video tutorials and act in software. |
| Long-horizon GUI | MementoGUI-Bench | paper | Memory consistency and decision-making over long GUI trajectories. |
| Document-guided GUI | DocOS | paper | Agents must search documentation and ground procedures into GUI actions. |
| GUI exploration | ScreenSearch | paper | Ambiguity-aware desktop GUI state-graph exploration. |
| GUI grounding | ScreenSpot | paper, code | Mobile, desktop, and web grounding. |
| GUI grounding | ScreenSpot-Pro | paper, leaderboard | High-resolution professional screen grounding. |
| GUI grounding | PAGE Bench | paper | Precision-sensitive geometric GUI tasks with pixel-level actions. |
| GUI grounding | Precision-CUA-Bench | paper, code | Pixel-precise multi-turn cursor localization for software-engineering agents. |
| GUI grounding | DragOn | paper | Drag, swipe, highlight, resize, and slider grounding tasks for GUI agents. |
| GUI uncertainty | Argus | paper | Post-hoc uncertainty quantification benchmark for executable GUI grounding in CUAs. |
| Web-agent behavior | VAF | paper | Controlled evaluation of how visual interface attributes influence web-agent decisions. |
| CUA stability | Repeated Execution Reliability | paper | Evaluates whether CUAs remain stable across repeated executions of the same task. |
| Visual tool-use | VisualToolBench | paper | Tool-enabled image perception, transformation, and reasoning benchmark for thinking with images. |
| Multimodal agentic reasoning | Agentic-MME | paper | Process-verified benchmark for visual expansion, knowledge expansion, tool use, and intermediate-state auditing. |
| Multimodal search | InterLV-Search | paper, code | Interleaved language-vision agentic search with reusable textual and visual evidence. |
| Domain visual agents | CiQi-Bench | paper, dataset | Porcelain connoisseurship benchmark requiring visual-tool and retrieval-augmented cultural reasoning. |
| Visual statistical agents | DAWN | paper | Automated-workflow benchmark for VESTA-style visual statistical tool agents. |
| Visual memory/video agents | VisualClawArena | paper | Agentic benchmark requiring video evidence, documents, dynamic updates, and executable checks. |
| Visual memory | DMV-Bench | paper, code | Interactive visual-memory benchmark with pixel-only incidental cues for multimodal agents. |
| Collaborative multimodal agents | GPTNT | paper, project | Real-time multimodal collaboration benchmark based on Keep Talking and Nobody Explodes. |
| Interactive spatial agents | SpatialWorld | paper | Vision-only partial-observability benchmark for active spatial evidence gathering and long-horizon planning. |
| Agentic generation workflows | FlowBench | paper | ComfyUI workflow-generation benchmark for multi-agent visual-generation workflow builders. |
| Agentic generation | CIGEval | paper | Agentic evaluation framework for conditional image generation. |
| Agentic editing | LongBench-T2I-Edit | paper | Complex multi-turn image-editing benchmark for closed-loop editing agents. |
| Agentic editing | ArtEdit-Bench | paper, project | Photo-editing benchmark for self-evolving editor-evaluator agents. |
| Agentic editing data | MIRA-Editing | paper | Tool-use dataset for iterative image-editing agents with visual feedback. |
| Agentic document editing | MiLDEBench / MiLDEEval | paper | Multi-layer design-document editing benchmark and evaluation protocol. |
| Agentic editing | MTEditBench | paper | Multi-turn image-editing benchmark introduced with IMAGAgent. |
| Agentic generation | KnowGen | paper, project | Search-grounded image-generation benchmark for knowledge-intensive prompts. |
| Agentic generation | FactIP | paper | World-grounded image-synthesis benchmark for long-tail factual concepts. |
| Agentic generation | GenEvolve-Bench | paper, project | Multi-turn tool-orchestrated image-generation evaluation. |
| Agentic generation | IA-Bench | paper | Evaluates image-agent capabilities across planning, reasoning, search, and memory. |
| Resource | Type | Link | Primary use |
|---|---|---|---|
| Agent S | runtime | GitHub | Open computer-use agent framework. |
| UI-TARS Desktop | desktop agent | GitHub | Running GUI agents on local desktops. |
| ShowUI | GUI model | GitHub | GUI action model and datasets for screenshot agents. |
| Aguvis | GUI model | Project | Pure-vision GUI-agent models, datasets, and recipes. |
| LiteCUA | MCP CUA | GitHub | Computer-as-MCP-server environment contextualization for CUAs. |
| UI-Copilot | GUI agent | GitHub | Long-horizon GUI agent with tool-integrated policy optimization. |
| OpenCUA | CUA stack | GitHub | Annotation, data, training, and evaluation for computer-use agents. |
| UItron | GUI agent | Paper | Data engineering and interactive infrastructure for foundational GUI agents. |
| UI-TARS-2 | GUI agent | Paper | Multi-turn RL and hybrid GUI-environment training for native GUI agents. |
| UI-AGILE | GUI-agent training | Paper | RL and high-resolution grounding recipe for GUI agents. |
| Mirage-1 | GUI agent | Project | Hierarchical multimodal skills and online tree search for long-horizon GUI tasks. |
| Instruction Agent | demonstration agent | Paper | Converting expert demonstrations into verifiable GUI-agent instructions. |
| MGA | GUI memory | Paper | Observation-centric structured memory for GUI interaction. |
| CoAct-1 | hybrid CUA | Paper | Routing CUA subtasks between GUI actions and code execution. |
| Mobile-Agent | mobile GUI agent | GitHub | Mobile and multi-platform GUI-agent implementations. |
| GUIRoboTron-Speech | speech GUI agent | GitHub | Speech-instruction GUI action prediction from screenshots. |
| ClawGUI | GUI-agent stack | Paper | Unified training, evaluation, and deployment framework for GUI agents. |
| Agent Alpha | CUA planner | Paper | MCTS-style generation, exploration, and evaluation for CUAs. |
| EAM | GUI memory | Paper | Executable graph memory and value-guided search for long-horizon GUI automation. |
| Demo2Tutorial | tutorial distillation | GitHub | Turning screen recordings into multimodal tutorials for GUI agents. |
| ComAct | software agent | Paper | COM-as-action control for professional desktop software. |
| LLM-as-Code | CUA harness | Paper | Code-governed loop control for stable long CUA trajectories. |
| HiViG | critic | GitHub | History-aware visually grounded action critic for CUAs. |
| MolmoWeb | visual web agent | Paper | Open visual web-agent models, data mixture, and evaluation harness. |
| WebGym | web-agent environment | Paper | Realistic visual web-agent RL tasks and rollout infrastructure. |
| OpenWebRL | web-agent RL | Paper | Online multi-turn RL pipeline for live visual web agents. |
| AsyncWebRL | web-agent RL | Paper | Asynchronous rollout and training system for visual web agents. |
| Weblica | web-agent environment | Paper | Reproducible cached web replicas for scalable visual web-agent training. |
| GUICrafter | GUI-agent training | GitHub | Weakly supervised GUI-agent training from unannotated screenshots. |
| EDAgent | professional GUI agent | Paper | Reflection-driven GUI control for electronic-design automation software. |
| uxCUA | usability CUA | Paper | CUA-based GUI usability assessment through executed interaction flows. |
| BrowserGym | browser harness | GitHub | Reproducible browser-agent experiments. |
| AgentLab | experiment framework | GitHub | Running and comparing web-agent experiments. |
| Playwright MCP | browser MCP | GitHub | Browser automation tools exposed through MCP. |
| Stagehand | browser runtime | GitHub | Natural-language plus code browser automation. |
| OpenAdapt | desktop automation | GitHub | Recording and replaying GUI demonstrations. |
| ScreenPipe | local capture | GitHub | Capturing screen and audio context for local agents. |
| Cua | operator stack | GitHub | Infrastructure for computer-use agents and sandboxed desktops. |
| Resource | Type | Link | Primary use |
|---|---|---|---|
| OmniParser | GUI parser | GitHub | Screenshot parsing for pure-vision GUI agents. |
| UGround | GUI grounding | GitHub | Pure-vision grounding without accessibility trees. |
| Phi-Ground | GUI grounding | Project | Grounding model family for CUA click localization. |
| GUI-AIMA | GUI grounding | GitHub | Attention-aligned coordinate-free GUI grounding. |
| MEGA-GUI | GUI grounding | GitHub | Multi-stage grounding agents with ROI zoom and query rewriting. |
| SafeGround | grounding calibration | Paper | Risk-aware confidence calibration before GUI actions. |
| See, Point, Refine | closed-loop grounding | GitHub | Visual-feedback loop for pixel-precise CUA cursor localization. |
| AutoFocus | active perception | Paper | Training-free active search for high-resolution GUI grounding. |
| PAGER | GUI control | Paper | Point-precise geometric GUI control for pixel-level actions. |
| AQuaUI | token reduction | Paper | Adaptive screenshot compression for GUI agents. |
| AGMem | GUI visual memory | Paper | Action-grounded crop memory for GUI agents. |
| ChartAgent | visual tool agent | Paper | Chart-specific visual actions such as crop, annotate, localize, and decompose. |
| CodeV | code visual agent | Paper | Faithful visual reasoning through executable image operations and tool-aware rewards. |
| CiQi-Agent | domain visual agent | Dataset | Tool-augmented porcelain reasoning data and benchmark resources. |
| LMM-Searcher | multimodal search | GitHub | Long-horizon multimodal search with file-based visual memory. |
| POINTS-Seeker | multimodal search | Paper | Agentic seeding and visual-history compression for multimodal search. |
| ODE | search-agent data | Paper | On-policy data evolution for visual-native search agents. |
| Astra | simulator tool-use | Paper | World-simulator calls for imagined spatial evidence. |
| VESTA | statistical tool agent | Paper | Dynamic visual/statistical tool creation for data-model exploration. |
| Agentic-MME | agentic eval | Paper | Process-level evaluation of visual and knowledge tool expansion. |
| TACO | tool-use RL | Paper | Credit assignment for useful multimodal code-tool calls. |
| Visual Agentic Memory | visual memory | GitHub | Online indexing, hierarchical memory, and retrieval for long-video agents. |
| MuSEAgent | experience memory | Paper | Stateful multimodal experience bank for adaptive reasoning guidance. |
| TASKER | keyframe search | GitHub | Task-driven keyframe extraction for video-guided GUI agents. |
| DualMem | visual memory | GitHub | Visual-verbal memory for multimodal agents with pixel-grounded recall. |
| IPR-1 | active reasoning | Project | Interactive physical reasoning with world-model rollouts and action code. |
| Resource | Type | Link | Primary use |
|---|---|---|---|
| LLaVA-Plus | visual tool-use agent | Project | Tool selection across visual understanding, retrieval, and generation. |
| GenArtist | creative agent | Project | Decomposed image generation and editing with self-correction. |
| PlotEdit | chart-editing agent | Paper | Multi-agent chart editing from natural language over PDF or scanned chart images. |
| ComfyGPT | workflow agent | Paper | Self-optimizing multi-agent ComfyUI workflow generation. |
| RefineEdit-Agent | image-editing agent | Paper | Closed-loop iterative image editing with visual feedback. |
| MIRA | image-editing agent | Paper | Iterative perception-reasoning-action loop for image editing. |
| I2E | image-editing agent | Paper | Object-layer action environment for compositional image editing. |
| MiLDEAgent | design-editing agent | Paper | Layer-aware editing for multi-layer design documents. |
| GenAgent | generation agent | Paper | Generator-as-tool agentic image generation. |
| MSRAMIE | image-editing agent | Paper | Structured multimodal reasoning for multi-instruction image editing. |
| Gen-Searcher | search-generation agent | GitHub | Search-augmented image generation for knowledge-intensive prompts. |
| Unify-Agent | grounded generation agent | Paper | Multimodal evidence search, recaptioning, and grounded image synthesis. |
| Agent Banana | image-editing agent | GitHub | Planner-executor multi-turn image editing. |
| JarvisEvo | photo-editing agent | Project | Self-evolving editor-evaluator optimization for photo editing. |
| IMAGAgent | image-editing agent | GitHub | Multi-turn editing with planning, tool orchestration, and reflection. |
| SAGE | 3D scene-generation agent | Project | Agentic simulator-ready 3D scene generation. |
| GEMS | multimodal generation agent | Project | Agent loop, memory, and skills for multimodal generation. |
| GenEvolve | agentic generation | Project | Tool-orchestrated image-generation agent and benchmark resources. |
| GenClaw | code-driven generation | Paper | Executable sketches as controllable intermediates for image generation. |
| IEA | image-editing agent | Paper | Parameterized editing tools with transparent conversational edit traces. |
| Data2Story | multimodal story agent | Project | Multi-agent data storytelling with evidence-grounded generated visuals. |
| InterleaveThinker | interleaved generation | Paper | Planner-critic multi-agent pipeline for text-image sequence generation. |
| RS-Gen | generation agent | Paper | Reasoning and search-augmented image-generation framework. |
| Qwen-Image-Agent | image-generation agent | Paper | Context-aware planning, search, memory, and feedback for image generation. |
| ManimAgent | animation agent | Project | Self-evolving multimodal agent for Manim animation code generation. |
| Goal | Practical starting stack |
|---|---|
| GUI grounding research | ScreenSpot + ScreenSpot-Pro + PAGE Bench + Precision-CUA-Bench + DragOn + SeeClick + OS-ATLAS + ShowUI + OmniParser + UGround + Phi-Ground + GUI-AIMA + MEGA-GUI + SafeGround + See, Point, Refine + AutoFocus + Argus |
| Web-agent experiments | BrowserGym + AgentLab + WebArena + VisualWebArena + WebLINX + WebClick + WebGym + WebChain + Weblica + Ego2Web + OpenWebRL + AsyncWebRL + Surfer-H/Holo1 + WebSight + MolmoWeb + ColorBrowserAgent + PANDO + Multi-Agent Computer Use |
| Desktop CUA experiments | OSWorld + GUI-360 + OSWorld 2.0 + Windows Agent Arena + MacArena + GUI vs. CLI + WeaveBench + PPT-Eval + Agent S/S2 + UI-TARS Desktop + UI-TARS-2 + OpenCUA + LiteCUA + UItron + UI-AGILE + Mirage-1 + CoAct-1 + Instruction Agent + MGA + SEAgent + ComputerRL + UltraCUA + MAI-UI + Ferret-UI Lite + Agent Alpha + EAM + AGMem + ComAct + LLM-as-Code + StainFlow + OpenComputer + Workflow-GYM + PreAct + Learning from Failure |
| Personal and scientific CUA | MyPCBench + MementoGUI + ATMem + ScreenSearch + OpenComputer + LabOSBench + ProCUA-SFT + DocOS + AUI-Gym + LivingScreen + VG-GUIBench + VisualClawArena + GUI-EDA + EDAgent + uxCUA + Repeated Execution Reliability |
| Mobile-agent experiments | Android in the Wild + AndroidWorld + AndroidLH + OmniGUI + AppAgent + Mobile-Agent + Mirage-1 + GUIRoboTron-Speech + Mobile-Agent-v3/GUI-Owl + GUI-Owl-1.5 + Xiaomi-GUI-0 + GUICrafter + UI-KOBE + SecAgent + Faithful-Agent + AgentLens + ClawGUI + Aguvis + UI-Copilot + Ferret-UI Lite |
| Visual tool-use and search | VISPROG + ViperGPT + LLaVA-Plus + DeepEyes + Visual Agentic RFT + VTool-R1 + ChartAgent + CodeV + CiQi-Agent + Orion + MTA-Agent + LMM-Searcher + POINTS-Seeker + ODE + ProMMSearchAgent + MMSkills + VisHarness + VESTA + Astra + IAPO + PERIA + Visual-Seeker + S-Agent + ProMSA + Dynamo + TACO + SimpleSearch-VL + VisualToolBench + Agentic-MME + InterLV-Search |
| Visual memory and active perception | IPR-1 + MuSEAgent + CoSee + Visual Agentic Memory + AGMem + TASKER + VisualClaw + DualMem + DMV-Bench + VG-GUIBench + SpatialWorld + GPTNT |
| Agentic image, editing, and 3D creation | LLaVA-Plus + GenArtist + PlotEdit + ComfyGPT + RefineEdit-Agent + Street Design Multi-Agent Pipeline + MIRA + I2E + MiLDEAgent + GenAgent + MSRAMIE + Gen-Searcher + Unify-Agent + Agent Banana + JarvisEvo + IMAGAgent + SAGE + WorldAgents + VisionCreator + GEMS + Code-as-Room + GenEvolve + GenClaw + IEA + Data2Story + InterleaveThinker + RS-Gen + Qwen-Image-Agent + ManimAgent |
| Repository | Link | Notes |
|---|---|---|
| Awesome Multimodal Modeling | GitHub | Reference style and broader multimodal-modeling taxonomy. |
| Awesome-GUI-Agents | GitHub | Focused list for GUI agents and GUI grounding. |
| GUI-Agents-Paper-List | GitHub | Systematic paper list for GUI agents. |
| Awesome Multimodal Large Language Models | GitHub | Broader MLLM background; not the inclusion scope of this list. |
| Evolving Visual Generation | GitHub | Broader visual-generation background; only agentic entries belong here. |
Pull requests are welcome when they improve precision, coverage, or maintainability.
Recommended entry format:
| Work | Year | Links | Why it matters |
| --- | --- | --- | --- |
| Paper or project name | 2026 | [paper](...), [code](...) | One sentence explaining the agent loop, tool use, action grounding, memory, critique, or closed-loop evaluation relevance. |Before adding an item, check:
- Does it fit visual agents, agentic AIGC, agentic visual perception, benchmark, or tooling?
- Is there a primary source?
- Is it meaningfully different from entries already listed?
- Does the one-sentence relevance explain why a reader should care?
- Would it still belong here if all generic VLM, generation, OCR, detection, and segmentation papers were excluded?
- Keep the list selective. Do not add every adjacent arXiv paper.
- Remove duplicate rows and weakly scoped papers during each cleanup pass.
- Prefer primary sources over secondary summaries.
- Avoid future-looking claims unless the source is public and inspectable.
- Keep research papers, benchmarks, and tools separated.
- Re-check links and arXiv IDs before accepting large batches.
If this curated index helps your research or engineering work, you can cite it as:
@misc{awesome_visual_agent,
title = {Awesome Visual Agent},
author = {OpenEnvision},
year = {2026},
howpublished = {\url{https://github.com/OpenEnvision/Awesome-Visual-Agent}},
note = {Curated list of visual agents, agentic AIGC workflows, and agent-oriented visual grounding systems}
}Please follow the repository license when reusing this list. If no license file is present, add one before formal redistribution.