VC-Tooler: Learning Compositional and Adaptive Visual Tool Use
2026-08-03 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors introduce VC-Tooler, a model that helps visual language models (VLMs) use multiple visual tools more flexibly and effectively. Unlike earlier methods that rely on fixed tools and simple patterns, VC-Tooler learns to connect tools to images, combine several tools in sequence, and adjust its reasoning based on new visual information. They first build a set of example tool-using behaviors and then train the model step-by-step, starting with supervision and then reinforcement learning. Their approach shows better results on tests for general and complex visual tool use compared to other open-source models.
Visual Language ModelsMultimodal ReasoningVisual Tool UseTool CompositionReinforcement LearningSupervised LearningAgentic BenchmarkHierarchical Synthesis PipelineVisual Context Grounding
Authors
Yizheng Wu, Jiashen Hua, Bing Deng, Jieping Ye
Abstract
Agentic multimodal reasoning extends passive image understanding by allowing VLMs to actively acquire and refine visual evidence through visual tool interactions. Effective visual tool use requires three capabilities: grounding tool calls in visual context, composing tools across multiple steps, and adapting reasoning to tool-returned observations. However, existing approaches largely focus on grounding within fixed tool spaces and rigid invocation patterns, leaving composition and adaptation insufficiently addressed. We present VC-Tooler, which learns visual tool use as a compositional and adaptive capability. To this end, we first build a trajectory bank through a hierarchical synthesis pipeline covering three capability levels: single-tool grounding, multi-tool composition, and diverse tool contexts and interfaces. We then train the model in two stages: a supervised cold start that establishes these capabilities, followed by reinforcement learning that encourages accurate, efficient, and context-aware visual tool use. VC-Tooler achieves state-of-the-art performance among open-source models on both general-purpose and agentic benchmarks, including $95.8\%$ on V* and $35.3\%$ on VTC-Bench, and shows promising transfer under richer tool settings at inference time. Project page: https://w1zheng.github.io/VC-Tooler