OpenVLHarness boosts visual AI reasoning with tool-based memory
OpenVLHarness: Real World Visual Agentic Tool-Use with Multimodal Memory
Computation and LanguageArtificial IntelligenceComputer Vision and Pattern RecognitionMachine LearningMultimedia
Summary
Many visual AI tasks need information that a model can't figure out just by thinking alone. The authors created OpenVLHarness, a system that connects AI models to various tools and remembers the information they gather during a task. This memory includes pictures, data, and web links, which helps the model reason better over multiple steps. Tested on many tasks, it improved performance significantly compared to models working without these tools.
What this means in practice
- •For ai application developers: Create AI systems that solve complex visual problems by integrating multiple data sources and tools through persistent memory.
- •For digital content analysts: Use tool-enhanced AI agents to gather and interpret visual and structured data from diverse sources for detailed analysis tasks.
Authors
Yu Zhou, Cheng-Fu Yang, Rui Sun, Di Wu, Sibo Peng, Mingyang Zhang, Zhen Yang, Ruiyi Zhang, Nanyun Peng, Zhe Gan, Kai-Wei Chang
Abstract
Multimodal large language models (MLLMs) have made rapid progress on visual reasoning, yet many tasks require evidence that is difficult to obtain through internal reasoning alone: our controlled study on 22 benchmarks shows that simply increasing the inference budget of the base model yields only modest gains. Such tasks demand grounded perception, external knowledge, and numerical measurements obtained through heterogeneous tools, whose outputs must be made interpretable to the model and retained across reasoning steps. We introduce OpenVLHarness, a multimodal reasoning harness organized around a capability layer and persistent multimodal memory. The capability layer combines task-level tool interfaces with evidence rendering, providing consistent access to heterogeneous backends and converting their outputs into model-readable observations. Persistent multimodal memory retains the corresponding images, structured data, and web references as addressable artifacts throughout each query. A coding interface executes generated programs over stored data, enabling computations that combine results from multiple tools. Together, these mechanisms support an iterative process of acquiring, inspecting, and composing multimodal evidence. Evaluated on 22 benchmarks with eight open and proprietary backbones, OpenVLHarness improves average scores by 9.7-12.6 points over matched no-tool baselines, including +11.2 with Qwen3-VL-8B, +9.7 with Kimi K3, and +9.9 with GPT-6 Astra. With GPT-6 Luna at medium reasoning effort, OpenVLHarness outperforms the base model, Codex, and Visual Sketchpad at maximum effort by 7.7-8.5 points while incurring at most 37% of their model-token cost.