Visual workspaces improve reasoning accuracy in vision language models
VLM-in-Sandbox: Visual Workspaces for Agentic Visual Reasoning
Artificial Intelligence
Summary
Working with images and text together is tricky because computers need to remember visual clues from earlier steps without getting overwhelmed. The authors present VLM-in-Sandbox, a system that helps vision-language models manage visual information by keeping track of images and related evidence in an organized workspace. This method helps the models pick out and save important visual details without cluttering their memory, leading to better reasoning results in tests. Their approach works across several benchmarks and speeds up processing, showing the value of managing visual evidence separately.
What this means in practice
- •For software developers: Build interactive AI systems that process and reason about complex images step-by-step without overwhelming memory resources.
- •For computer vision engineers: Improve the performance of vision-language models by managing intermediate visual outputs explicitly during multi-step tasks.
Authors
Hexiong Yang, Mingrui Chen, Jie Cao, Ran He
Abstract
Sandboxed computer environments support multi-step reasoning with tools, executable programs, and persistent files, yet their extension from language models to vision-language models (VLMs) introduces a distinct state-management problem. Visual reasoning produces intermediate image-valued evidence---crops, masks, overlays, zoomed regions, and analytic renderings---that must remain addressable without accumulating unboundedly in multimodal context. We introduce VLM-in-Sandbox, a training-free framework for agentic multimodal reasoning in controlled computer environments. Its Visual Workspace registers generated artifacts in an image ledger, maintains a bounded active visual context, and lets the model explicitly promote selected evidence for subsequent inspection. This separates visual evidence generation, performed by sandbox tools, from visual evidence management. Across seven benchmarks and four base VLMs, VLM-in-Sandbox achieves the highest sample-weighted average accuracy among Vanilla VLM, Append-only Sandbox, and the proposed method. A compiler-matched $2\times2$ study on 1,260 examples further separates model-directed visibility from bounded retention: VLM-in-Sandbox reaches 66.27% accuracy with 18.6% fewer total tokens than the automatic, retain-all control. Over all 6,350 submitted GPT-4.1-mini examples, it produces 302 rescues and 142 regressions relative to Original Append-only. A local vLLM study with prefix caching confirms that the smaller request workload also reduces uncached tokens, time to first token, and end-to-end latency. These results identify explicit visual evidence state as a central abstraction for sandboxed VLM agents.