Visual agent learns to use tools by creating its own questions
Self-Evolving Visual Agent via Margin Reward
Artificial Intelligence
Summary
It can be hard to teach AI systems how and when to use external tools, especially because collecting example questions and correct answers is expensive. The authors propose Marvo, a method where one AI generates questions from unlabeled images, and another AI learns to use tools to answer them. Marvo encourages questions that are just hard enough for the AI to need tools but not too hard to solve, creating a self-improving learning loop. This helps the AI get better at reasoning tasks without needing human-labeled data.
What this means in practice
- •For software developers: Create AI that learns to solve complex visual problems by generating its own practice questions without needing labeled training data.
- •For robotics teams: Develop robots that improve visual reasoning and tool use skills autonomously from raw sensory inputs without manual instruction.
Authors
Wenxi Yang, Yuzhong Zhao, Baochang Zhang, Yunfan Liu, Qixiang Ye
Abstract
Visual agents that interleave reasoning with tool invocation have posed a promising direction for multimodal tasks. However, learning effective tool invocation usually relies on curated question-answer pairs or expert demonstrations at scale, which are expensive to acquire. In this study, we propose Marvo, a simple-yet-effective self-evolution approach that enables learning tool invocation from unlabeled images. Marvo consists of two agents: (1) a challenger that generates question-answer pairs from unlabeled images, and (2) a solver that learns tool invocation by solving questions through reinforcement learning. To encourage effective tool invocation, a good training question should be difficult enough to warrant tool use, yet remain within the solver's capabilities. To this end, we introduce a margin reward to the challenger, defined as solver's performance difference with and without access to tools. With the margin reward, we create an adaptive curriculum that encourages the solver to learn effective tool invocation. Across benchmarks covering agentic reasoning, visual search, mathematical, and general real-world reasoning, Marvo improves average accuracy by up to 7.1\% --- demonstrating margin reward as a preferred way to scale question complexity without human supervision.