V-Gym improves visual reasoning skills through adaptive practice loops
V-Gym: Enhancing Agentic Visual Reasoning via Skill-Data Co-Evolution
Computer Vision and Pattern Recognition
Summary
Visual reasoning agents sometimes repeat mistakes or fail to learn effectively when they have limited or unchanging practice data. The authors developed V-Gym, a system that helps these agents get better by automatically creating new practice problems tailored to their weaknesses while improving the skills they use to solve tasks. By repeating this cycle—updating both skills and practice data—agents can steadily improve their ability to handle diverse visual reasoning challenges. This process helps the agents learn from both successes and failures, making their skills more reliable and adaptable across different situations.
What this means in practice
- •For ai development teams: Enable agents to self-improve visual reasoning skills by generating targeted practice data and refining procedures autonomously.
- •For robotics engineers: Use adaptive skill and data co-evolution to improve robots’ performance on complex tasks requiring vision and reasoning.
Authors
Bei Yan, Yuecong Min, Jie Zhang, Junqi Yang, Shiguang Shan, Xilin Chen
Abstract
Advances in multimodal understanding, reasoning, and tool use enable agents to tackle increasingly complex visual reasoning tasks. By distilling past execution experience into reusable skills, agents can transfer lessons from both successes and failures into future reasoning, reducing repeated errors and improving capabilities. However, limited experience may produce unreliable, poorly generalizable skills, while static datasets may lack the targeted and diverse practice needed for refinement. To address this gap, we introduce V-Gym, an autonomous framework that iteratively co-evolves procedural skills and multimodal practice data from execution trajectories. During skill evolution, V-Gym analyzes trajectories to distill and refine hierarchical skills, updating procedural guidance and applicability conditions while retaining an update only if it improves validation performance. During data evolution, V-Gym selects generation seeds by balancing data utility and exploration, then translates trajectory-identified bottlenecks into diverse, targeted practice data that expand the data bank after quality checks. The resulting practice outcomes feed back into subsequent skill updates, closing the loop for continual skill refinement. Experiments across diverse multimodal reasoning benchmarks show substantial improvements over baselines with multiple backbone models. Its evolved skills generalize across domains and models, while evolved data support more effective skill refinement, enabling autonomous diagnosis, targeted practice, and continual self-improvement.