Reframe improves visual agent learning by aligning observations with actions
Reframe: Observation-Aligned Multi-Teacher On-Policy Distillation for Agentic Visual Reasoning
Computer Vision and Pattern Recognition
Summary
Visual agents that use tools can miss important information and make mistakes that limit their learning after training. The authors propose Reframe, a new method that focuses on better connecting the agent's observations with its actions. Reframe has two parts: one trains the agent to pay attention to useful observations returned by tools, and the other fixes poor actions during learning to improve training. Testing across many visual tasks showed Reframe leads to better performance than previous methods.
What this means in practice
- •For robotics developers: Improve autonomous robots' use of tools by training them to better interpret sensor feedback and correct poor actions during learning.
- •For video game ai engineers: Enhance game agents’ visual decision-making by integrating observation-aware training and action repair techniques to boost in-game reasoning.
Authors
Haozhi Yuan, Handong Li, Yifan Zhang, Qianshan Wei
Abstract
Agentic reinforcement learning (RL) and multi-teacher on-policy distillation (MOPD) offer complementary routes to post-training. For visual agents, however, tool use presents two related challenges. Models may overlook useful tool-returned observations, while poorly grounded actions can limit the evidence available for subsequent distillation. We introduce Reframe, a post-training framework that treats the observation state, rather than only the output distribution, as an object of alignment. First, Ground-Drop-GRPO (GD-GRPO) trains spatial and temporal specialists with evidence-aware rewards and stochastic removal of the original visual input, encouraging them to acquire grounded evidence and use the returned observations. Second, Observation-Gated-MOPD (OG-MOPD) replaces deficient tool actions with canonical ones mid-rollout and resumes distillation from the repaired observation. The two stages complement each other, as learning to rely on tool-returned observations can help the policy benefit from observation repair. Neither stage requires a task-specific SFT cold start. Extensive experiments over 14 image and video benchmarks demonstrate that Reframe outperforms both Mix-RL and vanilla MOPD. Ablation results show that GD-GRPO increases observation dependence and tool gain compared with outcome-based GRPO.