Interactive visual gym evaluates AI assistants in everyday tasks

Interactive Visual Gym: Evaluating Visual Interaction Models with Simulated Users in Interactive Worlds

Computer Vision and Pattern Recognition

Summary

Helping people with physical tasks using AI needs tests where the AI and human actions can affect each other. The authors built Interactive Visual Gym, which lets AI assistants work with simulated users in virtual worlds like kitchens and homes. Their tests showed current AI models struggle to give useful advice and act quickly enough during tasks. They confirmed their test results agree well with real human opinions and showed the system can help improve AI through practice.

What this means in practice

  • For ai assistant developers: Test and improve AI assistants by simulating realistic user interactions in everyday task environments like cooking or household work.
  • For mobile app developers: Evaluate and refine AI-powered interfaces by incorporating simulated mobile user behaviors and errors to enhance responsiveness and accuracy.

Authors

Chenhao Zheng, Zixian Ma, Kai Li, Kanchana Ranasinghe, Jieyu Zhang, Artemis Panagopoulou, Weikai Huang, Jianxu Shangguan, Yuanyuan Tan, Silvio Savarese, Juan Carlos Niebles, Ranjay Krishna, Ran Xu, Honglu Zhou

Abstract

Visual interaction models that support users during ongoing physical tasks require evaluation in closed loop, where assistant responses can change subsequent user behavior and task outcomes. Existing benchmarks largely rely on offline replay and therefore cannot capture these effects. We introduce Interactive Visual Gym, a framework for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users. We contribute three simulators that recreate everyday tasks and human-aligned action spaces across cooking, household manipulation, and mobile-interface domains, together with a persona-conditioned user simulation harness that collaborates with the assistant while incorporating controlled execution errors and other out-of-plan behaviors. We instantiate the framework as IVG-Bench, a benchmark with 315 interaction scenarios. Evaluating state-of-the-art VLMs as visual interaction models reveals substantial limitations: the best-performing model, Gemini 4 Argon, completes only 53.4% of tasks within the time limit, exhibits significant weaknesses in factual grounding and actionable guidance, and is often too slow to prevent errors when response latency is considered. We verify our evaluation protocol with a human study which finds 78.5% agreement between our interaction-quality metric and participant preferences. We further show that the environments provide useful learning signals: in a proof-of-concept GRPO experiment, closed-loop training improves both task success and interaction quality for Qwen3.5-4B model in our cooking environment.