Video in-context learning struggles in multimodal interactive agents
V-ICAL Bench: Evaluating Video In-Context Learning for Multimodal Agents in Interactive Environments
Computer Vision and Pattern Recognition
Summary
Using example videos to teach AI agents how to act in interactive environments is hard. The authors created a large test called V-ICAL to see how well these agents learn from video demonstrations. They found that current AI models perform poorly, far below humans, and often can’t turn what they see in videos into good actions. This shows there is a big challenge for teaching AI using videos in dynamic settings. The authors highlight the need for better methods in this area.
What this means in practice
- •For robotics developers: Assess and improve robot control algorithms that learn from video examples in varied real-world tasks using the V-ICAL benchmark.
- •For game ai engineers: Evaluate game agents’ ability to learn new strategies from gameplay video snippets across many environments with V-ICAL.
Authors
Ziqian Fan, Shibo Xu, Junjie Li, Xiangyu Zhao, Shengyuan Ding, Yifan Yang, Zhenjie Yang, Haodong Duan, Yue Zhou, Zhihang Zhong, Xue Yang
Abstract
While In-Context Learning (ICL) enables models to adapt from exemplars without parameter updates, multimodal ICL remains largely underexplored, particularly regarding video demonstrations in interactive environments. For multimodal agents, learning from videos presents unique challenges: they must translate in-context demonstrations into executable policies, ground these policies in novel visual states, and iteratively refine actions based on environmental feedback. We introduce V-ICAL, a novel benchmark designed to evaluate video-based ICL for multimodal agents. Comprising 342 interactive tasks across 37 environments, V-ICAL utilizes human-curated demonstration videos as task-specific behavioral exemplars, evaluating agents through sustained interaction from a target initialization. The benchmark seamlessly connects in-context knowledge induction with core agentic capabilities, including state grounding, temporal memory, planning, and adaptation in dynamic environments. Extensive evaluations across 19 state-of-the-art multimodal agents reveal significant limitations: the best-performing model, Seed-2.1-Pro, achieves a score of only 54.4/100, while other leading models (e.g., Gemini-3.1-Pro, GPT-5.6) fail to surpass 50, far below the human baseline of 83.6. Controlled comparisons further demonstrate that current agents struggle to reliably translate video exemplars into effective policies, failing to yield consistent performance gains. Ultimately, V-ICAL exposes a critical gap in the ICL capabilities of multimodal agents, underscoring an urgent need for future research.