CoWorld Reasoner improves video based visual reasoning across tasks
CoWorld Reasoner: Agentic Multimodal Reasoning with the Evolving Video World Model-Driven Harness
Artificial Intelligence
Summary
Understanding what's happening in videos involves tracking motion, actions, and changes in view. The paper presents CoWorld Reasoner, a system that uses a video world model to imagine and create new visual clues to help answer questions about videos. This system can choose different ways to generate helpful images and use them to reason better than just looking at the original video. The authors show that this approach improves performance on several tests involving spatial, temporal, and causal reasoning.
What this means in practice
- •For video analysis teams: Enhance video content understanding by generating additional visual evidence for complex reasoning tasks beyond raw footage.
- •For robotic system developers: Improve robot perception and decision making by enabling flexible visual imagination to interpret dynamic scenes over time.
- •For security monitoring operators: Support better event detection from surveillance by supplying imagined views and causal visuals to supplement live video feeds.$Commercial implications: Makes possible advanced video reasoning features for security tools that can be marketed to surveillance service providers.
Authors
Dongjie Cheng, Ruifeng Yuan, Ran Wei, Ning Song, Yongqi Li, Wenjie Li
Abstract
As Vision-Language Models (VLMs) advance, they are increasingly expected to perform spatiotemporal reasoning over motion, actions, and view changes beyond what the input images show. Video World Models (VWMs) provide a natural unified engine for visual imagination by generating task-relevant visual content as additional evidence. However, existing approaches typically invoke a VWM through fixed pipelines centered on a predefined form of visual imagination, such as novel-view synthesis. To support flexible visual imagination for diverse reasoning tasks, we introduce CoWorld Reasoner, an agentic framework that uses a VWM as a unified visual reasoning engine. Its Video World Model-Driven Harness exposes multiple visual operations through tools and uses skills to guide the VLM in deciding whether to imagine, selecting an operation, constructing a generation request, and interpreting the resulting visual evidence. We further introduce VLM-VWM Harness Co-Evolution, which jointly revises tools, skills, and execution code based on execution feedback while keeping both model weights fixed. Across seven benchmarks spanning spatial intelligence, temporal and causal reasoning, and perceptual reasoning, CoWorld Reasoner outperforms direct reasoning with the same VLM. The strongest variant improves on all seven benchmarks, including relative accuracy gains of 11.8% on MMSI and 13.0% on ERQA, and outperforms all five evolving baselines using the same backbone across five benchmarks.