Synthetic framework improves reasoning across multiple videos
SYNCR: Diagnosing and Learning Cross-Video Reasoning from Simulation
Computer Vision and Pattern RecognitionMachine Learning
Summary
Figuring out connections and comparisons across different videos is tough for AI because it needs to match events, recognize the same people or objects, and integrate incomplete information. The authors created SYNCR, a testing system with simulated videos and thousands of questions to evaluate these skills in AI models. They found many large AI models struggle with comparing physical traits and understanding whole scenes, even if the models are larger. Fine-tuning a strong model with SYNCR questions greatly improved accuracy, and some improvements even carried over when tested on real-world video tasks.
What this means in practice
- •For multimedia software developers: Improve AI features that require reasoning across different video clips by training models with SYNCR's synthetic cross-video question sets.
- •For autonomous vehicle engineers: Enhance video-based scene understanding and event alignment for better decision-making by adapting SYNCR-trained models to real-world temporal tasks.
Authors
Sara Ghazanfari, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami
Abstract
Reasoning across videos requires aligning events, matching identities, comparing motion, and integrating partial observations. Evaluating these capabilities and testing how to improve them requires both reliable labels and targeted supervision. We introduce SYNCR, a simulator-grounded framework that connects these two needs through shared task generators. Built on Habitat, Kubric, and CLEVRER, SYNCR derives answers from environment state and provides 4,000 evaluation questions and 15,960 training questions over disjoint videos, spanning eight cross-video reasoning tasks. Visual ablations and human evaluation assess dependence on the supplied evidence and answer recoverability. Evaluation of 22 multimodal large language models reveals persistent difficulties in physical comparison and scene integration that increasing model size does not consistently resolve. Supervised fine-tuning raises Qwen3-VL-8B's average SYNCR accuracy from 32.6% to 61.6%, with gains extending to task configurations and video sources absent from training for those tasks. Transfer to real footage is most consistent for temporal ordering: accuracy improves by 9.0-20.5 percentage points on constructed Assembly101 and Panoptic ordering sets across three checkpoints spanning two model families and two model sizes, with additional gains on existing temporal reasoning benchmarks. These results establish SYNCR as a controlled setting for diagnosing cross-video reasoning failures, testing their learnability, and identifying where synthetic supervision transfers.