Panoramic video reasoning benchmark improves cross-view and time understanding
Think in 360°: Benchmarking and Bootstrapping Panoramic Video Reasoning
Computer Vision and Pattern Recognition
Summary
Panoramic videos show everything around a camera, making it hard to understand what’s happening because important details appear in many directions and times. The authors created the first test set called PanoVideoBench with lots of questions to check how well AI models understand panoramic videos. They found current models struggle to follow objects across views and combine information over time. To help, they built a large set of example questions and answers that teach a model to think through panoramic video scenes better. Their improved AI showed big gains on the test, especially in connecting views and events over time.
What this means in practice
- •For virtual reality developers: Improve scene understanding in 360-degree videos for more responsive VR environments by enhancing object tracking across views and times.
- •For video surveillance teams: Enhance panoramic camera analysis by enabling models to better connect observations around the camera and over time for improved event detection.
Authors
Lidong Lu, Lidong Zou, Xinyi Mao, Jiaxin Lv, Zhenquan Li, Liangrui Dong, Tong Lu, Guo Chen
Abstract
A panoramic video captures all directions around the camera, but relevant evidence can appear in different directions and at different times. Understanding it requires models to connect observations across the viewing sphere and time. We introduce PanoVideoBench, the first manually annotated benchmark for comprehensive panoramic video understanding. Its 1,000 questions span five categories and 15 question types, with diagnostic annotations for target search and panoramic capabilities. Evaluation of 11 proprietary, open-source, and panoramic-specialized models identifies cross-view tracking and sphere-time evidence integration as their weakest capabilities. To strengthen these capabilities, we construct PanoVideoCoT, the first reasoning dataset grounded in real panoramic video, through sphere-time-grounded synthesis that converts verified panoramic observations and successful agentic explorations into 13,465 QA-CoT pairs. Starting from a general video MLLM, PanoVideoReasoner learns from this corpus through Panoramic CoT SFT and Evidence-Privileged On-Policy Self-Distillation, a panoramic instantiation of Vision-OPD's OPSD in which the teacher uses verified perspective views to guide student-generated trajectories. It reaches 55.7% on PanoVideoBench, improving the backbone by 12.5 points and the previous SOTA by 2.6. The gains concentrate on the same spatial, cross-view, and temporal capabilities exposed by the benchmark, showing that grounded sphere-time supervision can equip a general video MLLM with panoramic reasoning.