Beverage improves 3d spatial reasoning from video for visual language models
BEV3SR: Evidence-Aware 3D Spatial Reasoning from Video
Computer Vision and Pattern Recognition
Summary
Understanding 3D space from videos is hard because videos only show things from a moving camera, not a stable map of the scene. The authors introduce BEV3SR, a way to turn videos into a bird's-eye view map that keeps track of objects and their positions. It also decides if a question needs geometry calculations or visual clues to answer and uses the right method for each. This makes visual language models better at answering questions about 3D scenes from videos, especially for tricky spatial questions.
What this means in practice
- •For robotics engineers: Build robots that better understand and navigate 3D environments by answering spatial questions from video inputs.
- •For autonomous vehicle developers: Enhance self-driving cars' perception systems to interpret spatial relationships and scene layout from multi-view video data.
Authors
Fan Zhu, Rui Yu
Abstract
3D spatial reasoning from video remains challenging for VLMs because videos provide sequential, view-local observations rather than persistent metric scene records. Many questions must be answered in a query-specified perspective rather than the observed camera viewpoint, and prompting with a fixed set of RGB views may miss sparse or occluded visual cues. To address these gaps, we introduce BEV3SR, a training-free and backbone-agnostic framework for RGB-D videos with camera parameters. BEV3SR turns video observations into queryable metric bird's-eye-view (BEV) scene memory and formalizes each query into an evidence-aware spatial task. The Spatial Task Formalizer specifies the query's reference frame, spatial constraints, objective, and evidence requirement. The evidence requirement determines the appropriate reasoning pathway. A query treated as requiring geometric evidence is handled by the Geometric Reasoner, which deterministically computes the answer using the metric BEV records in the query-specified reference frame. Otherwise, a query treated as requiring visual evidence is passed to the Visual Evidence Aligner, which uses BEV-indexed object records and their associated supporting views to retrieve object-consistent and scene-context RGB frames for VLM grounding. Within a standard 16-frame budget, BEV3SR consistently improves different VLM backbones across multiple 3D scene QA benchmarks, delivering large gains on spatially intensive questions and significant improvements on open-ended scene QA. These results demonstrate that BEV3SR offers an effective, efficient, and interpretable framework for explicit 3D spatial reasoning from video.