CoVeR improves 3D scene understanding by smartly reducing image data
CoVeR: Coverage-Based Token Pruning for Multi-View 3D Reasoning in VLMs
Computer Vision and Pattern RecognitionMachine Learning
Summary
Understanding 3D scenes using 2D images often involves looking at many pictures from different angles, which creates a huge amount of repeated information. The authors found that current ways to reduce this information either miss important parts of the scene or keep too many similar images of the same area. They created CoVeR, a method that picks images to cover the whole scene efficiently without learning from data. CoVeR keeps only about 8% of all image data while still maintaining most of the accuracy, helping computers reason about 3D scenes faster and better.
3D scene representationmulti-view imagesvisual language modelstoken pruningspatial coveragevoxelizationattention mechanismsdeterministic selectiontoken budget3D reasoning benchmarks
Authors
Nhat-Tan Bui, Varshini Elangovan, Arun Reddy Anugu, Sreyas Mohan, Wei Ye, Dilin Wang, JQ Huang, Rakesh Ranjan, Aviral Chharia, Fernando De la Torre
Abstract
Representing a 3D scene as multi-view images allows 2D VLMs to reason in 3D by reusing priors from pre-training, sidestepping the scarcity of annotated 3D data. However, it produces thousands of redundant visual tokens whose cost grows with every view. Existing visual token pruners fall into two families, each limited in the 3D multi-view setting. Learned importance methods rank tokens by attention or encoder features; because redundancy here is fundamentally spatial, they keep near-duplicate tokens from a few prominent regions and leave most of the scene unrepresented. Voxelization methods improve spatial coverage but cannot enforce an exact token budget and saturate as multi-view observations overlap in 3D, capping retention well below the target. We show that spatial coverage is associated with 3D reasoning performance and introduce CoVeR, a deterministic, training-free selector that uses only token coordinates, with no learned signals. CoVeR selects tokens that collectively cover every region of the scene, and solves the limitations of both families: it enforces an exact per-scene budget, breaks the voxelization saturation plateau, and avoids the near-duplicate selections of learned importance. Extensive experiments show CoVeR outperforms prior SOTAs on all three 3D reasoning benchmarks and generalizes as a plug-and-play module tested across four VLMs. Notably, with only $\approx$8% of visual tokens, it preserves 93.5% of full-token performance, surpassing SOTA by 3.9 percentage points on average across benchmarks.