Sparse representations speed up 3D visual question answering tasks
SparseTalk - Sparsifying 3D Gaussian Language Fields for Efficient 3D Visual Question Answering
Computer Vision and Pattern RecognitionMachine Learning
Summary
3D visual question answering means asking questions about objects in a three-dimensional scene. Usually, the computer stores lots of detailed information, but this can be slow and use a lot of memory. The authors studied how to reduce this data without losing much accuracy. They found that they can keep only a tiny fraction of the information—focused mostly on objects—and still answer questions well, making the process much faster and more efficient.
What this means in practice
- •For augmented reality developers: Create faster and more memory-efficient systems for answering questions about 3D scenes by using sparse spatial and semantic information.
- •For robotics engineers: Enable real-time understanding and interaction with complex 3D environments by selectively focusing on key object features to reduce processing load.
Authors
Davit Soselia, Joseph JaJa, Amitabh Varshney
Abstract
3D Gaussian language fields provide an explicit, spatially grounded representation for 3D visual question answering (VQA), but their dense semantic features can require tens of thousands of embeddings per scene, resulting in substantial storage, memory, and inference costs. We investigate how much of this representation is actually necessary for downstream reasoning. Starting from a full embedding representation, we systematically sparsify its semantic embeddings, including the previously underexplored regime below a single image-equivalent block down to 8 visual tokens. We compare random, geometric, semantic, and joint spatial-semantic selection strategies and introduce an object-based sparsification method that distributes the token budget across detected object instances while retaining background context. Experiments on ScanQA and MV-ScanQA reveal substantial redundancy in dense Gaussian language fields. Strong VQA performance is retained with only a few hundred semantic embeddings, corresponding to less than 1% of the original representation. Object-based selection performs well relative to others, with only modest observed changes down to 256 tokens. At this budget, SparseTalk retains 0.80% of SplatTalk's 32,076-token inference input and 0.332% of the mean 77,207-Gaussian dense field, increasing inference throughput while reducing decoded-feature memory 125-fold.