Efficient token pruning and coding cut data for 3D visual Q&A models
TP-JSCC: Efficient Token Pruning and Joint Source-Channel Coding for LVLMs-based 3D-VQA in Cloud-Edge Collaboration
Computer Vision and Pattern Recognition
Summary
Large vision-language models for answering questions about 3D scenes need to send lots of visual data from devices to the cloud, which takes time and can get corrupted. The authors developed a method called TP-JSCC that smartly reduces the amount of data by selecting only the most important image parts and compressing them to send reliably even with noisy connections. Their approach keeps nearly all the important information while massively cutting data size and computing needed. This helps run big visual question-answering models on simple devices that work together with cloud servers.
What this means in practice
- •For edge device developers: Reduce data and computation on embedded devices for cloud-assisted 3D visual question answering to improve latency and reliability.
- •For network infrastructure teams: Handle variable and noisy communication channels when transmitting vision-language data from edge devices to cloud by adapting coding dynamically.
Authors
Zhengxin Li, Chongzhen Tian, Hui Yuan, Hao Liu, Sam Kwong
Abstract
Deploying large vision-language models (LVLMs) on resource-constrained embodied agents for 3D visual question answering necessitates cloud-edge collaboration. However, offloading massive visual data to the cloud over bandwidth-limited and noisy channels introduces severe latency and transmission distortion. To address this, we propose TP-JSCC, an efficient token pruning and joint source-channel coding method for LVLMs-based 3D-VQA in Cloud-Edge Collaboration. First, we develop a Local and Global Semantics-based Hybrid Token Pruning (LGS-HTP) module to significantly reduce the visual token count. By decoupling foreground-background tokens and jointly selecting local and global semantics, LGS-HTP effectively reduces redundancy while preserving reasoning-critical tokens. Second, we introduce a lightweight Token Joint Source-Channel Coding (T-JSCC) module to further compress the pruned tokens and ensure their robust transmission over error-prone channels. By employing lightweight cascaded linear layers and Signal-to-Noise Ratio (SNR)-adaptive mechanism, this module adapts to fluctuating channel conditions while maintaining minimal computational overhead. Extensive experiments on ScanQA and SQA3D datasets indicate the exceptional efficiency and robustness of our approach. Specifically, on ScanQA validation set, the LGS-HTP module could achieve 90% pruning ratio while incurring less than a 1% CIDEr loss. Building on this, the T-JSCC further compresses the tokens by 18 times, resulting in an overall 180 times data volume reduction for robust transmission, while saving 90% in computational complexity for LVLM deployment. Our code is available at https://github.com/yuanhui0325/TP-JSCC.