Video perception improved with world-model guided packet transmission
Guess and Send: World-Model-Guided Semantic Packet Transmission for Robust Video Perception
Multimedia
Summary
Sending video over limited wireless networks is tricky because receivers already have some scene information, so sending redundant data wastes bandwidth. The authors designed a system called WorldGT that predicts what the receiver already knows using a shared world model, and then smartly decides what new data to send. This approach prioritizes important information, protects key data better, and works well even when the connection is poor. Tests show that WorldGT improves video understanding tasks like object detection, especially in challenging signal conditions.
What this means in practice
- •For wireless video engineers: Improve video analysis performance over bandwidth-limited wireless links by adapting transmissions to predicted receiver states and channel conditions.
- •For autonomous vehicle developers: Support robust real-time scene understanding in vehicles by transmitting only novel or impactful features to perception modules under poor network signals.
- •For mobile network operators: Enhance wireless network resource usage for video-based services by implementing receiver-aware semantic packet scheduling and error protection schemes.$Commercial implications: Enables selling advanced network features that improve video delivery quality and efficiency under constrained and unreliable channels.
Authors
Yan Gong, Zheng Chu, Wei Koong Chai, Hari Mohan Pandey, Jiankang Zhang, Jingrui Hou, Miao Zhang, Yuanjian Li, Pei Xiao
Abstract
Reliable video perception over bandwidth-constrained wireless links requires transmitting information that the receiver cannot already recover from its evolving scene representation. This paper introduces WorldGT, a world-model-guided semantic packet transmission framework that prioritizes information according to its contribution to the receiver's perception. A shared feature-space world model predicts the receiver's scene state through affine motion compensation and learned residual displacement, while sender-side verification identifies reusable predictions. An acknowledgment-synchronized receiver twin estimates task-impact scores through two counterfactual evaluations, guiding packet selection and unequal error protection over LDPC-coded wireless channels. A compact 3-bit feature representation increases spatial coverage under stringent bandwidth constraints, while predictive fallback sustains perception during packet outages. Experiments on KITTI and VisDrone under Rayleigh fading and AWGN demonstrate substantial gains in low-SNR conditions. On KITTI at a 4.13% reference bandwidth budget, WorldGT achieves 47.5% J&F at 0 dB Rayleigh SNR, outperforming H.264 by 11.3 percentage points. Controlled ablations show that world-model prediction contributes up to 8.7 points under Rayleigh fading. Object-detection experiments further demonstrate the transferability of receiver-relative predictive transmission across visual perception tasks. These results establish world-model-guided, receiver-state-aware transmission as an effective approach to robust video perception under severe communication constraints.