Semantic Video Communication via Multi-Scale Convolution and Dynamic Routing for Next-Generation Networks
2026-07-06 • Computer Vision and Pattern Recognition
Computer Vision and Pattern Recognition
AI summaryⓘ
The authors developed a new method to send meaningful video information efficiently over limited network bandwidth, especially for devices with low computing power like IoT gadgets. They created a special video encoder that looks at motion over different time lengths without using too much processing power. To better match video parts with related text queries, they used a capsule network that improves how segments connect to questions over time. Their combined approach performs well on a benchmark dataset, showing it can accurately understand and align video content with language while being efficient for edge devices.
semantic communicationtemporal convolutional encodercapsule networkdynamic routingIoT devicescross-modal alignmentmulti-task learningActivityNet CaptionsRecall@0.5mean IoU
Authors
Gengtian Shi, Jinze Yu, Chenhao Wu, Shaofei Wang, Eiji Fukuzawa, Junjie Tang, Hiroshi Onoda, Jiang Liu
Abstract
The exponential growth of video traffic demands novel semantic communication paradigms that transmit meaning rather than raw bits. We present a generative AI-enabled framework for semantic video communication addressing two critical challenges: efficient hierarchical temporal modeling for bandwidth-constrained transmission and robust semantic alignment between video content and natural language queries at network edge devices. Our approach introduces a multi-scale temporal convolutional encoder that captures motion patterns across different temporal granularities with O(T) complexity suitable for resource-constrained IoT deployments. We further propose a capsule-based dynamic routing mechanism that iteratively refines segment-query associations, enabling flexible modeling of non-monotonic semantic alignments essential for goal-oriented communication. These components are unified through a multi-task learning objective optimizing temporal boundary regression, cross-modal alignment, and capsule diversity. Experiments on ActivityNet Captions demonstrate significant improvements, achieving 42.9% Recall@0.5 and 41.1% mean IoU while maintaining computational efficiency critical for edge deployment.