Semantic aware video codec improves error resilience in low latency transmission
Semantic-Aware Neural Video Codec for Error-Resilient Low-Latency Transmission
Information TheoryMachine LearningMultimedia
Summary
Video systems that need to send video instantly over shaky internet connections can lose parts of the video, making it hard to watch or use. The authors developed a new video codec that knows which parts of the video are most important and sends those parts with higher priority so they are less likely to be lost. Their codec can recover better from missing data by letting each piece be decoded on its own, instead of depending on others. This method helps keep important details clear even when the connection is bad.
What this means in practice
- •For network engineers: Improve video streaming reliability over unstable wireless or internet connections by prioritizing critical video data packets.
- •For mobile app developers: Deliver low-latency video feeds in apps that remain clear on important content despite packet loss or poor network conditions.
Authors
Matin Mortaheb, Homa Esfahanizadeh, Jinfeng Du, Harish Viswanathan
Abstract
Emerging physical AI systems require low-latency, task-oriented video communication over unreliable channels. We propose a semantic-aware multi-level neural video coding method for robust low-latency video transmission over unreliable channels that are abstracted as multi-level packet erasure channels. Built upon the real-time DCVC-RT neural video codec, the proposed framework introduces a semantic- and feature-aware coding strategy that partitions encoded representations into packets carrying different levels of semantic and latent-feature importance and assigns these packets to different streams, each associated with a priority level when transmitted over unreliable communication channels. We also developed an error-resilient entropy model that removes inter-packet dependencies, allowing each packet to be decoded independently under packet losses. The complete system is trained end-to-end over the abstracted multi-level packet erasure channels, enabling learning of channel-aware representations together with importance-aware packet assignment while facilitating the network for differentiated packet prioritization. Experiments show that the proposed framework significantly improves robustness over baseline DCVC-RT under packet erasures, achieving graceful degradation in less important regions while better preserving task-relevant visual content.