Neural video coding improves by using limited frame delays
Neural Video Coding under Limited Delay: Theoretical Optimality and System Design
Machine LearningInformation Theory
Summary
Video compression traditionally processes each frame as it comes, without looking ahead, which misses some information that could make compression better. The authors show that allowing a small delay to see a few future frames helps reduce uncertainty when coding video. They prove mathematically that this limited delay approach can achieve the best possible compression in certain cases. They also create a system that can add this look-ahead ability to existing video codecs without big changes, and test it to show better video quality for the same file size.
What this means in practice
- •For video streaming engineers: Improve video streaming quality by integrating limited-delay neural video coding to reduce bandwidth without degrading visual quality.
- •For video codec developers: Upgrade existing neural video codecs with a lightweight module that exploits short frame-delay information for better compression efficiency.
Authors
Likun Cai, Jun Chen, Ashish Khisti
Abstract
Traditional neural video compression (NVC) methods predominantly follow strictly causal predictive-coding paradigm, treating zero-frame delay as a fixed constraint rather than delay as an explicit design parameter. From an information-theoretic perspective, allowing limited frame delay enables a codec to exploit correlations with future frames within the delay window, thereby reducing uncertainty that is inaccessible to strictly causal prediction and potentially improving the rate-distortion tradeoff. In this work, we formalize this intuition through a rigorous rate-distortion analysis builds on the clique factorization of Markov networks, and propose Graph-Factored Sequential Coding (GFSC), a principled framework for sequential video coding under limited delay. To our knowledge, we provide the first theoretical result showing that, for general $K$-order Markov sources with positive distributions, a $K$-frame delay is sufficient to attain joint-coding rate-distortion optimality over a nontrivial distortion region, a guarantee unavailable to zero-delay causal coding in general. In practice, we instantiate GFSC through lightweight neural modules, allowing existing causal predictive codecs to be augmented without modifying their core architectures. Experiments across multiple NVC models and datasets show that GFSC consistently improves rate-distortion performance over the corresponding causal counterparts. Under matched delay budgets, GFSC also achieves competitive performance compared with representative bidirectional codecs.