Accurate registration improves real-time image upscaling quality and speed
Frozen History: When Temporal Losses Stop Accumulation in Learned Real-Time Supersampling
Computer Vision and Pattern Recognition
Summary
Real-time image upscaling tries to make low-resolution images look higher resolution by combining information over time. This paper shows that if you perfectly align images from previous frames, a certain common technique to reduce flicker actually prevents good accumulation of detail, which can hurt image quality. The authors identify specific causes of misalignment that can hide this problem, and they show that fixing these improves the sharpness and stability of upscaled videos. They also demonstrate their approach works better than existing commercial and classical methods, running fast on current hardware.
What this means in practice
- •For real-time graphics developers: Implement image upscaling that better preserves detail and reduces flicker by improving how previous frames are aligned before accumulation.
- •For mobile device manufacturers: Integrate improved image upscaling methods that run efficiently on mobile GPUs to enhance video or game visuals with less blur and noise.$Commercial implications: Enables higher quality image scaling on mobile devices, improving user experience and product appeal using efficient neural supersampling.
Authors
Hao Yu, Guoying Zhao
Abstract
Learned real-time supersampling accumulates jittered low-resolution samples along motion vectors. Many methods use a temporal consistency loss to reduce flicker. We show that with accurate registration, this loss can suppress accumulation: on static content, the network copies its warped history at almost no temporal cost. Camera jitter in motion vectors raises this cost and can hide the effect. We identify four registration errors that training can conceal: jitter-carrying motion vectors, one motion field for both resolution lattices, jittered depth, and an unregistered skip path. A model-free oracle shows that correct registration makes freezing two orders of magnitude cheaper on static pixels. Controlled training shows that registration reverses the temporal weight's effect on PSNR. Under exact registration, increasing the weight reduces pixel variance at the expense of accuracy and perceived quality, with a 0.7 to 0.9 dB loss on static frames at weight 0.25. This reversal recurs across three network families and two scales; the loss also reduces accuracy on unjittered STSS data. With a small temporal weight and a normalised sample accumulator, a plain registered network improves the best published QRISP PSNR by 0.34 to 0.62 dB at $\times$2, $\times$3 and $\times$4 upscaling to 1080p. At $\times$2, its static pixel standard deviation is one quarter that of DLSS 2. Without retraining, it outperforms classical temporal upsampling by at least 4.4 dB on Arm's jittered Bistro capture. Our most accurate $\times$4 model runs in 2.03 ms per 1080p frame on an H200. Code: https://github.com/CV-AC/Frozen_history