Continuous space time video super resolution improves clarity and smoothness
Residual Motion Splatting for Continuous Space-Time Video Super-Resolution
Computer Vision and Pattern Recognition
Summary
Low-resolution, choppy videos can be made clearer and smoother by increasing their details and frame rates. The authors developed a method that predicts how each pixel moves over time with high precision, even between original frames. Their technique combines multiple blurry video frames onto a sharper grid while handling missing parts or overlaps, making the process faster and resulting in better quality. This approach merges frame interpolation and resolution enhancement into one step and works well on standard benchmarks.
What this means in practice
- •For video streaming services: Enhance low-resolution streaming videos to higher resolution and frame rates in real time for improved viewer experience.$Commercial implications: Enables products that upscale streaming content with better quality and responsiveness, appealing to media service providers.
- •For video editing professionals: Produce smoother and clearer slow-motion or high-resolution effects from low-quality source footage during post-production.
Authors
Vincent Herfeld, Eva Coupet{é}, Julien Mairal, Jean Ponce
Abstract
Continuous space-time video super-resolution produces a high-resolution, highframe-rate video from a low-resolution, low-frame-rate one at arbitrary spatial scales and temporal instants. Classical spatial multi-frame super-resolution has shown that detail beyond a single frame's sampling limit can be recovered by aligning complementary observations with sub-pixel precision and accumulating them on a finer grid. Existing learning-based methods largely depart from this principle: some align and fuse frames only at low resolution before regressing a continuous high-resolution representation, while others interpolate each frame to the output resolution before splatting. In both cases, the per-frame samples are resampled before fusion, so their sub-pixel positions are lost before frames are combined. Restoring this principle in a learned continuous model raises two challenges: estimating sub-pixel motion at unobserved times, and handling holes and occlusions when splatting a coarse grid onto a finer one. We address the first by predicting per-pixel continuous trajectories that refine linear optical-flow paths with smooth residual motion, using features integrated over the full video sequence. We address the second by directly splatting consecutive low-resolution features in a learned latent space onto a high-resolution grid at their predicted sub-pixel positions with predicted visibility weights, before a decoder resolves remaining ambiguities. This makes temporal interpolation and multi-frame superresolution a single operation. Our method improves both reconstruction fidelity and perceptual quality on multiple standard benchmarks over the state of the art, while running up to an order of magnitude faster.