Pixel based video generation improves resolution and responsiveness
Back to Pixels: Lifting Latent Video Diffusion into Pixel Space
Computer Vision and Pattern Recognition
Summary
Generating realistic videos with AI often involves compressing frames into smaller latent spaces, which makes training easier but limits resolution and fine control. The authors introduce PixVid, a method that takes a trained video model and adapts it to generate videos directly in pixel space instead of latent space. This means videos can be created at higher resolution and each video frame responds quickly to new inputs, which is helpful for editing or interactive use. Their approach performs competitively, keeping video quality while improving flexibility and detail.
What this means in practice
- •For video game developers: Create high-resolution, frame-responsive videos for dynamic game environments using AI video generation directly in pixel space.
- •For video editing software teams: Integrate faster and more precise video editing tools that maintain alignment during rapid motion by generating frames individually in pixel space.
Authors
Yanke Song, Shuo Chen, Chris Rockwell, Guande He, Nan Liu, Desai Xie, Tianpei Gu, Yong-Hyun Park, Aryan V S, Dennis Yuren Menn, Ming-Hsuan Yang, Xun Huang
Abstract
Frontier video generation models work in a VAE latent space that compresses video in both space and time. This compression made training tractable, but now limits high-resolution, controllable, and interactive generation -- doubling resolution quadruples token count, conditions apply to groups of frames rather than individual ones, and each response must wait for the rest of its multi-frame block. We present PixVid, a simple yet effective method that lifts a pretrained latent video model to directly generate every frame in pixel space. PixVid keeps the diffusion transformer and replaces the VAE decoder with a convolutional head that renders its outputs into pixels. We find that predicting clean video rather than velocity, together with a larger noise shift, is key to quality. Finetuned from Wan2.1-14B, PixVid is competitive with the latent baseline within $\sim 1.3$M training clips. Working framewise in pixels, it generates at $2\times$ higher resolution with the same per-frame token count, keeps depth-conditioned video editing aligned under fast motion, and responds to each action at the next frame. Project page: https://chenshuo20.github.io/PixVid/