Just Video Transformer simplifies pixel video generation with transformers

JVT: Just Video Transformer makes pixel-space video diffusion simple

Computer Vision and Pattern Recognition

Summary

Creating videos directly from raw pixels is difficult and often requires complex special tools. The authors introduce Just Video Transformer (JVT), a simpler way to generate videos using a type of AI model called a Transformer, without extra tricks or helpers. They studied which design choices work best to keep the quality high while saving computing power. Their method matches the quality of existing advanced approaches and can also create related video information like depth maps that help understand scenes better.

What this means in practice

  • For video content creators: Generate high-quality video sequences directly from raw pixels using simpler transformer models without pretraining complex encoders.
  • For augmented reality developers: Create RGB videos with aligned depth information to enhance scene understanding and improve camera tracking in AR applications.$Commercial implications: Enables selling improved video rendering and scene reconstruction tools for AR by directly generating RGB and disparity maps together.

Authors

Jianjun Zhou, Hua Geng, Haoyu Guo

Abstract

Direct diffusion modeling of high-dimensional raw video data is challenging, and existing pixel-space methods often rely on specialized architectures. We introduce JVT (Just Video Transformer), a simple large-patch DiT-style Transformer for pixel-space video diffusion, without a pretrained autoencoder, auxiliary detailers, cascaded stages, or U-Net-style skip connections. We systematically examine video patch embeddings, register tokens, and adaptive layer normalization to identify design choices that balance generation quality and computational efficiency. The resulting JVT design achieves generation quality competitive with latent diffusion baselines in both class-conditioned and unconditional settings. The pixel-space formulation of JVT also naturally supports joint generation of pixel-aligned modalities. We demonstrate this capability through camera-conditioned RGB-disparity generation, which improves camera following relative to RGB-only baselines and yields explicit scene geometry without an additional reconstruction model at inference. These results highlight the potential of simple DiT-style Transformers for pixel-space video diffusion, and we hope our findings offer useful insights and practical guidance for future research.