Visual autoregressive models improve image quality with next token imagination
Visual Autoregressive Generation with Next Token Imagination
Computer Vision and Pattern Recognition
Summary
Generating realistic images is hard because models usually learn only from examples they have seen. The authors suggest that allowing a model to imagine possible next parts of an image beyond what it has been explicitly trained on can help it create better pictures. They developed a method called Next Token Imagination, which lets the model explore plausible visual possibilities while staying realistic. This approach led to noticeably better images using less training effort on a large image benchmark.
What this means in practice
- •For computer vision teams: Enhance image generation models to produce higher-quality visuals with reduced training costs using the next token imagination approach.
- •For game developers: Create more realistic and diverse game textures by extending image generation beyond previously seen examples with imagination-guided token prediction.
Authors
Ruihang Jiang, Zongyao Zhao, Zhaolin Wang, Yuanwei Liu, Xiatian Zhu
Abstract
The space of possible visual content is enormous, and arguably unbounded. Any training set, however large, therefore covers only a tiny fraction of it. Yet generative models are typically trained to fit only the images they are given. In visual autoregressive (AR) generation, for example, each token is predicted from the preceding ground-truth (GT) tokens and supervised solely by the GT next token. Confining learning to observed data in this way limits how much of the visual distribution a model can capture. In this work, we hypothesize that letting a model imagine beyond its training data, by exploring plausible visual realizations it has never observed, can ease this limitation. We use visual AR generation as a proof of concept, since its fixed, discrete token space makes imagination easier to steer and constrain. We propose Next Token Imagination (NTI), a method that allows the model to explore beyond the GT next token, subject to a carefully designed criterion that keeps this imagination bounded by the underlying visual distribution. Extensive experiments on ImageNet show that NTI substantially and consistently improves the fidelity of generated images, approaching state-of-the-art diffusion and AR models while using a smaller model and lower training cost.