Adapting pretrained image models to generate images in any order
Unlocking Any-Order Generation in Pretrained Autoregressive Image Models
Computer Vision and Pattern RecognitionMachine Learning
Summary
Making digital image generators more flexible is hard because they normally create images in just one fixed order. The authors figured out a way to teach existing image generators to work in any order without starting from scratch. This method lets computers fill in missing parts of images, extend images, or edit them more easily than before. Their approach works well and doesn’t need much extra computing power.
What this means in practice
- •For computer vision engineers: Enable flexible image editing capabilities such as inpainting and outpainting with pretrained autoregressive image models.
- •For graphics software developers: Integrate any-order image generation features into image editing tools without retraining models from scratch.
Authors
Rishav Pramanik, Marco Pedersoli, Zhaozheng Yin
Abstract
Autoregressive (AR) image generators are powerful in principle but inflexible in practice: training with a fixed raster-scan order hardcodes a single factorization of the image distribution into the model, leaving it unable to inpaint, outpaint, or edit at inference time. Existing solutions, including the modern discrete-diffusion family, train from scratch on a mixture of token orderings, paying roughly 3x the compute of standard raster-scan training; discrete diffusion further sacrifices compatibility with the AR inference ecosystem (KV caching, vLLM) by construction. We argue that this cost is unnecessary. A pretrained raster-scan AR model has already learned the joint distribution over image tokens; what it lacks is not knowledge but the ability to express that knowledge through non-sequential conditional pathways. We propose a recipe for adapting pretrained raster-scan models into any-order generators using two components: future-aware positional embeddings that resolve a target-position identifiability problem at negligible overhead, and an ordering curriculum that traces a stable optimization path from the pretrained solution to a good any-order basin. Applied to four base models across two architectures and two scales (LlamaGen-L/XL, RAR-L/XL), the recipe substantially outperforms MaskGIT on ImageNet inpainting (FID 6.21 vs. 10.44) and outpainting, is competitive with MAR-L, trades a 1.2-1.9 increase in class-conditional FID for this flexibility, and costs 13-33% additional compute beyond pretraining, while leaving the standard AR inference stack untouched. Inpainting, outpainting, and editing, our results suggest, need not be added to autoregressive image models. They can be unlocked from the ones that exist. Code is available at https://github.com/Ordered-AR/unlockedadaptation.