Visual autoregressive models improve image quality with new dependency approach
Logit Refiner: Improving Visual Autoregressive Models via Intra-Scale Dependency Modeling
Computer Vision and Pattern RecognitionArtificial IntelligenceMachine Learning
Summary
Many image generation models create pictures piece by piece but treat all parts at the same detail level independently, which can cause local visual glitches. The authors found this happens because the standard step in these models ignores connections between nearby pixels at the same scale. They introduced a small add-on called the Logit Refiner that samples image tokens one by one in order, capturing these local connections without retraining the original model. This improvement leads to clearer, more coherent images across various model sizes and works for both class-based and text-based image generation.
What this means in practice
- •For image generation engineers: Integrate the Logit Refiner into existing visual autoregressive models to improve local image coherence without retraining large backbones.
- •For text-to-image developers: Enhance text-to-image systems by adding sequential intra-scale sampling to improve image quality beyond current parallel decoding methods.
Authors
Meimingwei Li, Stefan Andreas Baumann, Felix Krause, Björn Ommer
Abstract
Visual Autoregressive Models (VAR) generate images through next-scale prediction, producing all tokens within each scale in parallel. We show that this parallel decoding constitutes a mean-field-style approximation that discards spatial dependencies among same-scale tokens, causing locally incoherent samples regardless of backbone capacity -- a limitation of the decoding rule. Addressing this limitation, we introduce the Logit Refiner, a lightweight autoregressive module that restores intra-scale dependencies by sequentially sampling tokens conditioned on frozen backbone features. Adding only ~10% parameters and less than 5% of the base model's training compute, it plugs into any pretrained VAR checkpoint without retraining. Controlled ablations isolate joint intra-scale sampling -- rather than additional capacity or training -- as the critical ingredient. Across backbones from 310M to 2B parameters on class-conditional ImageNet 256x256, the refiner consistently improves generation quality, enabling a 1.1B-parameter model to surpass one twice its size. The approach further generalizes to text-to-image generation, confirming that the mean-field bottleneck persists across VAR variants and is effectively alleviated by our method. Project page: https://compvis.github.io/logit-refiner/