Text to image models can be trained without language pretraining
Text-to-Image Generative Modeling Without Language Pretraining
Machine LearningComputer Vision and Pattern Recognition
Summary
Text-to-image models usually rely on language learning from the internet before they can create pictures from words. The authors introduce Halcyon, a new model family trained from scratch only using a large dataset of image-text pairs. They found that training on text alone beforehand does not improve the image quality. Halcyon matches or beats other models that needed much more computing power and language pretraining. This shows strong text-to-image generation is possible without language-based pretraining.
What this means in practice
- •For ai developers: Train efficient text-to-image models from scratch without relying on large language model pretraining, reducing computational cost.
- •For digital content creators: Use models like Halcyon to generate images from text prompts with reduced computational resources and simpler training pipelines.
Authors
Keshigeyan Chandrasegaran, Yue Zhao, Armin W. Thomas, Ehsan Adeli, Jiajun Wu, Juan Carlos Niebles, Li Fei-Fei, Michael Poli
Abstract
Modern text-to-image (T2I) systems use internet-scale language pretraining for text-conditioned image generation. We ask whether language pretraining is necessary for strong T2I generation. We introduce Halcyon, a family of early-fusion multimodal autoregressive models pretrained entirely from scratch on GPIC, a dataset of 100M image-text pairs. Halcyon-0.52B outperforms LlamaGen-XL/8 (48.0 vs. 63.3 gFD) with 18.2$\times$ less pretraining compute. Further, Halcyon-1.6B outperforms JiT-T2I-XXL/8++ (35.3 vs. 41.0 gFD) with 275$\times$ less pretraining compute. Controlled experiments show that text-only pretraining does not improve T2I generation. Halcyon also learns image captioning by replacing caption dropout with I2T supervision, maintaining competitive T2I generation quality under the same pretraining budget. We show that strong T2I systems can be trained from scratch without internet-scale language pretraining. Code and models are available at https://cs.stanford.edu/~keshik/halcyon