Language informed loss improves text to image generation quality
A Language-Informed Loss for Visual Generation
Computer Vision and Pattern RecognitionMachine Learning
Summary
Generating images from text is usually done by training models to match pixels in images, but that can miss the meaning behind the images. The authors introduce a new method that uses language to guide the image generation process more effectively by training a special image encoder that links image features to text captions. This encoder enables a loss function that helps the generation model create images that align better with the input text. Their approach improved image quality on a standard benchmark compared to just using pixel differences.
What this means in practice
- •For text to image model developers: Improve model training by using language-informed losses to generate images that better match input text descriptions.
- •For multimodal ai system builders: Incorporate invertible image encoders to connect visual features and language for richer supervision signals in model training workflows.
Authors
Suchir Agarwal, Keshigeyan Chandrasegaran, Ziyu Chen, Kyle Sargent, Juan Carlos Niebles, Li Fei-Fei, Jiajun Wu
Abstract
End-to-end visual generation with pixel-space loss has recently achieved competitive performance without the need for learned image tokenizers. In this work, we study whether language-informed supervision can provide a richer training signal than pixel loss for visual generation. We introduce LILIE (pronounced "lily"), a Language-Informed and Lossless Image Encoder. LILIE is an invertible image transform trained such that a vision-language model can recover captions from noised features. The encoder uses a normalizing flow transformer trained jointly with a vision-language model via next token prediction. We then use the encoder to extract a simple language-informed loss for text-to-image generation. We train JiT models with LILIE loss, reducing gFD by 14 points (66.84 $\to$ 53.01) against a pixel-only baseline on the GPIC dataset. Our code, encoders, and generative models are available at https://suchiragarwal.com/lilie.