Language based ordering improves image coding and captioning quality

Making Language the Low-Rate Code of Vision: Successive Refinement from Words to Pixels

Artificial Intelligence

Summary

Images can be compressed into codes that let you see a rough version first and finer details later. Usually, those codes focus on pixels and not on text descriptions (captions). The authors show that if you build the code to prioritize language early on, you get better captions without much loss in image quality. They created a system called RUNG that puts words first, then pixels, improving how well captions match images and how images are generated from text. This helps machines understand and create images more like humans do, by starting from language and refining details step by step.

What this means in practice

  • For image generation developers: Improve text-to-image systems by building codes that generate captions first and refine pixels later for better caption coverage and image quality.
  • For multimedia compression engineers: Design image codecs that prioritize language-based prefixes to enhance low-rate coding efficiency and provide interpretable intermediate representations.

Authors

Ruilin Gong, Hao Gong

Abstract

An ordered visual code lets a short prefix stand in for an image, and the training loss decides what that prefix contains. Under pixel reconstruction the first bits go to the directions of highest variance, and we prove that a caption can then be independent of the entire first half of a code that is rate-optimal for pixels at every length. In the rate-distortion sense a caption is therefore not the low-rate code of its image, unless the code is built to make it one. We study that construction and what it costs. We cast captioning and text-to-image generation as successive refinement under a distortion measure that changes along the code, from caption log-loss in the early stages to pixel distortion in the last, so that each stage is compared with its own optimum. Putting words first then costs nothing among the caption stages exactly when the captions form a summarization chain, and it costs the pixels at most the caption uncertainty left in a pixel-optimal reconstruction, which is large for short prefixes and small at full length. We build RUNG, an ordered code whose prefixes are supervised by a summarization ladder of captions and whose full length reconstructs the image; generation extends the code past the prompt and captioning truncates it. With data and capacity held fixed, ordering by language instead of reconstruction raises prefix-caption information 2.9 times and GenEval from 0.76 to 0.83, at a measured cost of 2.8 dB PSNR at a quarter of the code and 0.1 dB at full length. RUNG-XL (3.4B) reaches 0.88 GenEval and 87.6 DPG-Bench in 32 network evaluations and 0.91 GenEval when errors are redrawn at the rung where they appear; its coarse and fine captions agree 94.1% of the time, and it edits images without inversion.