Papers for

multimedia compression engineers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Language based ordering improves image coding and captioning quality

Making Language the Low-Rate Code of Vision: Successive Refinement from Words to Pixels

Abstract: An ordered visual code lets a short prefix stand in for an image, and the training loss decides what that prefix contains. Under pixel reconstruction the first bits go to the directions of highest variance, and we prove that a caption can then be independent of the entire first half of a code that is rate-optimal for pixels at every length. In the rate-distortion sense a caption is therefore not the low-rate code of its image, unless the code is built to make it one. We study that construction and what it costs. We cast captioning and text-to-image generation as successive refinement under a distortion measure that changes along the code, from caption log-loss in the early stages to pixel distortion in the last, so that each stage is compared with its own optimum. Putting words first then costs nothing among the caption stages exactly when the captions form a summarization chain, and it costs the pixels at most the caption uncertainty left in a pixel-optimal reconstruction, which is large for short prefixes and small at full length. We build RUNG, an ordered code whose prefixes are supervised by a summarization ladder of captions and whose full length reconstructs the image; generation extends the code past the prompt and captioning truncates it. With data and capacity held fixed, ordering by language instead of reconstruction raises prefix-caption information 2.9 times and GenEval from 0.76 to 0.83, at a measured cost of 2.8 dB PSNR at a quarter of the code and 0.1 dB at full length. RUNG-XL (3.4B) reaches 0.88 GenEval and 87.6 DPG-Bench in 32 network evaluations and 0.91 GenEval when errors are redrawn at the rung where they appear; its coarse and fine captions agree 94.1% of the time, and it edits images without inversion.

Mon 12 OctArtificial Intelligence
The gist
Images can be compressed into codes that let you see a rough version first and finer details later. Usually, those codes focus on pixels and not on text descriptions (captions). The authors show that if you build the code to prioritize language early on, you get better captions without much loss in image quality. They created a system called RUNG that puts words first, then pixels, improving how well captions match images and how images are generated from text. This helps machines understand and create images more like humans do, by starting from language and refining details step by step.
Open → 2610.16552v1