OpticalRAG compresses long documents into few tokens with images

OpticalRAG: Pixel-Space Compression for Token-Efficient Retrieval-Augmented Generation

Computation and Language

Summary

Long documents are hard to fit into language models because they only accept a limited number of words, and compressing text loses details. The authors show that by turning pages into pictures and using a vision encoder, they can pack documents into very few tokens without losing important meaning. Their system, OpticalRAG, retrieves and summarizes these image tokens so a language model can use them efficiently. This approach works well on several benchmarks and improves performance with fewer tokens than usual.

What this means in practice

  • For large language model engineers: Build more token-efficient retrieval systems that handle long documents by compressing text content as images for better use in LLM context windows.
  • For enterprise search teams: Improve search result relevance by efficiently incorporating detailed document content into language model queries using image-based compression.

Authors

Senhao Liu, Yuheng Zhang, Chunyu Wei, Yueguo Chen, Xinran Zhang

Abstract

Retrieval-augmented generation (RAG) is bottlenecked by the token budget of large language models: feeding long documents into the context window is expensive, while aggressive textual compression discards fine-grained information that is irreversibly lost in a one-dimensional token sequence. We argue that this bottleneck is not fundamental to information density but to the choice of compression \emph{modality}. Modern visual encoders can pack the contents of a rendered text page into roughly a hundred visual tokens with little semantic loss, opening a new compression-fidelity tradeoff that purely textual methods cannot reach. We present \textbf{OpticalRAG}, a framework that renders documents as page images, encodes them with a frozen vision encoder, and distills the retrieved visual tokens into as few as $16$ injected tokens before passing them to the LLM. Two ingredients make this work without expensive cross-modal training: (i) \emph{Encoder-Bridged Retrieval}, which projects text-query states into the visual feature space of the frozen encoder, through a projection shared with the compressor, to retrieve visual tokens from a text query, with no contrastive training; and (ii) \emph{Query-Driven Distillation}, a lightweight cross-attention compressor that condenses the retrieved visual tokens into a small set of query-conditioned prefix tokens. On LongBench-v2, BAMBOO, and LooGLE-v2, OpticalRAG achieves the best compression-aware accuracy (CAP) on all three benchmarks while injecting only $16$ tokens, and improves the closed-book Qwen2.5-7B-Instruct backbone by $5.17$ points on LongBench-v2.Code is available at https://github.com/brighterr/OpticalRAG.