Elastic Token Compression for Pixel-Space Diffusion Transformers
Computer Vision and Pattern Recognition
Summary
The authors explain that normal diffusion models process every small patch of an image separately, which can be wasteful because many patches are just flat or similar areas. They discovered these redundant areas form connected shapes and propose grouping patches based on their position along a Hilbert curve to keep neighbors together. Their method, called Region Token Interface (RTI), merges these patches into region tokens and adapts the model to work with them efficiently. This approach speeds up the generation process without losing much image quality and works well across different token budgets.
Authors
Eduard Zamfir, Christian Reisswig, Zongwei Wu, Yongqin Xian, Radu Timofte
Abstract
Natural images concentrate their detail in a small fraction of the frame, yet diffusion models spend a full token on every patch, in every layer and at every timestep. The waste is largest in pixel-space models, with no autoencoder to absorb low-level redundancy first. Probing a pretrained pixel text-to-image transformer, we find its middle-block tokens redundant wherever the image is flat. The redundancy occupies connected, content-shaped regions, and exploiting it requires tokens with the same geometry. Cutting a Hilbert ordering of the patches provides them. Consecutive positions are always image neighbours, so any contiguous run is a connected region whose size and shape follow the content, and grouping in two dimensions becomes a cut in one. Existing reductions each lose part of this. Similarity merging scatters its groups, latent bottlenecks discard position, and skipping deletes what it should summarize. We cut where the model's features change most and pool each run into one region token. Our Region Token Interface (\method{}) adapts a diffusion model to these tokens, with the region count drawn at random during fine-tuning so one checkpoint serves every budget. \method{} leads prior reduction methods at matched budgets, matches dense quality at $2.0\times$ the speed, and stays close at $2.6\times$. The code and models are open-sourced at https://eduardzamfir.github.io/rti