Calibrating token pruning improves multi-image synthesis speed and quality
Calibrating Attention-Based Token Pruning for Efficient Multi-Image Synthesis in UMMs
Computer Vision and Pattern Recognition
Summary
Generating images from multiple pictures requires a lot of computer power because the process looks at many details all at once. The authors focus on speeding this up by carefully trimming less important details in a way that avoids mistakes other methods make. They created a new approach called CATP that keeps the important parts better aligned and uses a smart math correction to maintain quality. Their tests show it can almost double the speed without making the generated images worse.
What this means in practice
- •For machine learning engineers: Accelerate multi-image synthesis pipelines by integrating CATP for faster inference with preserved output quality.
- •For graphics software developers: Implement CATP to reduce computation in image generation workflows involving multiple input images while maintaining results quality.
Authors
Zimeng Wu, Qijian Tian, Penghui Du, Changqian Yu, Jiaxin Chen, Yunhong Wang
Abstract
Unified Multimodal Models (UMMs) enable powerful multi-image synthesis but incur substantial computation from long visual contexts. While token pruning offers a direct route to accelerating inference, existing methods are primarily adapted from understanding tasks, suffering from insufficient reference image exploration, confused conditional guidance, and pooling-induced ambiguity that degrades generation quality. To address these issues, we propose CATP, the first training-free context-compression method tailored to UMM generation, which \textbf{C}alibrates the discrepancies of general \textbf{A}ttention-based \textbf{T}oken \textbf{P}runing for better quality-efficiency trade-off. Concretely, building on independent layer-wise pruning during denoising, we introduce Guidance-Consistent Pruning (GCP) to synchronize pruning decisions across conditioning branches, and a Taylor-Calibrated Attention Metric (TCAM) that derives a theoretically grounded second-order correction for pooling-induced bias. In addition, a Fast Scoring Operator (FSO) fuses the calibration into tiled online Softmax to reduce pruning cost. Extensive experiments across representative UMMs and benchmarks demonstrate that CATP consistently outperforms state-of-the-art token-reduction methods, achieving up to 2.09$\times$ attention-core speedup.