Multi level caching speeds up high resolution image generation
MG-Cache: Multi-Granularity Caching for Accelerating Diffusion Transformer Inference
Machine LearningPerformance
Summary
Generating high-resolution images with Diffusion Transformer models can take a long time because of costly calculations. The authors propose MG-Cache, a way to store and reuse parts of these calculations at different steps and layers, reducing the amount of work needed. Their method does not require retraining the models and keeps outputs precise while making image generation up to 4 times faster. This approach works by figuring out which steps and layers have repetitive or stable calculations that can be reused safely.
What this means in practice
- •For machine learning engineers: Reduce computation time when deploying diffusion transformer models for generating high-resolution images without retraining the models.
- •For video synthesis developers: Speed up video generation pipelines that use diffusion transformers by applying multi-granularity caching to reduce inference costs.
Authors
Zesen Chen, Youxuan Xu, Shigang Li
Abstract
Diffusion Transformer (DiT) models, while having become a powerful backbone for generative synthesis, suffer from high inference cost on generating high-resolution images. Diffusion caching has emerged as a promising acceleration technique to reduce inference cost. However, existing caching strategies fail to accurately identify the varying importance of different computational modules and rarely coordinate cache reusing across the timestep and layer granularities, making it difficult to control error accumulation. To address these limitations, we propose multi-granularity caching (MG-Cache) for DiTs, a novel and fully training-free approach to exploit computational redundancy both on timesteps and Transformer layers. Our core innovation contains a two-stage caching strategy: an offline timestep profiling analysis that identifies redundant timesteps for cache reuse, followed by an adaptive layer-wise caching strategy that combines input stability with importance of each Transformer layer to selectively cache stable Attention and MLP submodules. Extensive experiments on mainstream open-source DiT models validate the effectiveness of our approach, achieving significant acceleration while maintaining high fidelity to the uncached output, with up to a 4.18x acceleration on HunyuanVideo.