Phantom accelerates diffusion image generation with temporal caching
Phantom: Temporal Caching for Diffusion Models
Hardware Architecture
Summary
Generating images with diffusion models takes a lot of computing power because the process repeats many steps. The authors see that changes between these steps mostly affect only small parts of the data. Their system, called Phantom, saves and reuses parts that don’t change, only recomputing what’s different. This approach lets it run about four times faster and use less energy while keeping picture quality high.
What this means in practice
- •For ai hardware engineers: Build specialized accelerators to speed up diffusion model inference while lowering power consumption using temporal caching techniques.
- •For mobile device developers: Integrate faster and more energy-efficient diffusion image generation into devices with limited hardware resources.
Authors
Yuzhe Fu, Cong Guo, Zishan Shao, Bowen Duan, Chiyue Wei, Haoxuan Shan, Ting Jiang, Hancheng Ye, Jonathan Ku, Changchun Zhou, Hai "Helen" Li, Yiran Chen
Abstract
Diffusion models have become the dominant paradigm for high-quality image generation, but their iterative inference across tens of timesteps incurs substantial computational overhead. Prior accelerators exploit temporal similarity in diffusion models via differential quantization or output sparsity, yet suffer from increased memory footprint, design complexity, and image degradation. In this work, we present Phantom, an algorithm-architecture co-designed accelerator built on temporal caching. We view temporal similarity as differential sparsity: between adjacent timesteps, only a small fraction of tokens (outliers) change significantly. Phantom recomputes only these outlier tokens and reuses the rest from a temporal cache. Phantom runs on standard systolic arrays with lightweight additions: a Phantom Engine at the global buffer that detects outliers and updates the cache as activations stream through, and a condensed pipeline that maps the resulting sparse updates onto dense systolic computation for FC, convolution, and attention layers. Evaluations on modern diffusion models show that Phantom achieves an average 4.0x speedup and 5.5x energy saving over GPUs and prior accelerators while preserving generation quality.