Audio video generation speeds up with selective caching approach
CAVC: Convergence-Adaptive Audio-Visual Cache for Modality-Asymmetric Diffusion Generation
Computer Vision and Pattern RecognitionSound
Summary
Generating synchronized audio and video through diffusion models usually requires equal work on both to maintain quality, which can be slow. The authors noticed audio can be cached and reused in the middle of generation without hurting much, while video needs more careful, continuous processing. They created a caching method called CAVC that smartly saves audio computations mid-way and finishes with some joint cleanup, speeding up generation while keeping good video and audio quality. Their tests showed faster results with better video motion and timing, and music sounds just as good as before.
What this means in practice
- •For multimedia software developers: Accelerate audio-visual content generation pipelines by selectively caching audio computations without compromising synchronization quality.
- •For animation studios: Improve rendering speed of synchronized animated videos with sound by preserving video refinement and caching audio steps mid-process.
Authors
Lifan Jiang, Yibo Zhao, Senlin cheng, Bolei Xu, Xiaobo Li, Can Ye, Dehai Zhao, Deng Cai, Boxi Wu
Abstract
Diffusion-based audio-visual generation methods typically perform joint denoising of audio and video latents throughout the entire sampling trajectory, repeatedly executing modality-specific denoising branches and cross-modal attention at every timestep. However, we observe that such uniform joint computation is not always necessary. The key is a modality-asymmetric recoverability: audio-side drift introduced by mid-stage caching can be substantially reduced by only a few terminal joint denoising steps, whereas video-side degradation caused by caching leads to persistent loss of motion and temporal details. This asymmetry suggests that inference acceleration should selectively cache recoverable audio-side computation while preserving full video refinement. Motivated by this observation, we propose \textbf{Convergence-Adaptive Audio-Visual Cache (CAVC)}, a training-free modality-asymmetric caching framework. CAVC adopts a sandwich-structured computation allocation: after detecting that audio denoiser outputs enter a cache-tolerant regime, it caches and reuses the audio denoiser output to bypass audio denoising and cross-modal attention in the middle stage, while the video branch continues independent spatio-temporal refinement; a few joint steps are then restored at the end to reduce accumulated cross-modal drift and improve synchronization. Experiments across multiple public benchmarks demonstrate that CAVC achieves 1.10--1.35x wall-clock acceleration, improves several visual dynamics and temporal statistics, and maintains competitive audio fidelity and audio-visual alignment.