Video diffusion transformers prune experts to save memory and improve use
DIET: Deletion-response Expert Trimming for Video Diffusion Transformers
Artificial Intelligence
Summary
Large video diffusion transformer models use many expert components that make them hard to store and run on common GPUs. The authors present DIET, a way to decide which expert components can be removed without retraining, by testing how the model responds when each expert is deleted. This approach shrinks the model size nearly in half while maintaining or even improving performance, letting it run on smaller GPUs. DIET works better than other pruning methods adapted from language models.
What this means in practice
- •For video model engineers: Shrink large video diffusion transformers to fit memory limits without retraining by selectively pruning experts based on deletion response.
- •For gpu infrastructure teams: Enable running large video models on smaller GPUs by halving expert counts while keeping or improving output quality.
Authors
Jiachang Zhang, Teng Hu, Bohao Feng, Songhang Shen, Wenqiang Wang, Hongqian Deng, Ran Yi
Abstract
Video diffusion transformers (DiTs) increasingly adopt mixture-of-experts (MoE) architectures to reduce active computation, but their full expert storage remains costly. Existing one-shot pruning criteria mainly rely on static activation or routing statistics and cannot capture layer-level re-routing after expert deletion. We introduce DIET, a training-free expert pruning framework based on deletion responses. A single all-expert calibration pass records expert outputs and router states for matched conditional and unconditional tokens. Candidate deletions are then replayed from cached tensors, requiring no additional model forward passes. The resulting deletion-response signatures characterize each expert by the changes induced when it is removed. DIET selects retained experts by minimizing Overall Diversity Loss (ODL), which preserves directional coverage in signature space, and combines intra-layer local search with an inter-layer regression-guided budget search to allocate experts across layers. On LingBot-Video 30B-A3B, pruning 50% of experts (6,144 to 3,072) reduces the checkpoint from 57 GB to 30 GB and enables single-card deployment on a 48 GB GPU without fine-tuning. Under a fixed 284-case VBench protocol, the VBench Total increases from 0.7941 to 0.8115. Across tested retention budgets, DIET consistently outperforms competitive pruning baselines adapted from large language models.