Accelerating Unified Multimodal Models with Core-Expansion Routing and Unified Computation Scheduling
Artificial Intelligence
Summary
The authors studied unified models that handle both understanding and generation of multimodal data and found that the computation done is often repetitive. They discovered that understanding mainly relies on a consistent core of important information, while generation builds on this with additional changes over time. To make the process more efficient, they created CE-Router, which smartly separates the shared core computations from generation-specific expansions. This reduces unnecessary work during inference, leading to faster processing without losing much accuracy. Their tests showed it keeps over 98% of understanding quality while almost doubling the speed.
Authors
Wengyi Zhan, Chenqian Yan, Songwei Liu, Mingbao Lin, Rongrong Ji
Abstract
Unified multimodal models jointly support understanding and generation, but incur substantial redundant computation across tokens, layers, and generation timesteps. Through token-importance probing, we identify an asymmetric core-expansion structure: understanding exhibits a stable importance component, while generation largely shares this component but requires progress-dependent corrections. We therefore propose CE-Router, which uses a task-shared core scorer and progress-conditioned generation expansions, optimized through generation decomposition and cross-task core alignment. At inference, CE-Router compacts token computation and supplies a learned routing signal to Unified Computation Scheduling, which coordinates layer skipping, FFN pruning, diffusion-head cache reuse, and denoising-step early exit. Experiments on two representative UMM architectures demonstrate consistent quality--efficiency improvements across both tasks, retaining 98.03\% of dense understanding performance with a 1.93$\times$ end-to-end inference speedup.