TempoMoE improves multimodal reasoning speed and accuracy adaptively
TempoMoE: Identifying and Steering Fast-Slow Thinking Experts for Efficient Multimodal Reasoning
Computation and LanguageComputer Vision and Pattern RecognitionMachine Learning
Summary
Multimodal reasoning models can take a long time thinking even for easy tasks, wasting effort and sometimes making mistakes. The authors found that different parts of the model work like fast and slow thinkers, influencing how long and deep the reasoning is. They created TempoMoE, a method that figures out which parts work fast or slow and adjusts how much each part contributes depending on task difficulty. This speeds up easy tasks and improves accuracy overall without retraining the model.
What this means in practice
- •For ai engineers: Control inference steps in multimodal models dynamically to speed up responses on simple queries while keeping accuracy on complex ones.
- •For product managers in ai services: Offer more efficient multimodal AI products that reduce delay and cost by tailoring computational effort to input difficulty.$Commercial implications: Enables development of AI products with faster and cheaper multimodal reasoning by adaptive expert steering according to task difficulty.
Authors
Xiyue Zhang, Keru Zhou, Ziyi Wang, Bing Hu, Li Wang
Abstract
Multimodal Mixture-of-Experts (MoE) reasoning models rely on long chain-of-thought (CoT) for complex tasks, yet often overthink easy inputs, increasing token cost and latency, and sometimes degrading accuracy. Existing methods optimize reasoning length during training or impose external control at inference, but leave the underlying reasoning mechanism a black box. We find that fast and slow thinking exhibit distinct expert-routing patterns: holding the input fixed and varying only the prompted reasoning tempo produces stable routing shifts. Suppressing slow-thinking experts shortens the reasoning chain, whereas suppressing fast-thinking experts lengthens it, providing causal evidence that these experts influence reasoning depth. Based on this finding, we propose TempoMoE, a training-free, two-stage framework. It identifies fast- and slow-thinking experts via tempo-contrastive routing analysis and redistributes routing weights within the original top-k set according to sample difficulty. It amplifies fast-thinking experts and suppresses slow-thinking experts on easy inputs and reverses this adjustment on hard ones, enabling difficulty-adaptive control over reasoning depth. We evaluate TempoMoE on Kimi-VL, Qwen3-VL, and ERNIE-4.5-VL across six benchmarks. Compared with Vanilla, TempoMoE consistently improves accuracy while reducing output tokens, with accuracy gains of up to 2.6 points and token reductions of up to 35.4%.