Multimodal language models get faster answers with deferred prefill
Raincheck: Deferred Multimodal Prefill for Efficient Multi-Turn MLLM Inference
Computation and LanguageArtificial Intelligence
Summary
Long conversations with multimodal AI models can take a long time before the AI starts answering because it needs to process a lot of input first. The authors present Raincheck, a way to let the AI start processing part of the input right away and gradually include more information as it generates an answer. This helps reduce the waiting time for the first answer without losing important context later. Their approach works well on two models and several methods, cutting initial wait times significantly while keeping the quality high.
What this means in practice
- •For multimodal ai developers: Reduce initial response delay in AI systems handling multi-turn conversations with images and text by gradually processing input tokens during generation.
- •For cloud ai service operators: Lower latency and optimize resource use during multimodal query processing by scheduling token prefill in unused compute windows to improve user experience.
Authors
Wenjie Huang, Jun Zhang, Hongyaoxing Gu, Haoyang Huang, Tianqi Xu, Xinyue Hao, Xinyi Hu, Xinle Du, Jun Dai, Shuang Ge, Chen Lei, Yue Li, Mingcheng Wan
Abstract
Multi-turn conversations with multimodal large language models (MLLMs) reuse input context, but long multimodal inputs increase time to first token (TTFT). Query-aware hard pruning reduces prefill tokens but permanently excludes information that later questions may require. We propose Raincheck, a framework for deferred multimodal prefill that decouples initial token selection from later information availability. An existing pruning method selects tokens for initial prefill; Raincheck preserves and progressively prefills unselected tokens in source order using unused compute capacity during decode. Unfinished prefill continues after answer return during available scheduling gaps. Attention visibility control and affine state mapping enable context updates after generation begins in full-attention and hybrid recurrent models. Prefill-decode fusion reduces repeated weight reads. We evaluate Qwen3-VL-8B and Qwen3.5-9B with 30% initial multimodal-token retention. Across both models and four pruning methods, Raincheck recovers on average 77.7% and 60.4% of the quality lost to matched hard pruning on MultiVerse and ConvBench, respectively. On MultiVerse, the SGLang implementation reduces mean first-turn TTFT by up to 24.1% relative to full prefill, with less than 0.5% increase in mean time per output token (TPOT).