Recurrent models speed prefill with mixed precision computing
RAMP: Recurrent Adaptive Mixed Precision for Efficient Prefill in Recurrent-Depth Language Models
Computation and Language
Summary
Long language inputs take more time for some AI models because they repeat calculations many times. The paper finds that using less precise math early in the process causes smaller mistakes than expected because later steps reduce errors. The authors propose a new method called RAMP that uses low-precision math at the start and switches to high precision at the end. This keeps output quality nearly the same while speeding up processing up to about 14%.
What this means in practice
- •For machine learning engineers: Reduce latency in language model inference by integrating mixed precision early in recurrent depth computations to accelerate prefill steps.
- •For ai hardware developers: Design hardware accelerators that optimize mixed-precision arithmetic tailored for recurrent-depth language models to improve speed without degrading output quality.
Authors
Ke Wan, Chen Chen
Abstract
Recurrent-depth language models improve inference-time reasoning by repeatedly applying a shared network to refine internal representations, but the additional recurrent computation also increases inference cost. This overhead is particularly pronounced during long-context prefill, where the full input sequence is repeatedly processed across recurrent depth. Weight-and-activation quantization offers a natural way to accelerate this computation through lower-precision arithmetic, yet repeatedly applying quantized computation introduces numerical perturbations that accumulate along the recurrent trajectory. We systematically analyze how such errors propagate across recurrent depth and show that quantization errors introduced earlier are attenuated by more subsequent recurrent steps and therefore have a smaller effect on the final representation. Matched-budget experiments strongly support this prediction, showing that placing quantized computation earlier in recurrent depth consistently yields the best fidelity. Based on this insight, we introduce RAMP (Recurrent Adaptive Mixed Precision), which performs early recurrent refinement using low-precision weight-and-activation computation and restores high precision for terminal refinement. Across context lengths up to 4K, RAMP preserves near-full-precision prefill quality while achieving up to 1.16x end-to-end first-prefill speedup on an NVIDIA H100, corresponding to a 13.9% latency reduction.