Language models reuse prior steps to boost reasoning without delay
P2N: Reusing Deep Representations for Greater Effective Depth
Computation and Language
Summary
Language models need to think through hard problems step by step, but adding more layers to do this is slow and costly. The authors created a method called P2N that reuses the deep information from the previous word to help with the current word’s understanding, making the model effectively deeper without extra computation time. This improves model performance on math problems and other language tasks by a noticeable margin. Their approach doesn’t make the model slower when it produces answers.
What this means in practice
- •For ai model developers: Increase the effective reasoning depth of language models during generation without adding inference delay by reusing prior token computations.
- •For automated tutoring platforms: Enhance math problem-solving accuracy in tutoring systems using language models with improved reasoning from reused token information.
Authors
Yiqin Hao, Boyi Zeng, Zitong Wang, He Li, Shixiang Song, Feichen Song, Yifan Liu, Zhouhan Lin
Abstract
Mathematical problem solving and autonomous agents are increasingly important applications of language models, demanding stronger reasoning capabilities. Computational depth is a key resource for such reasoning, but adding model layers increases both training and inference costs, while test-time scaling incurs substantial additional inference computation. We introduce P2N (Previous to Next), which reuses the previous token's deep representations, already available during generation, to increase effective computational depth with negligible decoding overhead. Specifically, P2N adds the previous token's deep hidden state to the current token's shallow representation, allowing subsequent layers to build on previously completed deep computation. This recurrence extends effective computational depth across tokens, while introducing only a single vector addition per decoding step. We train the recurrence using parallel Jacobi updates restricted to a middle-layer core. Experiments with Qwen-style models show that, at 1.5B parameters, P2N improves GSM8K accuracy by over 10 percentage points under both zero-shot and few-shot prompting, with nearly unchanged decoding latency. The benefits also extend to language modeling and average performance on general downstream benchmarks. on general downstream benchmarks.