Every Token Leaves a Ripple in the Stream of Thought: Eliciting Model-Internal Token Saliency for Chain-of-Thought Compression
2026-08-31 • Computation and Language
Computation and Language
AI summaryⓘ
The authors study how to make long reasoning steps in AI models shorter without losing important information, which helps reduce computation time. They propose looking inside the model itself to see which words contribute most to the final answer by measuring how much removing or isolating each word affects the prediction. Their method, called MIST, scores these words based on both their necessity and sufficiency to keep the reasoning clear but brief. Tests show that MIST works better than previous methods at compressing reasoning steps while keeping the model's answers accurate.
Chain-of-Thought ReasoningToken-level CompressionResidual StreamAnswer LikelihoodNecessitySufficiencyModel-internal SaliencyInference Cost
Authors
Tianyi Zhao, Yinhan He, Wendy Zheng, Chen Chen
Abstract
Chain-of-thought (CoT) reasoning improves multi-step problem solving, but long reasoning traces inflate inference cost. Token-level CoT compression reduces this cost by pruning full reasoning chains into shorter traces for model adaptation, making token selection the central challenge. Existing methods often rely on external scorers or heuristic signals only indirectly tied to the model's internal answer computation. We instead adopt a model-internal perspective: as the model forms an answer, each reasoning token leaves a ripple in the residual stream, the model's \emph{stream of thought}, and the magnitude of this ripple reflects the token's contribution to the answer computation. Building on this view, we propose \textsc{MIST} (Model-Internal Saliency for Token-level CoT compression), which defines token importance along two complementary axes: \emph{necessity}, the drop in answer likelihood when a token's internal contribution is removed, and \emph{sufficiency}, the gain in answer likelihood when that contribution alone is provided. Combining the two yields a unified importance score for pruning. Across four reasoning benchmarks and four models, \textsc{MIST} consistently outperforms baseline methods, suggesting that model-internal saliency provides an effective proxy for reasoning-token importance.