State-conditioned steering improves reasoning exploration efficiency
Efficient Reasoning Exploration via State-Conditioned Latent Steering with Progress Guidance
Computation and Language
Summary
The paper addresses how reasoning models often get stuck exploring similar paths during problem solving, limiting their effectiveness. The authors propose a new method called SPS that uses past progress to guide the model toward more meaningful next steps without retraining. By using stored directions tailored to different points in the reasoning process, SPS helps models explore more diverse and productive solutions. Experiments show that SPS consistently improves performance on reasoning tasks compared to previous techniques.
What this means in practice
- •For ai developers: Improve automated reasoning systems so they find better solutions by guiding exploration toward more productive solution paths.
- •For software engineers: Enhance AI reasoning components in applications like question answering or planning by integrating latent steering for efficient inference.
Authors
Hengyuan Zhang, Chenming Shang, Zunhai Su, Xiao Liang, Hui Shen, Jing Xiong, Dawei Li, Shiping Yang, Kailai Yang, Wei Zhang, Ruobing Xie, Hayden Kwok-Hay So, Ngai Wong
Abstract
Best-of-$N$ is a widely used inference strategy for complex reasoning, whose effectiveness depends on whether sampled candidates can cover diverse and high-quality reasoning paths. However, post-trained reasoning models often suffer from \emph{exploration collapse}, where independent rollouts repeatedly follow similar reasoning paths and limit the gains from increasing the rollout budget. Existing methods alleviate this issue by promoting broader exploration, but do not explicitly guide exploration toward continuations that make meaningful progress, resulting in limited exploration efficiency. To address this, we propose \emph{\underline{S}tate-conditioned \underline{P}rogress-guided \underline{S}teering} (SPS), a training-free latent steering framework. Specifically, SPS constructs a state-conditioned Direction Bank containing multiple progress-guided steering vectors for different prefix-state regions. During online inference, SPS retrieves a suitable steering vector based on the current prefix state and applies it at high-uncertainty transitions to guide the next reasoning step toward meaningful progress. Extensive experiments across multiple model scales and benchmarks demonstrate that SPS consistently outperforms strong baselines. Further analyses validate the effectiveness of its key designs and offer valuable insights for future research. The code is available at https://github.com/rattlesnakey/SPS.