Steady thought reduces excessive reasoning switches in large ai models
SteadyThought: Mitigating LLM Under-Thinking via Thought-Level Preference Optimization
Artificial Intelligence
Summary
Large AI models that solve problems by thinking step-by-step often switch too much between different lines of thought, which wastes effort and lowers performance, a problem called under-thinking. The authors propose Steady Thought, a method that encourages the model to stick with promising ideas longer by learning to prefer better reasoning paths. Their experiments show that this approach cuts down on wasted steps while keeping the models just as accurate. It also helps models to apply this improved thinking consistently across different languages and topics.
What this means in practice
- •For ai application developers: Deploy large reasoning models with reduced redundant internal switching to lower computing costs while maintaining accuracy.
- •For multilingual software engineers: Enhance reasoning consistency of language models across different languages by reducing unnecessary thought switches.
Authors
Zhenyue Peng, Lin Yang, Jie Wang, Xiaoqi Ni, Hanzhu Chen, Xize Liang, Yanjun Wu, Rujing Wang, Kaiwen Zhou, Jianye Hao
Abstract
Flexible switching between reasoning trajectories (i.e., thought switching) has significantly enhanced the reasoning capabilities of Large Reasoning Models (LRMs). However, existing models often switch excessively yet fail to sustain promising reasoning thoughts, a phenomenon termed "under-thinking". While recent efforts suppress switching to mitigate this, such over-correction may discard valuable trajectories. To address this challenge, we propose Steady Thought (ST), a novel thought-level preference optimization framework. ST formalizes under-thinking as a preference issue at switching points, forcing single-path continuations from identified points to yield high-quality reasoning. Then, ST performs thought-level preference optimization by treating the newly generated response as preferred and the original one as dis-preferred. Experiments across multiple models and datasets show that ST effectively reduces token consumption, achieving reductions of up to 63.8%, while maintaining strong reasoning accuracy. Further analysis suggests that ST helps models acquire a generalizable ability to reduce redundant switching across domains and languages. Code is available at https://github.com/MIRALab-USTC/SteadyThought.