Improving large language model reasoning by encouraging diverse thinking paths
Parallel Thinking for Large Language Model with Path Diversity
Artificial Intelligence
Summary
Sometimes big language AI models try to figure things out by thinking along several similar routes, which limits their effectiveness. The authors found that these models often get stuck making the same kind of reasoning steps during parallel thinking, which isn’t helpful. They created a way to encourage the AI to explore more different paths at once by rewarding it when its reasoning paths are less alike. This method helped the models perform better on tasks that require reasoning by keeping a variety of solutions alive. Their approach shows that having different ways of thinking at the same time improves AI reasoning.
What this means in practice
- •For ai developers: Integrate diverse parallel thinking to improve reasoning accuracy in AI services for complex problem solving.
- •For automated customer support teams: Use path diversity techniques to reduce repetitive and narrow responses, creating more varied and comprehensive automated answers.
Authors
Jiaru Qian, Yihong Dong, Xue Jiang, Chun-Wah Hsu, Zhi Jin, Ge Li
Abstract
Parallel thinking has emerged as a promising paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs) by exploring multiple reasoning paths simultaneously. However, we identify that parallel thinking via Reinforcement Learning (RL) typically suffers from the mode collapse of reasoning paths, where parallel thinking trajectories degenerate into similar ones. This homogeneity undermines the fundamental motivation of parallel thinking, that is, maintaining multiple hypotheses concurrently to prevent premature loss into a suboptimal solution. In this paper, we propose Diverse Parallel Thinking (DPT), a simple yet effective RL approach for parallel thinking designed to encourage path diversity. DPT rewards the model to develop distinct reasoning strategies by penalizing the prefix similarity of parallel thinking paths, thereby enhancing a more comprehensive exploration of the reasoning space. We conduct various experiments and analyses across multiple reasoning benchmarks to demonstrate the effectiveness of DPT. Results show that DPT successfully outperforms previous parallel thinking methods, achieving state-of-the-art performance over baselines. We validate path diversity as a key element to improve parallel thinking. Our code is available at https://github.com/JiaruQian/DPT.