Search method improves math reasoning in language models

Improving Math Reasoning through Value-guided Informative Search

Artificial Intelligence

Summary

Solving math problems with AI models is tricky because they need to find the best steps to get the right answer. The authors noticed that just trying lots of different ways doesn’t always help. They created a new method called APIVIS that carefully picks better steps during the search and uses those improvements to teach the model. This makes the AI better at solving math problems. They tested it on common math benchmarks, and it worked better than earlier methods.

What this means in practice

  • For ai developers: Improve AI models’ ability to solve complex math problems by integrating APIVIS for more effective training with limited search budgets.
  • For automated reasoning engineers: Enhance automated mathematical reasoning systems with value-guided search to increase solution accuracy and reliability.

Authors

Shaohuai Liu, Yuning Wu, Haoran Liu, Enzo Jia, Devin Chen, Kai Wei

Abstract

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the mathematical reasoning capabilities of large language models. Recent work introduces search into RLVR rollouts to increase trajectory diversity, but diversity alone does not ensure that the search-induced rollout policy improves upon the current policy. To address this gap, we propose APIVIS, a training-time framework that adapts finite-budget Gumbel search to chunk-level mathematical reasoning. APIVIS combines direct and searched responses within each rollout group, allowing improvements found by search to produce informative relative rewards. It further applies selective supervision to search-improved tokens, preserving a learning signal when uniform group rewards render GRPO ineffective. We show that exact value-guided selection improves the expected verifier reward at each searched state and that this guarantee extends to the complete rollout policy, with a corresponding approximate guarantee under bounded value-estimation error. Experiments on widely recognized mathematical reasoning benchmarks and different model scales demonstrate substantial improvements over competitive search-based methods, validating the effectiveness of APIVIS.