Reasoning state propagation improves intermediate process reward learning
Learning Process Rewards via Reasoning State Propagation
Artificial IntelligenceMachine Learning
Summary
Evaluating the steps that a computer takes to solve a problem helps improve its learning, but training this evaluation usually needs many detailed examples that are expensive to provide. The authors propose a method called Reasoning State Propagation (RSP) that connects early steps in a reasoning process to the final result by modeling whether steps stay valid or become invalid and vice versa. This lets the system learn from fewer detailed examples by also using final outcomes to guide the understanding of intermediate steps. Tests show this method leads to better performance in tasks like searching for answers, picking the best response, and reinforcement learning.
What this means in practice
- •For ai engineers: Improve training efficiency for AI systems that learn to solve problems by reasoning through intermediate steps.
- •For chatbot developers: Enhance chatbot response quality by better evaluating and selecting reasoning paths leading to accurate answers.
Authors
Kai Gan, Zi-Hao Zhou, Bo Ye, Jian Zhao, Min-Ling Zhang, Tong Wei
Abstract
Process reward models (PRMs) have demonstrated notable effectiveness in test-time scaling and reinforcement learning by providing fine-grained signals for evaluating intermediate reasoning states, but their training relies heavily on costly process annotations. A natural way to alleviate this dependence is to complement limited process supervision with scalable outcome supervision. However, existing PRMs often model reasoning prefixes independently, providing no explicit mechanism for effectively using final outcome to guide the learning of intermediate reasoning states. We introduce Reasoning State Propagation (RSP), which represents each reasoning prefix with a binary validity state and models transitions between successive states across the reasoning trajectory. Specifically, RSP predicts a break probability that a valid state becomes invalid and a repair probability that an invalid state returns to valid. By propagating these transitions, RSP connects intermediate states to the final state, allowing process annotations to supervise intermediate states while outcome labels supervise the final state and can provide learning signals to preceding steps. Across reasoning search, response selection, and reinforcement learning, RSP consistently outperforms representative PRM baselines, with average improvements over Qwen2.5-Math-PRM of 5.6% in beam search and 2.1% in reinforcement learning.