Reinforcement learning improves math problem solving with function graphs
Function-Structured Reinforcement Learning with Executable Verifiers for Mathematical Reasoning
Machine Learning
Summary
Solving math problems with computers is hard because just knowing the final answer doesn’t always show if the method is correct. The authors developed a way to break down problems into smaller parts using function graphs and check each step with multiple verifiers. They trained a system to write code solving math problems and improved it by giving rewards for correct steps, not just final answers. This led to better math solutions and more reliable reasoning.
What this means in practice
- •For software engineers: Enhance automated code generation for complex math problem solving by integrating function-structured feedback and reinforcement learning.
- •For automated testing teams: Use verifier-guided reinforcement learning to improve the correctness and reliability of code that handles mathematical computations.
Authors
Zihan Liu, Xurong Xie
Abstract
Algorithmic mathematical reasoning requires reliable decomposition, computation, and aggregation. Final-answer rewards provide limited guidance on intermediate errors, while successful execution does not guarantee mathematical correctness. This work proposes Function-Structured Graph Reinforcement Learning (FSG-RL), connecting subproblem graphs and Python implementations with multi-verifier feedback. The policy first learns to generate code from function graphs through supervised fine-tuning (SFT). Group Relative Policy Optimization (GRPO) then optimizes the policy using answer-gated rewards and span-level credit assignment. The framework also supports teacher supervision and structured memory. A benchmark curated from Grade School Math 8K (GSM8K), MathQA, MATH, and Omni-MATH pairs public function graphs with private verification specifications. Under a unified evaluation protocol, GRPO improves final-answer accuracy from 43.25% to 67.50% and full solution success from 32.25% to 52.25% over SFT. Continued reinforcement learning (RL) with teacher supervision yields additional gains. The gains extend beyond producing correctly formatted code, supporting verifier-guided reinforcement learning for mathematical reasoning. Code is available at https://github.com/ZihanLiummyycc/FSG-RL.