Hierarchical hints improve ai math reasoning with reinforcement learning
H$^2$R: Hierarchical Hint-Based Reward for Mathematical Reasoning via Reinforcement Learning
Machine Learning
Summary
Solving complicated math problems step-by-step is hard for AI. The authors introduce a way to give AI helpful intermediate hints based on parts of the problem to guide its thinking process. These hints act like checkpoints that make solving the whole problem easier. They tested this approach on a new set of math problems and showed that AI models performed better when using these hints during training.
What this means in practice
- •For ai developers: Improve mathematical problem-solving performance of AI models by integrating hierarchical hint-based rewards during reinforcement learning training.
- •For edtech developers: Build AI tutoring tools that provide stepwise feedback by modeling math problem solving with intermediate logical hints to better guide students.
Authors
He Tu, Siye Li, Hongyi Wen
Abstract
Reinforcement Learning with Verifiable Rewards has shown great promise in enhancing mathematical reasoning in large language models. However, existing approaches provide limited intermediate supervision for complex, multi-step reasoning tasks. To bridge this gap, we propose \textbf{Hierarchical Hint-based Reward}, a framework that leverages the intrinsic sub-question structure of mathematical problems as structured intermediate conditioning signals during RLVR training. Unlike chain-of-thought prompting, these hierarchical hints serve as logically grounded intermediate milestones that effectively reduce the reasoning horizon toward the final answer. To validate this framework, we introduce \textbf{HiMath}, a benchmark of hierarchically structured mathematical problems. Across diverse LLMs, hierarchical hints consistently improve reasoning performance even without task-specific training. When integrated into RL training via H$^2$R, they boost Qwen2.5-Math-7B accuracy on HiMath from 30.4\% to 39.1\% using hierarchical hints, and up to 45.6\% with Oracle Hierarchical Hints. This pronounced improvement demonstrates that incorporating reliable intermediate checkpoints into reward design is essential for sustaining complex mathematical reasoning.