Policy improves test-time model reasoning by adding smart proposal choices
Value-Conditioned Proposal Acquisition for Test-Time Reasoning
Artificial Intelligence
Summary
Sometimes, AI models pick answers from a limited set, missing the right one if it’s not included. The authors created a way for the model to decide whether to stop, keep trying, or ask a special helper for a suggestion during reasoning. This method learns a value estimate to compare these choices and keeps the comparison consistent across tasks and models. Their approach improved math problem-solving accuracy by a notable margin and works well on different models and question types. The key benefit comes from deciding when to get help rather than just having helpers available all the time.
What this means in practice
- •For machine learning engineers: Improve AI model decision quality during inference by selectively acquiring specialized proposals to better solve complex reasoning tasks.
- •For automated tutoring platforms: Enhance automated problem-solving engines by enabling smarter use of auxiliary reasoning skills to boost accuracy on math problem sets.
Authors
Zeyan Li, Jianfeng Xu
Abstract
Sampling and selection spend test-time compute inside a candidate pool, but they cannot recover a correct answer that never enters it. We introduce ValueFormation, a test-time policy that decides after each primary-model draw whether to stop, continue sampling, or acquire one proposal from a heterogeneous reasoning specialist. The three actions are compared on one cost-regularized value scale learned from vote-state features; the calibration is frozen across tasks and primary models, and acquired proposals are accepted only through a frozen posterior comparison. On held-out MATH-500, one Llama-fitted calibration improves over the strongest of eight identically evaluated baselines by more than fifteen points for each of three 8B primaries and transfers unchanged to Qwen3-14B. Removing acquisition lowers Llama MATH accuracy by 20.43 points, while same-trace and same-access replays attribute the gain to state-conditioned acquisition rather than specialist access alone. Where primary coverage is high, as on GSM8K, acquisition is rare and gains mainly come from cheaper allocation; on MBPP, a fixed code route attains the top accuracy at the lowest interaction cost. Varying the token price monotonically reduces calls without retraining.