Phase-based reward shaping improves large language model reasoning efficiency

Phase-Conditioned Reward Shaping for Token-Efficient LLM Reasoning

Machine LearningArtificial Intelligence

Summary

Long responses from language models can mix helpful calculations with repeated or unnecessary checks, making them inefficient. The authors present a way to recognize different parts of the reasoning process and adjust penalties on tokens accordingly during training. This approach helps models produce shorter, more focused answers without losing accuracy. Testing on math and science problems showed slightly shorter answers and better results when multiple attempts were allowed.

What this means in practice

  • For ai model trainers: Design training routines that reduce redundant text generation in large language models for complex problem solving.
  • For software developers: Improve multi-step reasoning applications by making language model outputs more concise without sacrificing accuracy over multiple attempts.

Authors

Zhe Chen, Xiaolin Zhou, Xiyang Hu

Abstract

Extended reasoning can help language models solve complex problems, but long responses often contain both useful computations and redundant checks. Conventional length penalties encourage shorter responses without distinguishing the functions of intermediate steps. We propose Phase-Conditioned Reward Shaping (PCRS), a framework that combines phase identification with training-time reward allocation. We investigate two classification routes: PCRS-H with a hidden-state classifier and PCRS-T with a frozen text classifier. Our main text classifier implementation assigns overlapping reasoning, execution, and revision labels to native chunks, then uses these labels and exact repetition to adjust existing token penalties. We compare the completed configurations on six mathematical and scientific reasoning benchmarks and examine four reward-weight settings with the text classifier. Compared with a baseline using token-level length penalties, the default text classifier configuration reduces mean response length by 3.13%; sampled Pass@1 is approximately 62.07% for both methods, while oracle Pass@16 rises from 86.08% to 87.67%. These results show how phase-informed penalty allocation can reduce output volume and improve multi-attempt coverage without inference-time classification.