LLM agents learn when to spend more time reasoning to improve tasks
PORA: Learning When More Reasoning Pays Off for LLM Agents
Artificial Intelligence
Summary
Large language model (LLM) agents often have to choose between thinking briefly or more deeply before taking each action in a complex task. Thinking longer can help avoid future mistakes but uses up limited brainpower that might be needed later. The authors created a system called PORA that learns when spending extra time reasoning actually pays off by comparing short and long thinking in training. This lets the agent decide efficiently during real use whether longer reasoning is worth the cost, improving results across different tasks while saving effort.
What this means in practice
- •For software developers: Optimize code generation agents to spend reasoning effort efficiently, improving code correctness while reducing computation.
- •For retail analytics teams: Improve decision processes in retail data agents by dynamically allocating reasoning effort where it most improves prediction accuracy.
Authors
Bo Yan, Qi Zhang, Zhen Tan, Tianlong Chen, Song Wang
Abstract
LLM agents perform tasks over multiple steps and revise their actions based on feedback. As the total reasoning budget is often limited and must support future steps, the agent must choose between short and long reasoning before each action. Longer reasoning may improve the final task outcome by preventing later errors, but it also uses budget capacity that may be needed for future decisions. This creates two challenges: estimating the downstream value of long reasoning and deciding when that value is worth using the remaining budget. We propose Paired Outcome-guided Reasoning Allocation (PORA), which learns this decision by comparing short and long reasoning from the same interaction state. During training, \method{} completes the task twice from each selected state, once with a short reasoning allowance and once with a long allowance, and compares their final quality and total processing to estimate when longer reasoning is valuable. During inference, a controller weighs this estimate against learned prices on remaining-budget pressure and chooses either short or long reasoning without running both alternatives. We evaluate PORA across application, memory, code, retail, and mathematics tasks, achieving strong results across all settings. On BFCL memory, it exceeds the strongest existing method by 17.4--32.9 percentage points. Within the same agent, it matches or exceeds always using long reasoning on code and mathematics with 50.6% fewer actor tokens and 31.2% fewer model requests, respectively.