Hierarchical reinforcement learning improves coding agent efficiency
Doing More with Less Tokens: Hierarchical Reinforcement Learning for Efficient Coding Agents
Software Engineering
Summary
Coding agents help write software by interacting with development tools, but they often use too many tokens, which costs time and money. The authors found that some tasks can be solved with fewer tokens and that wasted effort is linked to unpredictable actions. They created a new learning method called HERO that teaches agents to focus on solving tasks efficiently without losing important details. Tests showed HERO balances task success and token use better than previous methods.
What this means in practice
- •For software development teams: Reduce interaction costs and latency in automated coding assistants by optimizing token use while maintaining task success rates.
- •For customer support automation teams: Enhance natural language interaction efficiency in multi-turn conversational agents by prioritizing task resolution alongside message length.
Authors
Haobin Li, Liang Jiang, Zhenyu Huang, Mouxing Yang, Xi Peng
Abstract
Recently, coding agents have emerged as a dominant paradigm for real-world software engineering (SWE) scenarios, which solve complex tasks through multi-turn interactions with development environments. However, frequent interactions with environments would inevitably introduce substantial token overhead, leading to high usage costs and latency. Although recent studies have explored reducing token usage by context manipulation and interaction limits at inference time, these approaches focus on improving token efficiency while overlooking the risk of discarding task-relevant information, thus struggling to balance the trade-off between resolution rate and token efficiency. In this paper, we study a more general paradigm without suffering from the limitation, i.e., training token-efficient coding agents with promising resolution performance, which is a highly-practical yet less-explored problem. To this end, we reveal two core observations in SWE scenarios: i) Efficiency Variation: successful resolution could be achieved with fewer tokens; ii) Entropy Correlation: unproductive behaviors are associated with turn-level entropy. Motivated by observations, we propose a novel reinforcement learning framework, dubbed HERO. Specifically, HERO prioritizes task resolution over token efficiency during policy optimization and encourages efficient reasoning patterns at both trajectory and turn levels. Extensive experiments on SWE-bench Verified and SWE-bench Multilingual demonstrate that HERO achieves a favorable trade-off between resolution rate and token efficiency compared with state-of-the-art coding agents and reinforcement learning methods.