Coding agents save energy by managing reused context carefully
The Physical Cost of Context: Quality-Constrained Resource Accounting for Coding Agents
Software EngineeringArtificial Intelligence
Summary
When computer programs called coding agents talk to large language models, they send a lot of repeated information each time. This recycled context can either save or waste effort depending on how well it is managed. The authors found that some ways of cutting down the repeated text actually cause more work and energy use if they discard reusable parts. Their study measured the real costs like time and energy, not just token counts, showing that careful history reuse saves more physical work than simply shortening prompts.
What this means in practice
- •For software engineers: Lower energy costs in AI applications by designing prompt reuse that maximizes caching and reduces unnecessary recomputation.
- •For cloud infrastructure teams: Optimize GPU resource use in model serving by applying analysis techniques that measure physical cost beyond token count.
Authors
Zhongming Yu, Yitong Ma, Hengjia Yu, Boqin Yuan, Xiangye Lin, Hejia Zhang, Jishen Zhao
Abstract
Coding agents resend a growing context on every model call, and context policies are usually judged by the prompt tokens they save. Executed work also depends on call count and prefix reuse: a caching runtime prefills only the tokens after the longest cached prefix. A shorter prompt can therefore cost more if it rewrites cached history. Our quality-gated protocol splits executed prefill into logical prompt minus reused prefix and measures KV residency, time, and energy. On 450 localization cases derived from SWE-rebench V2, preloading ten retrieved blocks (Eager) and adding anchored history compaction (Compact) both pass a five-point non-inferiority gate. In a separate 900-cell study, Eager cuts model calls by 35.7% and the GPU-energy proxy by 33.2% against on-demand search (Grep); Compact saves nothing further. Enabling prefix caching leaves Eager's prompt volume similar (+8.3%; its 95% CI spans zero) but cuts its uncached prefill by 67.7%. Deleting history can turn a token saving into extra work. With mini-SWE-agent on 100 SWE-bench Verified instances, keeping only the last five observations (Mask-5) cuts Eager's prompt volume by 33.3% but loses 135.9k reusable tokens per task. Uncached prefill rises by 608.5%, the energy proxy by 125.6%, and the policy misses the quality gate. Policies thus save physical work by shortening the trajectory and can multiply it by editing reusable history.