Coda speeds coding agent servers by smarter task admission
Coda: Exploiting Admission Flexibility for Coding-Agent Serving
Distributed, Parallel, and Cluster Computing
Summary
Serving coding agents powered by large language models can be slow due to uneven costs of preparing memory states and mixing tasks with different lengths inefficiently. The authors identify that being flexible about when and how requests start can save time and resources without losing progress. Their system, Coda, manages the order and grouping of tasks to reduce computational waste and improve throughput. Experiments show that Coda delivers significantly better performance compared to existing systems.
What this means in practice
- •For cloud service operators: Improve the efficiency of coding-agent AI services by better managing request admission and resource allocation.
- •For ai infrastructure engineers: Optimize multi-worker setups for language model applications by routing requests based on memory state location and task compatibility.
Authors
Youhe Jiang, Fangcheng Fu, Binhang Yuan, Krishna Malladi, Ehsan K. Ardestani, Zhan Shu, Adnan Aziz, Yi Xu
Abstract
Coding agents powered by large language models (LLMs) repeatedly alternate between model inference and tool calls, creating long-lived sessions with reusable key-value (KV) states and asynchronous request resumptions. Logical readiness, however, does not ensure efficient admission in a shared serving system. Through direct trace analysis and trace-driven replay, we identify two mismatches: reusable KV states reside across storage tiers and incur unequal computational costs for state preparation, while requests with heterogeneous context lengths can decode inefficiently together. Our central insight is that exploiting admission flexibility can improve serving performance while preserving request progress. We present Coda, a coding-agent serving system that realizes this insight through a readiness-informed admission layer incorporating two mechanisms. Tiered-Aging state admission exploits bounded flexibility in admission order to improve admission efficiency and preserve request progress. Compatibility-Aware execution admission exploits flexibility in attention grouping within each model iteration to reduce mixed-context interference and improve shared decoding efficiency. For multi-worker configurations, Coda introduces a separate routing layer that considers KV-state residency, worker load, and request-to-worker context-length compatibility to guide efficient request placement. We evaluate Coda across different models and workloads against state-of-the-art coding-agent serving systems and vLLM. Across the single-worker and multi-worker experiments, Coda improves output-token and SLO-compliant throughput by 20.3% and 70.5% on average, with peak gains of 29.3% and 140.2%.