SWE-Serve measures real correctness of AI inference feature changes
SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving
Artificial IntelligenceSoftware Engineering
Summary
Building new AI features for production systems is complex and requires many coordinated code changes. The authors created SWE-Serve, a special set of tasks that test if an AI agent can successfully implement these production-level changes, not just small isolated code snippets. This benchmark shows a big gap between passing local tests and fully correct production deployment, highlighting difficulties AI systems face in real-world engineering. SWE-Serve helps track progress toward AI agents that correctly handle complex, multi-part engineering jobs in live systems.
What this means in practice
- •For machine learning engineers: Evaluate AI agents’ ability to implement production-ready inference features beyond local correctness.
- •For software quality assurance teams: Use SWE-Serve to validate that inference feature code changes pass full end-to-end production tests.
Authors
Jennifer Williams, Dave Farris, Jeff Farris, Jiantao Jiao
Abstract
We introduce SWE-Serve, a benchmark for evaluating agents on production inference engineering tasks. Implementing an inference feature can require coordinating multiple changes across the serving stack, including model support, runtime execution, and public APIs. Existing benchmarks provide limited coverage of production inference engineering: repository-level software engineering benchmarks do not target inference, while general terminal-agent benchmarks include only a few inference tasks. Dedicated inference benchmarks, meanwhile, focus primarily on isolated kernel generation or performance optimization rather than repository-scale production feature implementation. SWE-Serve provides 53 repository-grounded tasks derived from recent production changes to SGLang, spanning six inference engineering families. Each task executes on either CPU or a single GPU (H100) and is evaluated with hidden functional and regression tests, including, where applicable, end-to-end (E2E) serving tests and calibrated performance gates. Executable no-op and oracle controls, adversarial verifier review, and closed-book execution support task validity and evaluation integrity. Across 11 models and 31 model-effort configurations, the best-performing configuration achieves 75% mean pass@1. SWE-Serve exposes a substantial gap between completing tasks locally and achieving production correctness. On 19 tasks with end-to-end coverage, model-serving E2E tests reject roughly one-third of patches that pass every other test (45.9% under the verifier versus 69.4% with E2E tests excluded from scoring), with pass rate increasing for each model's best-performing configuration. By making the production correctness gap directly measurable, SWE-Serve enables the field to track whether future agents move beyond completing tasks locally to achieving production correctness.