DeltaSelect cuts coding agent test costs while tracking performance
DeltaSelect: Affordable A/B Testing for Coding Agents
Software EngineeringArtificial IntelligenceMachine Learning
Summary
Testing how well coding agents work can be expensive and sometimes unreliable because single test runs don’t always reflect the overall performance. The authors created DeltaSelect, a tool that picks a small, affordable set of tasks that closely follow full benchmark results. This helps developers make consistent decisions faster and cheaper during development, without running full expensive benchmarks every time. They demonstrated cost savings and better scoring on repeated tests with a popular coding agent.
What this means in practice
- •For software development teams: Reduce cost and time of testing coding agent improvements by using a small, reliable task set to track performance differences.
- •For automated code evaluation services: Offer more affordable, repeatable A/B testing features that approximate full benchmarking with fewer resources.$Commercial implications: Enables commercial API providers to sell cost-effective coding agent testing at scale to software companies.
Authors
Nicholas J. Conn
Abstract
Coding-agent benchmarks are built for broad and comprehensive comparisons, not frequent development decisions. Individual runs vary, full suites are expensive, and the benchmark harness may differ from the harness used in practice. In a resampling analysis of DeepSWE's published trials, only 19.5% of tasks (22 of 113) had a fifth-percentile Pearson correlation of at least 0.50 with full-benchmark performance. The paper presents DeltaSelect, an open-source method that identifies tasks whose one-run results consistently track full-benchmark performance using Pearson correlation, maps fractional verifier results to a common score using linear regression, and selects a fixed task set within a dollar budget. DeltaSelect is intended for repeated baseline-versus-candidate comparisons during development, not model rankings. In a gpt-5.6-luna low-reasoning case study, DeltaSelect was used to revise custom skills and instructions. Across 13 evaluations, the recorded cost was USD 27.86 at rates published August 16, 2026. The adopted version cost 58.1% less than the initial version (USD 1.75 versus USD 4.18; p=0.008), while the calibrated score was higher (42.36% versus 36.46%; published-analog variance p=0.326).