Papers for

automated code evaluation services

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

DeltaSelect cuts coding agent test costs while tracking performance

DeltaSelect: Affordable A/B Testing for Coding Agents

Abstract: Coding-agent benchmarks are built for broad and comprehensive comparisons, not frequent development decisions. Individual runs vary, full suites are expensive, and the benchmark harness may differ from the harness used in practice. In a resampling analysis of DeepSWE's published trials, only 19.5% of tasks (22 of 113) had a fifth-percentile Pearson correlation of at least 0.50 with full-benchmark performance. The paper presents DeltaSelect, an open-source method that identifies tasks whose one-run results consistently track full-benchmark performance using Pearson correlation, maps fractional verifier results to a common score using linear regression, and selects a fixed task set within a dollar budget. DeltaSelect is intended for repeated baseline-versus-candidate comparisons during development, not model rankings. In a gpt-5.6-luna low-reasoning case study, DeltaSelect was used to revise custom skills and instructions. Across 13 evaluations, the recorded cost was USD 27.86 at rates published August 16, 2026. The adopted version cost 58.1% less than the initial version (USD 1.75 versus USD 4.18; p=0.008), while the calibrated score was higher (42.36% versus 36.46%; published-analog variance p=0.326).

Thu 17 SeptSoftware EngineeringArtificial IntelligenceMachine Learning
The gist
Testing how well coding agents work can be expensive and sometimes unreliable because single test runs don’t always reflect the overall performance. The authors created DeltaSelect, a tool that picks a small, affordable set of tasks that closely follow full benchmark results. This helps developers make consistent decisions faster and cheaper during development, without running full expensive benchmarks every time. They demonstrated cost savings and better scoring on repeated tests with a popular coding agent.
Open → 2609.19607v1