DSEffi-Bench: Demystifying Large Language Models' Capability in Efficient Data Science Code Generation
2026-08-31 • Software Engineering
Software Engineering
AI summaryⓘ
The authors created a new benchmark called DSEffi-Bench to measure how fast code generated by large language models (LLMs) runs, not just whether it works. They tested 16 different models on 1,000 data science tasks and found that being correct doesn’t always mean being efficient. Some models with lower correctness scores still produced more efficient code. Their analysis showed many inefficiencies come from specific domain issues rather than just algorithm complexity, and they demonstrated that using these insights can help improve efficiency.
benchmarklarge language modelscode generationdata science librariesexecution efficiencyalgorithmic complexitymodel evaluationtaxonomyoptimizationcode correctness
Authors
Zhihao Gong, Junzhe Yu, Dong Huang, Zeyu Sun, Jie M. Zhang, Dan Hao
Abstract
Current data science (DS) code generation benchmarks equate correctness with quality, overlooking execution time differences that span orders of magnitude between correct solutions. We introduce DSEffi-Bench, the first benchmark specifically targeting execution efficiency in LLM-generated DS code, comprising 1,000 instances across 10+ DS libraries with stress-testing harnesses and human-validated references. Evaluating 16 models across 3 tiers, we find that correctness alone fails to characterize efficiency: GPT-5.4 leads in correctness (Pass, 66.9\%) but its efficiency score (B$|$P, 71.7\%) nearly matches GPT-5.4-mini (71.6\%), which solves 47 fewer tasks; Kimi-K2.5 ranks lowest in correctness among frontier models (40.2\%) yet achieves the highest efficiency score (73.6\%) across all 16 models. A human-annotated five-category taxonomy reveals that 79.1\% of efficiency deficits extend beyond algorithmic complexity to domain-specific root causes, with distinct failure profiles across model tiers and libraries. Two exploratory experiments provide initial evidence that these diagnostics can guide improvement, yielding up to +14.7\% efficiency gains via taxonomy-guided optimization and approaching Claude-Opus-4.6 Best@3 in efficiency at 13.0$\times$ lower cost via library-conditioned routing.