LLMs prove code optimizations beyond compiler capabilities improve speed
Pattern-Driven Evaluation of LLM Code Optimization: Isolating Compiler-Transcendent Semantic Reasoning
Artificial IntelligenceProgramming LanguagesSoftware Engineering
Summary
Measuring how much faster code runs after optimization doesn’t always show if a large language model (LLM) truly improved it. The authors created PatternBench, a new test to find optimizations that existing compilers miss. They check not only speed but also if the intended changes really happen in the code. Their tests show that just looking at speedups misses important details and that training models helps them optimize better. This means future evaluations should look at both performance and whether the specific code changes were made.
What this means in practice
- •For software engineering teams: Improve automated code optimization by verifying that model changes go beyond compiler improvements for better runtime performance.
- •For compiler developers: Use PatternBench to identify code transformations that current compilers miss, guiding new optimization features.
Authors
Wesley Lu, Runzhou Liu, Hongru Du, Xinyao Yi
Abstract
Existing evaluations of LLM-based code optimization primarily measure end-to-end speedup, obscuring whether a model actually performs the intended optimization. For compiled languages, this is especially problematic: observed gains may reflect transformations already recoverable by modern optimizing compilers rather than optimization behavior attributable to the model itself. We present PatternBench, a benchmark for studying compiler-transcendent optimization, i.e., performance improvements that remain after aggressive compiler optimization. PatternBench targets residual optimization opportunities across seven classes of transformations and pairs runtime evaluation with an AST-based optimization faithfulness analysis that verifies whether the intended optimization is realized in the generated program. Experiments with frontier, open-weight, and fine-tuned models show that runtime speedup alone provides an incomplete account of optimization capability. Optimization faithfulness is strongly associated with realized speedup, and fine-tuning improves both faithfulness and optimization effectiveness. These results suggest that evaluation of LLM-based code optimization should measure not only performance outcomes, but also whether the target transformation was successfully executed.