Large language models benchmarked on high performance computing code optimizations
HPC-Bench: A Comprehensive Benchmark for High Performance Computing Codes
Performance
Summary
Making computer programs run faster in high-performance computing is tricky because it depends on detailed, low-level code and how data moves around in memory. The authors created HPC-Bench, a new test that measures how well large language AI models can improve these computing codes across many real-world tasks. The test checks both if the optimized code still works correctly and if it runs faster. They showed that finding optimizations that are both correct and much faster is difficult for today’s AI models.
What this means in practice
- •For hpc software developers: Test and improve AI tools that optimize parallel and GPU-based high-performance computing code for better speed and reliability.
- •For ai tool integrators: Evaluate AI code generation models’ ability to produce efficient and correct HPC kernel code across diverse workloads, aiding product selection.
Authors
Bowen Cui, Junyu Yin, Tejas Ramesh, Leo Lim, Oscar Hernandez, Keren Zhou
Abstract
Large Language Models (LLMs) have shown strong capabilities in code generation and reasoning, motivating extensive benchmarks for evaluating functional correctness. However, performance optimization in High-Performance Computing (HPC) poses fundamentally different challenges that are not captured by existing evaluations: HPC performance depends on low-level code, intricate memory access patterns, and computational kernels embedded within sophisticated execution contexts. We introduce HPC-Bench, a performance-grounded benchmark for evaluating LLMs as optimizers of computational kernels in realistic HPC workloads, including 114 workloads across 16 HPC computational motifs. HPC-Bench provides a fully automated, end-to-end evaluation pipeline that combines program-level correctness verification with two correctness-gated metrics, Fast@k and Speedup@k, to characterize reliability and speedup under a fixed sampling budget. Our evaluation of three representative LLMs across serial CPU, OpenMP, and CUDA settings shows that stricter speedup requirements substantially reduce the probability of finding a correct and fast optimization. HPC-Bench is available at https://github.com/Deep-Learning-Profiling-Tools/HPC-Bench.