Language models generate gpu kernels but speed benefits are limited

How Much of a Real Workload Can LLM-Generated GPU Kernels Actually Reach?

Distributed, Parallel, and Cluster ComputingArtificial Intelligence

Summary

The paper looks at how well computer programs written by language models speed up GPU computing tasks compared to traditional methods. Some advanced models can write nearly correct programs for most tasks and speed up a few by about 1.2 times, but open-source versions perform worse. The researchers studied how much of actual computing time these programs could improve and found it ranges widely from under 10% to about 58%. For common AI models like transformers, improvements in kernel speed translate to only about a 1% overall speed gain. They also created a new benchmark for recommender system kernels and found modest improvements there too. The paper highlights issues in testing if kernels produce correct results and suggests better tests.

What this means in practice

Authors

Gaurav Agarwal, Ashish Garg, Isha Singhal

Abstract

Language models can now write GPU kernels that outperform PyTorch. We evaluate five model configurations on KernelBench level 1 and find that a frontier model produces correct kernels for 91.1% of problems and independently verified speedups on 22 of 56, including three convolutions, with a median of 1.235x. Open-weights models are far behind: the best reaches 30.4% correct with three verified speedups and solves zero convolutions. We then ask a question the literature does not: what fraction of a real model's wall clock do such kernels govern? Profiling seven workloads across three domains, we find the addressable fraction ranges from 8.9% to 58.2%. On transformers, 80-86% of runtime is spent in cuBLAS GEMM and FlashAttention, bounding realistic end-to-end improvement at roughly 1%, and the fraction shrinks with model scale. On recommenders it is 58.2%, concentrated in a single embedding kernel. We introduce DLRM-Bench, 12 recommender kernel problems in KernelBench format, and measure a 41.7% win rate at a 1.552x median there, projecting 8.63% end-to-end. Separately, we show that KernelBench's correctness check (torch.allclose with an absolute tolerance) is satisfied by a tensor of zeros on 4 of 60 level-1 problems. Two kernels in our own results exploited this before we detected them, including one scored at 283x that wrote 0.3% of its output buffer. We propose scale-invariant replacements and release all 879 evaluations.
View PDFOpen arXiv