Trusted benchmark improves measurement of gpu kernel speedups
AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines
Artificial Intelligence
Summary
Large language model tools try to speed up their GPU computing by testing different settings and keeping the best ones. But the measurements used to decide which settings are fastest are often unreliable or misleading. The authors studied many model runs and found common causes of wrong speedup claims, such as differences between machines and faulty comparisons. They created AutoTuneBench, a strict testing method that ensures all measurements are honest and comparable. Using this method, claimed speedups are much smaller but much more trustworthy.
What this means in practice
- •For infrastructure engineers: Ensure auto-tuning of GPU kernels for serving engines uses trustworthy performance measurements across hardware setups.
- •For performance testing teams: Use AutoTuneBench protocol to detect and eliminate misleading speedup claims in kernel and serving engine benchmarks.
Authors
Li Chen
Abstract
Large language model agents tune GPU kernels and serving engines through a closed loop of propose, measure, and keep, but the measurements behind this loop are not trustworthy. We characterize four failure modes from a four-day pilot corpus of 619 model calls: strawman baselines manufacture speedups, absolute times do not transfer across machines, saturated tasks nullify comparisons, and infrastructure defects impersonate science. We present AutoTuneBench, a benchmark and measurement protocol that makes trust architectural. The protocol is frozen as code with test-enforced provenance; a database-level validator rejects out-of-protocol results; anti-cheat checks run outside the agent's modification surface; comparisons follow pre-registered readouts; and measurements anchor to externally published results, grounded in paired-seed statistics with a 5\% cross-run coefficient-of-variation cap. Honest measurement rewrites the headlines: our best kernel reads 10.6x against a naive baseline but 2.03x against the honest one; one configuration delivers 1.174x on one machine and 1.0049x on another; a pre-registered on/off comparison nulls at a shared wall (2.4840 vs 2.4957\,ms); and the KernelBench Level-1 suite admits 51\% of tasks with median speedup 1.0001x over PyTorch eager. The protocol, the two-engine corpus (vLLM and SGLang), and its audit trail are released as open artifacts.