Technical Report: AI-Assisted Gated DeltaNet Optimization on NVIDIA Blackwell
Distributed, Parallel, and Cluster Computing
Summary
The authors studied how AI helps make GPU programs faster, focusing on a programming contest entry for NVIDIA GPUs. They found that just improving the main compute part of the program (the kernel) isn't enough to get the best speedups. Instead, you need to rethink the whole system, including how the algorithm works, how tests are measured, and how humans guide the process. Their work highlights that optimizing GPU code with AI is a complex task involving many parts, not just the code itself.
Authors
Hyunjun Shin, Jiseung Jang, Jaewoo Maeng, Hyunjun Kim
Abstract
AI-assisted GPU programming is often framed as a kernel-generation loop: ask a model to produce faster CUDA code, benchmark the result, and repeat. This case study argues that contest-grade optimization involves more than improving the kernel body. We examine the Agent-Assisted submission by our team, MSInfer, to the MLSys 2026 FlashInfer Contest. The submission optimized Gated DeltaNet decode and prefill on NVIDIA B200/Blackwell and achieved an official $1.58\times$ speedup, with approximate average latencies of $9.315\,μ\mathrm{s}$ for decode and $239.48\,μ\mathrm{s}$ for prefill. Our experience shows that even effective local kernel improvements can plateau when a workload requires structural reformulation and evaluator-aligned measurement. We therefore characterize AI-assisted kernel optimization as an end-to-end systems problem that encompasses algorithm design, workload specialization, measurement tooling, build and evaluation surfaces, evaluator alignment, and human interpretation.