Papers for

gpu kernel developers

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Flash attention method speeds up transformer training on Nvidia Blackwell GPUs

FLAMIA: FLAsh MIcroscaling Attention for Fast and Causally Safe Training on Blackwell

Abstract: Attention remains a substantial computational cost in Transformer training, particularly at long context lengths. NVIDIA Blackwell GPUs provide high-throughput low-precision matrix multiplication, but exploiting it in attention requires addressing quantization overhead, causal information leakage through shared scales, and numerical inconsistencies between the forward and backward passes. We present FLAMIA, a family of attention kernels for Blackwell with MXFP8 and mixed NVFP4/FP8 training paths. Our design combines token-block scaling that reuses quantized operands across matrix orientations, targeted safeguards against scale-induced leakage, and consistent forward and backward computations. Fused quantization and architecture-specific kernel optimizations reduce overhead, while document-local scaling and workload-aware scheduling support variable-length documents directly in packed layouts without repacking. Across various causal packed-document workloads, the MXFP8 path accelerates the combined forward and backward attention call, including quantization, by factors of $1.10$--$1.40\times$ on B200 and $1.06$--$1.42\times$ on B300 relative to the fastest evaluated baseline for each workload. We validate training stability and quality preservation on pretraining Llama-style models. These results show how jointly designing quantization, training numerics, and document scheduling translates low-bit hardware throughput into practical attention acceleration.

Mon 2 NovMachine Learning
The gist
Training transformers, a type of AI model, needs a lot of computing power because of a step called attention. The authors found a way to make this step faster on NVIDIA Blackwell GPUs using clever tricks with low-precision math and careful scheduling. Their method, called FLAMIA, keeps training stable and accurate while reducing the time needed. This helps AI models learn more quickly without losing quality.
Open → 2611.03344v1

Low precision GPU kernel tests made accurate with new variable tolerance method

Beyond atol and rtol: Sound Test Oracles for Low-Precision GPU Kernels

Abstract: Low-precision GPU kernels are commonly tested against a reference using hand-picked absolute and relative tolerances. Legitimate numerical error varies with the input, format and precision plan, so a fixed tolerance trades false alarms for missed faults. We present numoracle, which derives per-element acceptance regions from precision-annotated reference programs. Its calculus tracks shared rounding errors, provenance-ordered products, block-scaled quantizers and calibrated math-library errors. A rounding-interval check detects faulty stores that symmetric bounds can admit. Soundness is conditional on explicit rounding, hardware, domain and implementation assumptions; a fixed-input mode removes the zero-mean rounding assumption. We evaluate 13 Triton operators from Liger-Kernel, Unsloth and vLLM on two GPU generations. The main campaigns produce no false alarm on 33,960 correct-kernel and behavior-preserving-variant runs per GPU. Under input shift, the GPU campaigns record no false alarm while tuned tolerances flag up to 30% of correct runs; final-version reference checks identify inputs without complete finite-range certificates. On the RTX PRO 6000, numoracle kills 116 of 124 effective mutants, versus 59-83 for four fixed baselines. Among 400 tuned tolerance configurations, requiring at most 1% false alarms in every evaluated Liger input group permits at most 31 of 53 main-mutant kills, versus numoracle's 50. Historical replays expose accuracy faults in eight change pairs without rejecting their matched corrected controls; a ninth change concerns rounding parity.

Wed 28 OctMathematical SoftwareProgramming LanguagesSoftware Engineering
The gist
Testing programs that use low-precision math on GPUs usually involves checking if results are close enough to a reference using fixed tolerance values. These fixed tolerances can wrongly flag correct results or miss subtle faults. The authors developed 'numoracle,' a method that calculates personalized tolerance ranges for each output based on detailed error tracking, making tests more reliable. They tested numoracle on many GPU programs and found it greatly reduces false alarms and catches more real errors than traditional fixed thresholds.
Open → 2610.38983v1

Subspace Clifford Networks cut training time and parameters in image classification

Subspace Clifford Networks

Abstract: In this work, we propose Subspace Clifford Networks, a reformulation that replaces global cyclic rolling with grouped subspace interactions. Instead of sampling sparse cyclic diagonals across the entire $D$-dimensional channel space, we project the features into an aligned space $\R^{\tilde{D}} = \bigoplus_{g=1}^{G} V_g$ partitioned into $G$ low-dimensional groups $V_g \cong \R^k$ with $k \in \{2, 3, 4\}$. Within each $k$-dimensional group, we compute all $\binom{k}{2}$ wedge coordinates and concatenate them with $\operatorname{SiLU}$-activated coordinate-wise products, while sandwiching the grouped interaction between learnable $1\times 1$ linear projections to enable cross-group channel mixing. This design reduces intermediate feature expansion from $2|\mathcal{S}|D$ (typically $10D$ for 5 shifts) down to $1.5\tilde{D}$ ($k=2$), $2.0\tilde{D}$ ($k=3$), or $2.5\tilde{D}$ ($k=4$). Furthermore, exploiting the small, fixed group size $k \in \{2, 3, 4\}$, we implement a fused CUDA kernel that evaluates the forward interaction and a recomputation-based analytical backward pass using thread-local variables without allocating global-memory buffers for elementwise intermediates. Finally, we extend the backbone from an isotropic columnar layout to a multi-resolution Hierarchical Clifford Network with configurable per-stage subspace dimensions. Experiments on CIFAR-100 show that the proposed models substantially reduce parameter count and training time across both regimes, while the hierarchical variants also improve classification accuracy over the evaluated shift-based hierarchical baselines.

Tue 27 OctComputer Vision and Pattern Recognition
The gist
Training neural networks to recognize images can take a lot of time and computing power. The authors propose a new way to organize how the network’s internal features interact, grouping them in small sets rather than mixing all together. This results in fewer parameters and faster training while maintaining or improving accuracy. They also design efficient code that runs these grouped interactions quickly on GPUs. Their experiments on image classification show that their method speeds up training and performs better than some existing approaches.
Open → 2610.37226v1

Tensor memory accelerator usage rules improve GPU kernel performance

When and Why to Use TMA: A Workload-Grounded Characterization of Tensor Memory Acceleration on NVIDIA Hopper GPUs

Abstract: The Tensor Memory Accelerator (TMA), introduced on NVIDIA Hopper, is used pervasively in production GPU kernels yet its usage is conspicuously selective: expert kernel developers in CUTLASS, FlashAttention-3, and FlashInfer apply TMA to some tensor operands within a kernel but fall back to cp.async, direct global loads, or distributed shared memory (DSM) for others. Vendor documentation describes the API, and recent microbenchmarking work characterizes TMA throughput in isolation, but neither answers the question practitioners actually face: given a memory operand in a kernel, should it use TMA, and if so, with or without cluster multicast? We address this gap with a workload-grounded characterization study on H100. We catalog the TMA, multicast, DSM, and cp.async choices made inside FlashAttention-3 forward, isolate the discriminating features through targeted microbenchmarks (transfer-size crossover, multicast scaling), and produce rules of thumb for the load-mechanism choice on Hopper. We further compare TMA multicast against DSM as competing cluster-broadcast primitives, and isolate the FlashAttention-3 paged-key-value (KV) TMA-vs-cp.async dispatch on H100 across page size, datatype, model shape, and execution phase. Our results yield concrete, predictive guidance on when TMA pays off and when multicast helps or hurts on Hopper, and we extend the analysis to server Blackwell (SM100) via a CUTLASS GEMM study on B200 that shows the qualitative dispatch rules carry across generations while absolute crossovers shift with tensor-core and bandwidth scaling.

Mon 26 OctDistributed, Parallel, and Cluster ComputingHardware Architecture
The gist
Modern NVIDIA GPUs have a feature called the Tensor Memory Accelerator (TMA) that can speed up data loading in complex computations. But experts don’t always use TMA for every part of their work, switching to other methods depending on the situation. The paper studies when TMA and related techniques are best used, based on real-world workloads. This helps programmers decide which memory method to pick to get faster results on NVIDIA’s Hopper and Blackwell GPUs.
Open → 2610.34907v1

Speculative decoding speeds up certain mixture-of-experts models with deduplication

Speculative Decoding for Mixture-of-Experts Models: Expert Bytes, Kernel Bandwidth and the Best Draft Length

Abstract: Speculative decoding checks several drafted tokens in one verification pass. On a mixture-of-experts (MoE) model, consecutive tokens often select the same experts. A deduplicating engine reads each selected expert once per pass, so it requests fewer bytes than a per-slot engine, which reads it once per token. We write a step's time as the bytes each kernel requests over its effective bandwidth, plus fixed time. Under geometric acceptance, we derive from this step time when speculation is faster than plain decoding. The best draft length is a monotone staircase in the acceptance rate. We bound how a bandwidth error moves each result, and we check the main steps in Lean. Deduplication is faster exactly where its share $\bar u(n)$ of the per-slot bytes is below its ratio $r(n)$ of the per-slot bandwidth. On llama.cpp Metal, MLX-LM and llama.cpp CUDA, which we modified to deduplicate, the per-slot kernel has the higher effective bandwidth (r = 0.32 to 0.75). Thus deduplication is faster only from 5 to 8 tokens per pass, or never. We fit the bandwidths on one layer's expert operation, run alone, in under an hour per model. Bytes from routing traces over these bandwidths predict the time that deduplication saves. The measured saving lies inside its error bound at every point on CUDA and MLX-LM and at 36 of 48 points on Metal. The crossover is the first pass size at which deduplication is faster. The prediction places it within one pass size in 10 of 12 configurations of engine, model and depth. We computed these predictions after the measurements. On DeepSeek-V2-Lite, operation times measured before its speculative runs give the best draft length on CUDA within one at all 22 acceptance values.

Sun 25 OctMachine LearningDistributed, Parallel, and Cluster ComputingPerformance
The gist
Mixing experts in AI models can slow down the process when they repeatedly access the same expert parts. The authors study a way to speed this up by checking multiple guesses at once and only reading each expert once. They find when this approach is faster depends on how fast data can be read and how many tokens are checked together. Their method predicts the best number of guesses to try that saves the most time under different setups and hardware.
Open → 2610.34440v1

Racer finds hidden bugs in gpu kernels with symbolic checking

Racer: Hunting GPU Kernel Bugs with Symbolic Equivalence Checking

Abstract: GPU kernels are critical to the performance of today's AI services, yet the same operators are rewritten repeatedly: for each GPU generation, numeric format, serving regime, and kernel language. Every rewrite must preserve the function computed by the implementation it replaces. Today, this obligation depends almost entirely on differential testing: run the new kernel and a trusted reference on randomly sampled tensors, then compare their outputs within a tolerance. This approach has blind spots that additional test cases cannot eliminate. For example, a logical bug may produce an error smaller than the tolerance required to accommodate a correct optimized kernel. We propose symbolic equivalence checking, which executes a kernel and its reference on symbolic inputs, turning each output cell into an exact algebraic expression, and then checks whether the paired expressions agree for all inputs. Because these expressions are exact, symbolic equivalence checking distinguishes floating-point error from logical bugs, while a single check covers the entire input space. We implement Racer, a PTX-level symbolic equivalence checking engine, with three techniques that make this approach practical for production kernels: (1) Shape Reduction to control checking cost, (2) a lockstep CTA scheduler to handle concurrency, and (3) idiom templates with exact numeric refutation for otherwise inconclusive results. Across 2,408 production kernels from 41 collection targets, Racer found 82 real bugs in 23 libraries, including 76 previously unknown bugs and 41 confirmed so far.

Tue 20 OctSymbolic ComputationArtificial Intelligence
The gist
Modern GPU programs often have many versions written for different hardware and purposes, and each new version needs to produce the same results as the old one. Normally, people test them by running a few examples and comparing results, but this can miss subtle bugs that hide inside normal-looking outputs. The authors made Racer, a tool that analyzes GPU programs using math symbols instead of just examples. It checks if two versions always produce exactly the same outputs, which helps find errors that regular testing would miss.
Open → 2610.28429v1

NVIDIA Grace Blackwell memory performance varies with placement and use

Understanding NVIDIA Grace-Blackwell Unified Memory Through Placement-Controlled Microbenchmarks

Abstract: NVIDIA Grace--Blackwell systems make CPU--GPU memory increasingly unified at the programming level, but unified addressability does not make the memory system uniform. A CUDA managed allocation on GB200 may be served from GPU-local HBM, accessed through host/system memory, demand-migrated, prefetched, reused after GPU access, or evicted and reloaded under capacity pressure. The problem is that these states share a programming interface while exercising different hardware paths and runtime mechanisms, making a single ``Unified Memory bandwidth'' number misleading. This paper presents a placement-controlled microbenchmark study of CUDA Unified Memory on NVIDIA GB200, using GH200 as a Grace--Hopper baseline and GB10/DGX Spark as a validation platform. Our methodology separates local HBM access, host/system-memory access, explicit movement, managed first touch, prefetch, reuse, access-pattern sensitivity, oversubscription, and CPU--GPU handoff, then summarizes the observations with a measurement-calibrated model of compute, access, migration, eviction, synchronization, and handoff costs. On GB200, a 1~GB STREAM-style triad reaches 3963~GB/s from local HBM but 304~GB/s through host-mapped CPU/system memory; for a 256~MB streaming kernel, cold CPU-initialized managed memory reaches 14.1~GB/s, while managed prefetch and reuse reach 4606~GB/s and 4504~GB/s. Below the capacity boundary, prefetched and reused managed memory remain above 4.6~TB/s, but at 125\% of CUDA-visible memory they fall to 968~GB/s and 1214~GB/s, showing that Grace--Blackwell unified memory must be understood through placement, access history, locality, reuse, and capacity pressure.

Wed 14 OctDistributed, Parallel, and Cluster Computing
The gist
NVIDIA’s Grace Blackwell system uses a unified memory approach that lets the CPU and GPU share memory address spaces, but accessing that memory can behave very differently depending on where the data physically lives. The authors studied how memory performance changes when data is stored in GPU-local memory versus system memory, and how techniques like prefetching and reuse affect speed. They found that simply having unified memory doesn't guarantee consistent speed, and actual performance depends on factors like memory placement, usage patterns, and capacity limits. This research provides detailed measurements and a model to help developers understand and optimize memory use on these systems.
Open → 2610.20354v1

Gpu speedup cuts data conversion time for neuromorphic language models

FlashT2S: GPU-Accelerated Tensor-to-Spike Conversion Operators for Heterogeneous Neuromorphic LLM Backends

Abstract: Artificial intelligence faces two growing challenges: computational bottlenecks and energy inefficiency inherent to the von Neumann architecture. Neuromorphic computing offers a promising alternative, yet its immature ecosystem still relies on conventional systems for data supply, requiring efficient conversion between tensor and spike dataflow. We observe a clear mismatch, where data conversion is slow while downstream computation is fast. As data size grows quadratically with large language models, this mismatch becomes a major bottleneck. In this work, we examine the data-conversion components that appear on the critical path of training and inference in neuromorphic LLMs. The work presents FlashT2S (Flash Tensor-to-Spike): GPU-Accelerated Tensor-to-Spike Conversion Operators, which applies architecture-aware and data-centric optimizations to reduce conversion latency. Experimental results show that FlashT2S delivers an average speedup of 850.78x over the native operators across individual operators and 802.83x in realistic workloads. Additionally, compared with GPU kernels automatically generated by torch.compile, our kernels achieve 5-11x speedups. Notably, it achieves near-optimal absolute performance: large-scale conversions are reduced to tens of microseconds (10-99 us), while small-scale conversions reach single-digit microsecond levels (less than 10 us). Furthermore, its standardized output enables seamless integration across heterogeneous backends, including diverse neuromorphic hardware.

Mon 12 OctDistributed, Parallel, and Cluster Computing
The gist
Large language models use massive amounts of data, and converting this data into a format neuromorphic computers can understand is slow, creating a big delay. The authors show that by designing special GPU-based operators, this conversion can be made hundreds of times faster. Their method, FlashT2S, shrinks the conversion time to just a few microseconds, making it almost instantaneous. This lets different neuromorphic hardware work together more smoothly and efficiently.
Open → 2610.17035v1

Simplified algebra improves layout handling in gpu tensor computations

A Simplified Layout Algebra for GPU Kernels

Abstract: The layout algebra behind NVIDIA's CuTe library describes how tensor tiles map onto threads and memory. A layout is a shape with a stride for each entry, so a coordinate's address is the sum of its indices times their strides. As formalized by Shah and by Cecka, its central operations (e.g., composition, complement, division, product) are guarded by at least six interacting admissibility conditions. We show that this complexity is an artifact of a representational choice, not of the domain. We present a layout algebra in which every layout names the space its inputs and outputs live in: logical, physical, or thread-value indices. A layout into logical or thread-value space must be a bijection from its $N$ coordinates onto $[0,N)$. A layout into physical space must be alias-free: distinct coordinates land on distinct addresses, except under broadcast, where many threads read one value on purpose and the layout must mark where. Nothing maps out of a physical index. Composing $f$ with $g$ requires only that $f$'s range be exactly $g$'s flattened coordinate set. The index arithmetic can always be emitted naively: compute $f$, split the result by division and remainder into $g$'s coordinates, and apply $g$. Often the composite simplifies to a layout; we show that when it does, there is a coarsest such form, whose digits every other one refines, and we emit that. A warp's or thread's slice is a \emph{restriction}. A restricted layout is no longer composable, which forces the user to settle the whole view of a tile before taking slices. We validate the algebra by reproducing within it CuTe's operations, such as composition, complement, and division. We implement it in OCaml and use the implementation to rederive CuTe's atoms and usage patterns. The decision procedure is proved correct in Rocq.

Fri 9 OctProgramming Languages
The gist
Handling data layout for GPU computations can get very complicated, with many rules needed to keep things correct. The authors found that this complexity is due to how the data is represented, not the problem itself. They created a simpler algebraic way to describe how data maps to GPU threads and memory, making it easier to reason about and combine these mappings. They confirmed their approach by recreating the operations of an existing system called CuTe and implemented it in a programming language with formal correctness proofs.
Open → 2610.13679v1

WarpDRF clarifies safe warp-level programming on GPUs

WarpDRF: Data-Race Freedom for Warp-Level Programming

Abstract: Warp primitives such as tensor core operations, shuffles, reductions, and barriers are critical to high-performance GPU kernels, and every major GPU language supports some set of them. The threads that participate in a primitive, and therefore synchronize, are determined dynamically by how threads diverge and reconverge, and by intra-warp scheduling such as independent thread scheduling. In practice, many high-performance kernels use these primitives and behave as expected, following an intuitive but unwritten data-race-freedom contract that has never been stated precisely or empirically tested. We present WarpDRF, the first abstract warp programming model to make this contract precise, with participation rules parameterized by reconvergence guarantees and per-primitive requirements so that instantiations match different GPU languages. We prove (formalized in Rocq) that a kernel satisfying WarpDRF executes every warp primitive with the participants the reference semantics assigns and produces the same results, so a programmer can reason in the reference semantics alone. We implement the model in MLIR with a reference interpreter and fuzz 10K conformance tests for each of three configurations across CUDA, HIP, HLSL, Metal, and SPIR-V on 16 device and backend pairs; at least one WarpDRF configuration describes each, with CUDA honoring the strictly weakest configuration. Finally, we extend Faial, a static data-race analyzer for CUDA, into the first DRF checker for an empirically validated warp model, and apply it to llama.cpp, where most kernels that use warp primitives already satisfy the contract, but three contain previously unknown data races, showing the need for tools that check it.

Tue 6 OctProgramming Languages
The gist
Modern graphics processors run many threads in groups called warps that work closely together using special commands. These commands rely on threads cooperating without causing data conflicts, but the exact rules for this are complicated and weren't clearly defined before. The authors created WarpDRF, a precise way to describe when these threads can safely share data and synchronize. They tested this with real GPU languages and built tools that find programming errors in GPU code using these warp commands. This helps programmers write faster and more reliable GPU programs.
Open → 2610.07541v1

GPU kernel knowledge transfer varies across code languages

MetaKernelBench: Measuring GPU Kernel Knowledge Transfer Beyond Code

Abstract: Recent GPU kernel optimization agents retain what they learn in knowledge bases or as distilled skills. Kernel benchmarks score each attempt's implementation for correctness and speed but leave the reuse value of retained experience unmeasured. We introduce MetaKernelBench, which measures whether experience distilled from an attempt in one kernel domain-specific language (DSL) improves a fresh attempt at the same problem in another. Its 74 problems are fused subgraphs in six families, each posed as a pair of CuTe DSL and TIRx variants that differ only in the DSL. The agent first attempts each variant solo and is instructed to distill what it learns into a natural-language skill, which is transferred whether or not the source attempt passes verification. The skill is the only extra input to a skill-conditioned attempt by the same model in the other DSL. We compare each skill-conditioned attempt with the solo attempt on the same variant under matched per-attempt budgets, scoring correctness and end-to-end runtime. Across six models and both directions, paired lift over solo attempts ranges from -19% to +29%. Four models gain in both directions, yet regressions occur on 16% to 45% of problems in every model and direction. Outcomes follow the source attempt's result relative to the target's solo attempt rather than source success alone, improving in 71% of comparisons when the source stands above and regressing in 54% when it stands below. MetaKernelBench complements implementation-quality metrics by measuring same-problem cross-DSL kernel knowledge transfer.

Sun 4 OctMachine Learning
The gist
Making programs that run well on GPUs can be tricky, especially when switching between different coding languages designed for those tasks. The authors created MetaKernelBench, a way to test if what is learned from solving a problem in one GPU programming language can help solve the same problem in another language. They found that sometimes transferring knowledge helps improve performance, but other times it can cause setbacks. The success of this transfer depends more on how good the first attempt is compared to a fresh try in the new language, not just whether the first try passed tests.
Open → 2610.05014v1

SyclKittens boosts Intel GPU AI kernel performance by encoding efficient methods

SyclKittens: A Tile Programming Model for Programmers and Coding Agents on Intel GPUs

Abstract: New AI accelerators arrive before the kernels that make them fast, because peak kernel performance requires architecture-specific expertise in operand pipelines and data-movement techniques. Coding agents can now write, compile, and tune kernels on their own, so they could greatly accelerate kernel development and optimization. What agents produce depends on the interfaces they are given. These interfaces may expose a machine's raw capabilities or encode the known-good methods for exploiting its hardware features efficiently. We test the performance impact of these two interfaces on Intel GPUs through controlled experiments with three coding models on three kernels. With raw SYCL and execution feedback, Opus 4.8, the strongest of the three models tested, writes a single-GPU GEMM kernel reaching only 57.5% of Intel's tuned oneDNN library. We present SyclKittens, a hardware-aware tile programming model for Intel GPUs that encodes these known-good methods, so its operations place operands in matrix-engine layouts, prefetch through the L1 cache, and communicate over the fabric that links GPU stacks into a node. With SyclKittens under the same agent, task, and feedback budget, the agent reaches 82.1%, showing that encoded methods turn hardware capabilities into performance. Workload-specific policies stay programmable, and engineers and agents jointly refine these schedules in SyclKittens to build a kernel suite that reaches ~96% of oneDNN in geometric mean across GEMM shapes. The suite runs Llama-3.1-8B inference 1.59x faster than torch.compile on one GPU and up to 2.91x faster than a matched multi-GPU decode path on Intel's oneCCL. SyclKittens is open source and available at https://github.com/intel/SyclKittens.

Sat 3 OctDistributed, Parallel, and Cluster Computing
The gist
Writing fast programs for new AI chips is hard because it needs deep knowledge of the hardware. The authors show that when coding agents get simple, low-level interfaces, they produce slower programs. But if these agents use SyclKittens, a special programming model with expert tips built-in, they can create much faster programs on Intel GPUs. This leads to big speedups for AI tasks like running Llama models compared to existing tools. The new model is open source for others to use and improve.
Open → 2610.04277v1

FoldAttention speeds up AI decoding with deterministic exact sums

FoldAttention: Declared-Reference Softmax for Fast Decode and Deterministic Backward

Abstract: Autoregressive decode repeatedly streams a growing KV cache, making attention a major cost at long context. Existing high-performance kernels use online softmax, which discovers a row's normalization reference as it scans keys. Earlier contributions therefore remain provisional and may require rescaling. We argue that the reference need not be discovered: softmax is invariant to a common shift, so the reference only has to keep the weights in range. We present FoldAttention, an additive formulation of softmax attention that fixes a finite reference $Z_i$ before scanning the KV cache. Each weight $2^{s_{ij}-Z_i}$ is then final when computed, so contributions add across disjoint key ranges and their quotient equals softmax attention in real arithmetic. We use this property to develop two techniques for Hopper decode: (1) final weights gate key and value reads before the bytes are fetched, and a per-call depth $T$ cuts keys below $2^{-T}$ while keeping their mass, and (2) additive partials compose split KV and shared-prefix cascades without rescaling. On H100 at $T=16$, FoldAttention decodes seven real-model generations 1.36-2.30$\times$ faster than the fastest BF16 baseline, and up to 3.09$\times$ faster across MHA and GQA shapes, at an error within 1.5% of the lowest BF16 error on six of the seven; reading every key, it is 1.14-1.30$\times$ faster at matched error. We validate on Qwen3-8B that a whole decode step is up to 1.46$\times$ faster while likelihood and long-context accuracy match those under BF16 kernels. The same principle makes the backward deterministic: CTAs round bounded partial gradients onto an integer grid declared before the reduction and add them in any order. FoldAttention thereby removes the determinism tax: its deterministic backward is up to 1.84$\times$ faster than deterministic FlashAttention-3/4 and 1.05$\times$ faster than the fastest nondeterministic kernel.

Sun 27 SeptMachine LearningDistributed, Parallel, and Cluster Computing
The gist
When generating text or other sequences step-by-step, computers spend a lot of time calculating attention with huge data stored in memory. The authors introduce FoldAttention, a new way to compute attention that fixes a reference point before reading data, making each weight final when calculated. This lets the system skip unnecessary data early and combine partial results exactly, speeding up the process and making it deterministic. Their approach runs faster than previous methods on modern GPUs without losing accuracy.
Open → 2609.33410v1

TileBench compares tile-based AI programming models for GPU speed

TileBench: A Controlled Benchmark for Performance Evaluation and Bottleneck Diagnosis of Tile-Based Programming Models

Abstract: Tile-based programming models, such as Triton and cuTile, aim to simplify high-performance kernel development, but their practical performance, tuning behavior, and usability remain difficult to compare systematically. We present TileBench, a controlled benchmark for evaluating Triton and cuTile on NVIDIA B200 GPUs under matched operator semantics and comparable implementation structures. TileBench contains 45 operators covering diverse AI-kernel patterns and memory/computation behaviors. Each task provides a PyTorch reference, verified Triton and cuTile implementations, standardized data-types (dtype) and input-size sweeps, default and autotuned configurations, roofline-based metrics, and profiling-guided diagnosis. Our evaluation shows that performance gaps are workload-dependent: cuTile excels on a small cluster of Tensor-Core/TMA-friendly kernels, while Triton is stronger on many irregular, streaming, and bandwidth-bound operators. We further evaluate LLM-generated cuTile and Triton kernels and find that Triton is consistently more token-efficient than cuTile under the same iterative refinement protocol. TileBench is publicly available at https://github.com/Deep-Learning-Profiling-Tools/Tilebench.

Thu 24 SeptPerformanceProgramming Languages
The gist
Building fast programs for GPUs is tricky because different programming models behave differently depending on the task. The authors created TileBench, a test set of 45 AI-related tasks with matching implementations in two popular tile-based programming models, Triton and cuTile. They measured how each model performed on NVIDIA B200 GPUs and found that cuTile works better for some specialized tasks, while Triton performs better on many others, especially those with irregular or bandwidth-heavy operations. They also tested kernels generated by language models and saw Triton was more efficient in refining code. TileBench offers a fair way to compare these models for anyone developing GPU programs.
Open → 2609.29067v1

Tensor program bugs found faster by checking outputs one location at a time

The Output-Space Hypothesis: Enumerative Equivalence Checking for Tensor Programs

Abstract: Tensor programs, as used in deep learning models, are a prime target for optimization, as small performance improvements can have a large impact across training or inference workloads. However, such optimizations are complicated and can produce subtle bugs. Traditionally, correctness is assumed when differential testing against a reference on random inputs fails to reveal bugs. However, the inputs to these programs are massive tensors, and finding bugs can require generating extremely low likelihood inputs with precise relationships among their values. We propose a novel way to find bugs more consistently by flipping the quantifiers. Rather than generating a single input and checking all output tensor locations for equivalence, what if you could check a single output tensor location's equivalence for all inputs? We implement this idea in a system, \dirigo, by using a novel symbolic execution strategy. We demonstrate that \dirigo can find bugs effectively in a public dataset of 6,988 AI-written CUDA kernels that are all marked correct by differential testing. Of these, \dirigo finds 600 kernels that are actually buggy, and finds 97.3\% of those bugs within two minutes.

Thu 17 SeptProgramming LanguagesMachine LearningSoftware Engineering
The gist
Tensor programs used in AI models are tricky to optimize because small bugs can cause big problems. Traditional tests check many outputs for one input, but this misses rare bugs hidden in complex data. The authors flipped this by checking one output location for all possible inputs, catching subtle mistakes more reliably. Their system, Dirigo, found hundreds of bugs that older tests missed, most within minutes.
Open → 2609.19611v1

Argus automates gpu performance tracking across code regions

Argus: Orchestrating Cross-Layer GPU Performance Measurements around Semantic Regions

Abstract: GPU developers and automated optimizers need performance evidence for semantic code regions--such as neural-network operator implementations and pipeline stages--but this evidence is fragmented across profiling tools. Answering a region-level question can require manually constructing probes and program variants, isolating interfering measurements, and mapping evidence to regions and execution contexts. We present Argus, a region-centric measurement planner and runtime that automates this workflow. Clients identify regions with boundary markers and select signals and execution scopes. Argus preserves region identity across compilation, execution, and measurement variants, constructs interference-aware multi-run plans, and orchestrates transformations and profiling across backends. It joins compiler-, hardware-, and system-level evidence using region identity and dynamic execution context, producing reports that record measurement origins and attribution ambiguity. We evaluate Argus across agentic kernel optimization, persistent megakernel optimization, and cross-level PGO. Across 44 persistent-GEMM and attention configurations, Argus improves 39/44 cases and raises AlphaEvolve's geometric-mean speedup from 5.4% to 8.9%. On a persistent TinyLlama-1.1B decode megakernel, an optimization agent reaches 1.65 ms/token with Argus versus 4.92 ms/token without it, producing a kernel $2.1\times$ faster than PyTorch with CUDA Graphs. Finally, Argus-guided cross-level PGO improves compute--communication overlap, increasing throughput by 7% on average across five multi-GPU settings.

Fri 11 SeptDistributed, Parallel, and Cluster ComputingPerformance
The gist
Measuring how different parts of a program run on GPUs is tricky because the information is scattered and hard to get. The authors created Argus, a tool that automatically tracks and organizes performance details for specific code sections on GPUs. Argus helps developers understand performance better by connecting data from different tools and showing how the GPU runs each part. This leads to faster and more efficient programs, especially for complex tasks like neural networks.
Open → 2609.12299v1