WarpDRF clarifies safe warp-level programming on GPUs

WarpDRF: Data-Race Freedom for Warp-Level Programming

Programming Languages

Summary

Modern graphics processors run many threads in groups called warps that work closely together using special commands. These commands rely on threads cooperating without causing data conflicts, but the exact rules for this are complicated and weren't clearly defined before. The authors created WarpDRF, a precise way to describe when these threads can safely share data and synchronize. They tested this with real GPU languages and built tools that find programming errors in GPU code using these warp commands. This helps programmers write faster and more reliable GPU programs.

What this means in practice

  • For gpu kernel developers: Use WarpDRF rules and tools to detect and fix hidden data races in GPU code using warp-level primitives.
  • For static analysis tool builders: Build data-race detection features for GPU programming languages based on the WarpDRF formal model and conformance tests.
  • For graphics engine developers: Integrate WarpDRF-compliant checks to improve the reliability of performance-critical GPU shaders and compute workloads.$Commercial implications: Supports selling more reliable graphics/software development tools that prevent concurrency bugs in GPU programs.

Authors

Zheyuan Chen, Simon Kagle, Tiago Cogumbreiro, Tyler Sorensen

Abstract

Warp primitives such as tensor core operations, shuffles, reductions, and barriers are critical to high-performance GPU kernels, and every major GPU language supports some set of them. The threads that participate in a primitive, and therefore synchronize, are determined dynamically by how threads diverge and reconverge, and by intra-warp scheduling such as independent thread scheduling. In practice, many high-performance kernels use these primitives and behave as expected, following an intuitive but unwritten data-race-freedom contract that has never been stated precisely or empirically tested. We present WarpDRF, the first abstract warp programming model to make this contract precise, with participation rules parameterized by reconvergence guarantees and per-primitive requirements so that instantiations match different GPU languages. We prove (formalized in Rocq) that a kernel satisfying WarpDRF executes every warp primitive with the participants the reference semantics assigns and produces the same results, so a programmer can reason in the reference semantics alone. We implement the model in MLIR with a reference interpreter and fuzz 10K conformance tests for each of three configurations across CUDA, HIP, HLSL, Metal, and SPIR-V on 16 device and backend pairs; at least one WarpDRF configuration describes each, with CUDA honoring the strictly weakest configuration. Finally, we extend Faial, a static data-race analyzer for CUDA, into the first DRF checker for an empirically validated warp model, and apply it to llama.cpp, where most kernels that use warp primitives already satisfy the contract, but three contain previously unknown data races, showing the need for tools that check it.