Gpu method ensures identical large language model outputs across devices
Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures
Hardware ArchitectureMachine Learning
Summary
Large language models are supposed to give the same answers when you ask the same questions, but different graphics cards sometimes cause them to give different answers. This happens because of tiny differences in how math operations add up on different GPUs. The authors designed a new way to do these math operations so they always happen in the same order and way, no matter which GPU is used. Their method makes the outputs perfectly match across different Nvidia GPUs while also running faster and using less memory.
What this means in practice
- •For gpu hardware developers: Build GPU inference runtimes that produce consistent LLM outputs on all supported hardware architectures without slowdowns.
- •For machine learning engineers: Deploy LLM applications with guaranteed identical results across different Nvidia GPUs, improving debugging and reliability.
Authors
Liam Cooper, Shinnung Jeong, Hyeran Jeon, Jeffrey Young, Hyesoon Kim
Abstract
Large language model (LLM) outputs are expected to be reproducible under greedy decoding, yet in practice the same model, prompt, and software stack produce different outputs on different GPUs. The root cause is floating-point non-associativity combined with hardware-dependent kernel selection. Inference frameworks select different matrix-multiplication kernels on each architecture, with different parallel reduction orders and unspecified tensor-core arithmetic, and the resulting rounding differences can flip output tokens. Existing solutions have imperfect cross-architecture reproducibility and incur a significant performance penalty. We present a solution employing a set of fixed-configuration fused-upcast GEMM kernels that load 16-bit weights from memory, upcast them to FP32 in registers, and accumulate with IEEE-754 arithmetic in a reduction order that is a pure function of the problem shape and is therefore independent of the device, its SM count, or kernel scheduling. By fixing the floating-point reduction order as a function of problem shape alone, every GPU runs the same operation sequence, so cross-architecture reproducibility of the linear layers reduces to correct IEEE-754 arithmetic rather than to rounding differences staying below a tie-flip threshold. We confirm our solution's linear-layer outputs are bitwise identical across NVIDIA Ampere, Ada, and Hopper GPUs, while running $1.17$ to $3.1\times$ faster end-to-end than the state-of-the-art solution and cutting weight-memory traffic in half.