New method speeds up low-precision matrix math on GPUs by up to 83 times

FP64 Is All You Want, INT8 Is All You Need, FP4/6/8 Is All You Have

Distributed, Parallel, and Cluster Computing

Summary

Calculations using very precise numbers, like FP64, can be slow on graphics processors. The authors studied ways to do these calculations using simpler, smaller number types like INT8 or FP4, which GPUs handle faster. They developed a strategy to pick the best mix of these small number methods that use the fewest slower steps. Their approach found faster ways to do this math on current GPUs, running as much as 83 times faster than the usual precise method. This could help speed up programs that rely on big matrix calculations.

What this means in practice

  • For gpu software engineers: Implement faster matrix multiplication on GPUs by using optimized low-precision computation schemes.
  • For ai hardware architects: Design hardware or firmware that efficiently supports new low-precision floating-point formats for high-speed computation.

Authors

Pratyai Mazumder, Alexandru Calotoiu, Torsten Hoefler

Abstract

Ozaki scheme II emulates FP64 matrix products with INT8 ones through residues modulo pairwise coprime moduli, and variants for FP8 and FP4 have followed. We treat these schemes as one family and pose the choice of a scheme as a combinatorial program that minimizes the number of low-precision GEMMs. Given, for each modulus, a finite set of ways to compute products modulo it from low-precision GEMMs, we find the choice of moduli and ways with the fewest GEMMs, for any format, accumulator and inner dimension, and derive lower bounds on the GEMM count over the whole family. Applied to the formats of current GPUs, the method gives the first FP6 schemes, an FP8 scheme with fewer GEMMs than any previous one, and an FP4 scheme that the bounds show needs the fewest GEMMs of any scheme in the family whose moduli lie in a stated range. Implemented on three Blackwell GPUs, the INT8, FP8 and FP4 schemes run faster than native FP64, up to 83x on B300.
View PDFOpen arXiv