CertiSum ensures identical 4-bit training results across GPUs and layouts

CertiSum: Bitwise-Identical MXFP4 Training Across Platforms and Parallelism Layouts

Machine LearningDistributed, Parallel, and Cluster Computing

Summary

Training deep learning models using a 4-bit format called MXFP4 usually leads to tiny differences in results depending on the type of GPU or how the training is split up. The authors develop CertiSum, a method that guarantees the training produces exactly the same numbers every time, no matter the GPU or parallel setup. They achieve this by checking when sums of small numbers are exact and recalculating only when needed, preserving exactness while keeping the training efficient. This helps make debugging and verifying training more reliable across different hardware.

What this means in practice

  • For machine learning engineers: Guarantee reproducible and identical deep learning training results across different GPUs and parallel setups to improve debugging and auditing.
  • For ai hardware developers: Design GPU-compatible training systems that produce exact same numerical outputs across hardware variants for reliable model verification.

Authors

Steve Rhyner, Yubo Gao, Stefan de Lasa, Sankeerth Durvasula, Nandita Vijaykumar

Abstract

Training with MXFP4, a 4-bit microscaling format, is non-deterministic: identical inputs can produce different results across parallelism layouts and GPU vendors. These differences complicate debugging and training audits and can cause training and inference engines to disagree. We present CertiSum to make MXFP4 training bitwise invariant across runs, parallelism layouts, and GPU vendors. Its key insight is that native FP32 accumulation of MXFP4 products is often already exact for every summation order. CertiSum collects statistics during quantization to certify native matrix multiplication results, recomputes uncertified outputs exactly, and uses exact kernels directly where certification rarely succeeds. CertiSum keeps quantization blocks unchanged across layouts and merges partial sums without rounding, including in attention and weight-gradient computation. In 200-step Qwen3-8B-Base experiments, CertiSum preserves training states bit for bit across AMD MI350X and NVIDIA GB200 and across changes in data, tensor, and context parallelism on MI350X. Across three 8B--14B models on FineWeb-Edu using MI350X, it adds 11.8--60.5% to training-step latency relative to its non-exact MXFP4 counterpart at the larger evaluated input shapes, with speedups of up to $1.24\times$ over BF16 and $1.70\times$ over deterministic BF16.