Verifying and fixing gpu matrix errors with sketching after calculation
Sketching the Error, Not the Product: Post Hoc Fault Recovery for Half Precision GPU Matrix Multiplication
Distributed, Parallel, and Cluster Computing
Summary
Sometimes computers that run AI programs make mistakes without telling us, especially when using fast but less precise math. The authors came up with a way to quickly check for and find mistakes after the math is done, without changing how the math is done. Their method uses special summaries called sketches that spot errors and show exactly where they happened. This helps fix the mistakes and keep AI programs working well without much added time.
What this means in practice
- •For gpu system engineers: Detect and locate silent data errors in GPU half precision matrix multiplications after they are computed.
- •For machine learning infrastructure teams: Improve error recovery during large AI training to reduce the impact of hardware faults on model quality.
Authors
Pranav Napolean, Vikas Srivastava, Napolean Periathambi
Abstract
Silent data corruption (SDC) from defective accelerators now interrupts large scale training, yet deployed mitigations act on whole nodes. Algorithm based fault tolerance (ABFT) for a single GEMM has to be fused into the kernel or encode the operands, and it localizes at most one error per checksum. We present FP-Sketch, a verifier that runs after an unmodified tensor core GEMM whose half precision operands are accumulated and delivered at FP32. A sum sketch detects corruption on every call. Hashed first moment sketches, confirmed by independent recomputation, then localize several corrupted entries with no false positives by construction, and each fault yields a coordinate and a magnitude for fleet diagnosis. In floating point, sketch noise rather than bucket collisions limits localization. We measure that noise and find that its constant depends on the BLAS and the operand format and that the bucket count must grow as $n^{2.57}$ for a square product. Sizing the bucket count by measured noise rather than by a fitted power of $n$ raises recovery on eight transformer shapes from 0.402 to 1.000, and measuring the noise at run time adapts the bucket count to the kernel and the model. Instruction level injection with NVBit shows that upsets in a live accumulator are often only 2 to 9% of a typical entry, a population that output side injection cannot produce. Output side injection recovers every fault, while under NVBit the same engine sized for faults of typical magnitude recovers 0.550, and sizing for the measured magnitudes restores 1.000. On Llama-2-7B, guarding the MLP down projections removes 99.4% (BF16) and 99.9% (FP16) of the perplexity damage caused by 2048 bit flips, and the clean path probe costs 0.78 to 3.06 ms against GEMMs of 0.35 to 12.47 ms.