Low precision GPU kernel tests made accurate with new variable tolerance method
Beyond atol and rtol: Sound Test Oracles for Low-Precision GPU Kernels
Mathematical SoftwareProgramming LanguagesSoftware Engineering
Summary
Testing programs that use low-precision math on GPUs usually involves checking if results are close enough to a reference using fixed tolerance values. These fixed tolerances can wrongly flag correct results or miss subtle faults. The authors developed 'numoracle,' a method that calculates personalized tolerance ranges for each output based on detailed error tracking, making tests more reliable. They tested numoracle on many GPU programs and found it greatly reduces false alarms and catches more real errors than traditional fixed thresholds.
What this means in practice
- •For gpu kernel developers: Improve testing of low-precision GPU programs by automatically generating precise error bounds to detect bugs without false alarms.
- •For machine learning infrastructure engineers: Validate low-precision GPU operators in ML frameworks to boost trustworthiness of performance-optimized models without excessive manual tuning.
Authors
Xinyi Shi
Abstract
Low-precision GPU kernels are commonly tested against a reference using hand-picked absolute and relative tolerances. Legitimate numerical error varies with the input, format and precision plan, so a fixed tolerance trades false alarms for missed faults. We present numoracle, which derives per-element acceptance regions from precision-annotated reference programs. Its calculus tracks shared rounding errors, provenance-ordered products, block-scaled quantizers and calibrated math-library errors. A rounding-interval check detects faulty stores that symmetric bounds can admit. Soundness is conditional on explicit rounding, hardware, domain and implementation assumptions; a fixed-input mode removes the zero-mean rounding assumption. We evaluate 13 Triton operators from Liger-Kernel, Unsloth and vLLM on two GPU generations. The main campaigns produce no false alarm on 33,960 correct-kernel and behavior-preserving-variant runs per GPU. Under input shift, the GPU campaigns record no false alarm while tuned tolerances flag up to 30% of correct runs; final-version reference checks identify inputs without complete finite-range certificates. On the RTX PRO 6000, numoracle kills 116 of 124 effective mutants, versus 59-83 for four fixed baselines. Among 400 tuned tolerance configurations, requiring at most 1% false alarms in every evaluated Liger input group permits at most 31 of 53 main-mutant kills, versus numoracle's 50. Historical replays expose accuracy faults in eight change pairs without rejecting their matched corrected controls; a ninth change concerns rounding parity.