Gpu architecture causes different software fault results
BitIR: Cross-Architecture Fault Injection for Resilience Analysis of Heterogeneous GPU Applications
Distributed, Parallel, and Cluster Computing
Summary
Computer systems that use GPUs from different makers may behave differently when tiny errors occur during calculations. The authors created BitIR, a tool that introduces the exact same small error into programs before they get customized for different GPUs. They tested this on three top supercomputers with GPUs from NVIDIA, Intel, and AMD, finding that errors caused very different outcomes on each one. This means fixing errors and making software reliable needs to consider the specific GPU being used.
What this means in practice
- •For hpc system designers: Test resilience solutions that are adapted to each GPU vendor by using BitIR across different GPU architectures in supercomputers.
- •For software reliability engineers: Identify fault behaviors unique to each GPU backend to create better error detection and prevention methods tailored for NVIDIA, Intel, and AMD GPUs.
Authors
Maisy Dunlavy, Michael Papka, Zhiling Lan
Abstract
Modern GPU-based HPC systems rely on heterogeneous vendor stacks, yet resilience studies are largely limited to single architectures, leaving it unclear how faults behave across different GPU backends. We present \emph{BitIR}, a cross-architecture fault injection framework that injects deterministic single-bit faults at the LLVM IR level, ensuring semantically equivalent perturbations prior to backend lowering and enabling direct cross-vendor comparison across NVIDIA, Intel, and AMD GPUs. We evaluate BitIR on three production supercomputers -- Polaris (ALCF, NVIDIA A100), Aurora (ALCF, Intel GPU Max 1550), and Frontier (OLCF, AMD Instinct MI250X) -- representing the full spectrum of current leadership-class GPU architectures. Using representative heterogeneous benchmarks, we conduct large-scale injection campaigns across all three systems and classify outcomes into masked results, silent data corruptions (SDCs), and failures. Our results show that identical faults produce markedly different behaviors across vendors: Intel most often masks faults but exhibits more hangs when faults escape masking, NVIDIA exposes more detectable hard failures, and AMD alternates between SDC-dominant and failure-dominant behavior depending on the benchmark and fault site. These findings demonstrate that resilience is not backend-invariant, underscoring the need for backend-aware and workload-aware fault mitigation.