AMD GPU features keep some speed gains across real workloads and sharing
Capability Gain from ISA to Shared Execution on AMD MI300A and MI250X
Distributed, Parallel, and Cluster Computing
Summary
The team studied how new AMD GPU features that speed up certain calculations actually perform in real-world use, like in libraries, AI tasks, and when the device runs many jobs at once. They found that some features, like FP8 math, mostly keep their speed advantages in basic math libraries but not always in AI language models because of extra steps. Another feature, 2:4 sparsity, loses about half of its speed benefit in sparse math libraries. How well speed improvements show up when sharing the GPU depends more on hardware queues than number types. The study helps users understand when upgrading hardware or using new math tricks will really speed up their work.
What this means in practice
- •For gpu performance engineers: Assess how much GPU instruction-level speedups translate into real-world library and application speed improvements for AMD MI300A versus MI250X.
- •For data center infrastructure teams: Determine the practical performance gains of FP8 and sparsity features under concurrent GPU workloads to optimize resource sharing policies.
Authors
Aaron Jarmusch, Sunita Chandrasekaran
Abstract
Each new GPU generation is showcased by its peak instruction rates, yet users hardly ever run instructions directly: they reach the hardware's performance through vendor libraries, applications and devices shared with other jobs. How much of a theoretical speedup survives these layers is rarely measured, so a team deciding whether to move to a new generation, or to adopt FP8 or 2:4 sparsity, lacks adequate guidance. AMD's MI300A (CDNA3) adds new FP8 matrix instructions and 2:4 structured-sparse instructions that the predecessor, the MI250X (CDNA2), does not have. We analyze this theoretical speedup gap on AMD's MI300A (CDNA3) against the MI250X (CDNA2) by defining a new metric, capability gain. Capability gain is the fraction of a feature's instruction-level speedup that it keeps at a higher level, such as a library, an application or a shared device. We trace it from ISA microbenchmarks through hipBLASLt and hipSPARSELt to LLM inference and mixed-precision HPL, and onto a device shared by up to eight concurrent streams. FP8 keeps most of its speedup through the library, completing an $8192^3$ GEMM \genCapPkgEndToEndGainEightK{} faster than the MI250X with FP16, but unfused quantization turns it into a per-layer slowdown in an LLM, and 2:4 sparsity keeps at most about half of its speedup in hipSPARSELt. Under queue sharing, FP8's GEMM capability gain rises above 1. How fairly streams share the device depends on the number of hardware queues, not on the numeric precision, yet shared LLM serving gains nothing from FP8.