FuTransformer speeds up Llama model inference on NVIDIA Hopper GPUs
FuTransformer: FP8 prefill and INT4 decode for Llama-style inference on Hopper
Artificial Intelligence
Summary
Transformers are a type of AI model used for language tasks, but running them can be slow and costly. The paper presents FuTransformer, a new way to run Llama-style transformers faster on NVIDIA Hopper GPUs. It uses clever tricks like representing numbers with fewer bits and reorganizing data to reduce the amount of work the GPU has to do. This makes generating text with these models quicker and more efficient without significantly losing accuracy.
What this means in practice
- •For machine learning engineers: Run large Llama-style transformer models faster during inference on NVIDIA Hopper GPUs using lower-precision numeric formats without significant accuracy loss.
- •For cloud ai service operators: Reduce GPU time and resource costs for serving large language models by deploying FuTransformer’s optimized FP8 prefill and INT4 decode on Hopper-based servers.$Commercial implications: Allows AI cloud providers to offer more efficient Llama model inference services at lower operational costs and better throughput.
Authors
Wei Wang
Abstract
Transformer prefill reuses weights across tokens and is dominated by matrix multiplication and activation data movement; single-token decode has limited reuse and is dominated by weight and KV-cache reads. We present FUTRANSFORMER, a CUDA implementation of a Llama-style Transformer block for NVIDIA Hopper GPUs. It applies CODA's GEMM-plus-epilogue approach to inference, adding direct KV-cache placement, folding the fixed RMSNorm gain into weights, writing FP8 output directly with delayed scaling, and supporting prefill without padding. Decode uses INT4 projection weights and an INT4 KV-cache copy. Weights and keys follow Tensor Core fragment layouts for direct dequantization into MMA operands without full intermediate tiles. A block-diagonal Hadamard transform reduces cache quantization error from outlier channels. On one H100, prefill is up to 1.48x faster than compiled PyTorch FP8 and reaches 1.29 PFLOP/s. For gate/up projection with SwiGLU at the 8B shape, our BF16 path is 1.13x faster than CODA. Our FP8 path is 2.02-2.18x faster than the evaluated BF16 CODA path. Across 8B and 70B shapes at batches 1-32, INT4 decode is 1.3-2.4x faster than the faster of PyTorch's BF16 and INT4 blocks at each point: 2.1-3.3x faster than compiled, CUDA-graphed BF16 and 1.3-6.3x faster than the same block using PyTorch's INT4 projection kernel, which targets a single query row and loses to BF16 at batch 32. It is also 31-51% faster than our FP8 path. Complete Llama-3.1-8B decode takes 2.8 ms per step at batch one, versus 8.1 ms for the compiled transformers implementation. Decoding every WikiText-2 token through the INT4 weight path raises perplexity from 6.24 to 7.23 with round-to-nearest weights or 6.99 with GPTQ in the same format; the rotated INT4 cache adds 0.20. The kernels match a quantized reference computation within 0.001 perplexity.