Rowhammer bit-flips slow down GPU large language model inference persistently
Amplifying Latency and Cost of LLM Inference via Rowhammer Attack
Cryptography and SecurityMachine Learning
Summary
Modern large language models (LLMs) run efficiently on GPUs by using special code generated on the fly. The authors found a way to cause tiny memory errors called bit-flips in the CPU's memory that make this code less efficient without breaking the model’s output. These subtle changes slow down the model’s response time every time it runs, and the slowdown lasts because the inefficient code is saved and reused. They created a tool to find the exact bits to flip and showed the attack works on various GPUs and models. This reveals a hidden security risk that targeted hardware errors can raise the cost and latency of AI model services.
What this means in practice
- •For cloud service operators: Detect and mitigate Rowhammer-induced slowdowns that increase operating costs in GPU-powered AI model deployments.
- •For hardware security teams: Develop defenses against bit-flip attacks targeting compilation pipelines that degrade AI inference performance consistently.
Authors
Mrityunjay Shukla, Shubham Roy, Biswabandan Panda
Abstract
Large language models (LLMs) achieve high inference efficiency using optimized GPU kernels generated by just-in-time (JIT) compilation frameworks such as Triton. Triton compiles GPU kernels through an MLIR/LLVM-based pipeline. We investigate whether a Rowhammer-induced bit-flip in the CPU's DRAM can degrade GPU-based LLM inference performance while preserving output correctness. We present LLVMhammer, the first Rowhammer attack that causes LLM inference slowdown by inducing a bit-flip in Triton's compilation pipeline. A bit-flip in a performance-critical optimization pass causes Triton to generate a functionally identical but suboptimal GPU kernel. Since Triton caches the generated kernel on disk, a single successful bit-flip causes persistent slowdown across subsequent inference executions. Triton's target-independent stages are shared across GPU vendors, which makes our attack GPU vendor-agnostic. To identify exploitable bit-flip locations without manual inspection, we build Karnage, a tool that isolates useful bit-flip locations in Triton's target-independent compilation pipeline. We evaluate LLVMHammer across a diverse set of LLMs, including Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3, and Deepseek-llm-7B-chat, across NVIDIA RTX 4090 and AMD RX 7900 XTX GPUs. LLVMhammer increases inference latency of an LLM inference server running on a single GPU by 4-16\%, which, in turn, can translate into additional annual operating costs at the scale of multi-GPU deployments. We further show that LLVMhammer can also corrupt LLM outputs across different GPU vendors, reducing BERTScore to as low as 25\%, illustrating the broader security implications of bit-flips in Triton's compilation pipeline.