DeepSeek inference on AMD GPUs improved for speed and accuracy
DeepSeek-V4-Flash on AMD gfx90a: Correctness Recovery and Inference Performance Engineering
Distributed, Parallel, and Cluster ComputingHardware ArchitecturePerformance
Summary
Running complex AI models on AMD graphics cards can be tricky because of how the data and calculations are handled. The authors fixed a critical error in how the AI model's pieces fit together, making sure the results are correct. They then made the system run faster by using special ways to pack data and by improving hardware instructions and memory use. This work shows that both memory speed and how data is processed impact the AI’s performance on AMD GPUs.
What this means in practice
- •For ai inference engineers: Optimize large language model deployments on AMD GPUs for better accuracy and faster response times by correcting layout errors and leveraging hardware features.
- •For gpu performance analysts: Analyze and fine-tune GPU workloads for AI models by using fixed-token checks and topology-aware kernel optimizations specific to CDNA2 architecture.
- •For cloud service providers: Offer competitive AI serving solutions on AMD hardware by using the corrected and optimized DeepSeek inference approach to improve throughput and latency.$Commercial implications: Enables selling inference services with faster, more accurate AI outputs on AMD GPUs than previously feasible.
Authors
Siming Huang
Abstract
We present the enablement, correctness recovery, and performance engineering of DeepSeek-V4-Flash inference on AMD Instinct MI250 GPUs using the gfx90a/CDNA2 architecture. The system integrates native safetensors loading, tensor and expert parallelism, FP4 routed mixture-of-experts computation, FP8 dense projections, sparse attention, HIP graph execution, and OpenAI-compatible serving within SGLang. An initially fast execution path was found to be numerically incorrect because of a routed-expert W2 layout mismatch. We identify the output permutation, repair the weight layout at load time, and establish fixed-token and hash-based correctness checks before further optimization. On the corrected path, decode performance is improved through packed FP4 weights, INT8 activation quantization, CDNA2 dot-product instructions, peer-read all-reduce, and topology-aware kernel geometry. Prefill is accelerated using CDNA2 MFMA kernels, improved packed-weight reuse, reduced sparse-attention overhead, larger chunks, and retuned expert sorting. On four MI250 GCDs, TP4/EP1 native autoregressive decode reaches approximately 74.5 tok/s, while a 4,604-token prompt reaches 2.061-2.062 s TTFT, or approximately 2,234 input tok/s. The results show that efficient DeepSeek-V4-Flash inference on CDNA2 is limited not only by memory bandwidth, but also by FP4 execution-format mismatch, low-M utilization, and per-layer synchronization costs.