Custom GPSIMD operators speed up language model memory on AWS Trainium

Exploiting Native Hardware Heterogeneity: Custom GPSIMD Operators for Accelerating Emerging LLM Memory on AWS Trainium

Distributed, Parallel, and Cluster Computing

Summary

Large language models use special memory parts that do two kinds of work: some tasks are irregular and involve searching, others are more straightforward math. The authors show how to run these tasks faster on AWS Trainium chips by making new small programs for special processors (GPSIMD cores) inside the chip. They built a tool to add these programs into existing software, which made key tasks run up to almost 5 times faster and improved the overall model speed. This work helps use the chip's different hardware working together more efficiently.

What this means in practice

  • For ai infrastructure engineers: Optimize language model memory tasks on AWS Trainium chips to improve inference throughput by customizing GPSIMD operators.
  • For cloud platform developers: Integrate custom operator toolchains into AWS Trainium software stacks to accelerate emerging large model memory computations.

Authors

Zifan He, Jason Cong

Abstract

Emerging memory modules of large language models, such as Engram in DeepSeek-V4.1-Flash, mix irregular integer work (a 64-bit n-gram hash and a sparse table lookup) with dense computation (a GEMM and a gate). When deploying on a GPU-FPGA system, we can disaggregate operations to different devices for Engram to speed up by 5.9$\times$. AWS Trainium provides a similar heterogeneity natively: each Trainium NeuronCore pairs dense engines with a customizable GPSIMD Engine, and all engines share the on-chip SBUF. However, the Neuron Kernel Interface (NKI) exposes the GPSIMD cores only through fixed built-in instructions, and the official GPSIMD custom-operator toolchain does not support Trainium 2 or 3 and cannot be used with NKI. In this work, we build a toolchain that compiles C++ custom operators for the GPSIMD cores and injects them into the compiled NEFF from the NKI kernel on Trainium 3, expanding the customizability of GPSIMD with the current software stack. Over a hand-optimized NKI baseline of DeepSeek-V4.1-Flash, two GPSIMD operators speed up the n-gram hashing by 1.79$\times$ at decode, the table retrieval by up to 4.95$\times$, and the whole Engram computation by up to 1.14$\times$.