Microscaling cuts hardware costs in large language model acceleration
MIAO: Microscaling Intermediate Accumulation Optimization
Hardware Architecture
Summary
Running big language models usually requires a lot of computing power and memory, especially in parts that add up numbers. The authors found a way to make these adding parts use less space and energy by adjusting the precision at different stages. They created a tool called MIAO that picks how detailed numbers should be at each step, saving hardware and energy while keeping the models almost as accurate. This approach works well on multiple language models and different precision levels.
What this means in practice
- •For hardware engineers: Design more energy-efficient tensor accelerator chips by tailoring accumulation precisions at each stage to reduce area and energy costs.
- •For llm deployment teams: Deploy large language models with lower memory and energy requirements by integrating precision-optimized accumulation hardware configurations.
Authors
Chloe Wong, Nicolas Rakela, Man Shi, Hansung Kim, Ruohan Richard Yan, Coleman Richard Charles Hooper, Yakun Sophia Shao
Abstract
Microscaling (MX) quantization enables lower-precision large language model (LLM) inference with limited accuracy degradation, reducing memory requirement and multiplication cost. However, despite reduced operand precision, accumulation is often retained at high floating-point precisions, leaving a substantial hardware cost that operand quantization alone does not address. In particular, systolic arrays, as used in tensor accelerators such as Google TPU and AWS Trainium, distribute accumulation across distinct processing elements but conventionally use uniform accumulator precision throughout the array. This spatial organization offers an opportunity to reduce hardware cost by tailoring precision to individual accumulation stages. In this work, we present MIAO, a hardware-software co-design framework that jointly selects product precision and stage-dependent accumulator widths to minimize hardware area and energy cost under an end-to-end perplexity constraint. MIAO first calibrates MX block-scale shifts to reduce intermediate magnitudes while preserving operand quantization accuracy, then assigns narrower accumulators to earlier stages and progressively wider accumulators to later stages. To efficiently explore the resulting design space, MIAO combines hardware cost estimates derived from logic synthesis with a calibrated accuracy predictor to identify promising precision configurations, then validates the accuracy of promising candidates through bit-exact end-to-end LLM evaluation. Across MXFP8, MXFP6, and MXFP4, MIAO reduces processing element (PE) array area by up to 38.7% and energy by up to 36.1% relative to the all-BF16 accumulation baseline, with at most 1.60% additional perplexity degradation over operand-only MX quantization across six LLMs.