StrataFlux improves low-bit LLM inference accuracy and speed

StrataFlux: Memory-Hierarchy-Native Low-Bit LLM Inference

Machine LearningDistributed, Parallel, and Cluster Computing

Summary

Running large language models on computers usually needs a lot of memory and fast hardware, but cutting down the details (called quantization) can make the models less accurate. The authors propose StrataFlux, a way to use extra precision from computer memory and storage without slowing things down. Their method smartly fetches better data precision only when needed, keeping the model fast and memory-efficient while improving the results. They tested it on various GPUs and found it can get better quality and speed than prior approaches using very low bit levels.

What this means in practice

  • For ai model deployers: Run large language models more accurately on limited GPU memory by using selective precision streaming without increasing latency.
  • For cloud infrastructure teams: Reduce costly GPU memory requirements for language model inference by offloading refinement data dynamically to host RAM or SSD storage.

Authors

Jake Hyun, Ziyu Zhao, Jae-sun Seo, Mohamed S. Abdelfattah

Abstract

Quantization reduces the memory and bandwidth demands of large language model (LLM) inference but can degrade accuracy; offloading higher-precision data can instead stall generation. We introduce StrataFlux, a framework that challenges this tradeoff by recovering accuracy lost to low-bit weight and key-value (KV) cache quantization while retaining quantization's memory and speed advantages. StrataFlux co-designs refinement representation, selection, and delivery to make host RAM and solid-state drives useful sources of additional precision. Its selective precision streaming reduces transfer volume, overlaps delivery with computation, and reuses fetched precision across decoding steps so generation need not wait for every transfer. We evaluate 2-4-bit weights and 2-3-bit KV-cache on dedicated GPUs and unified-memory platforms. For 2-bit Llama-3.1-8B-Instruct weights, StrataFlux reduces WikiText-2 perplexity from 9.46 to 8.40 while maintaining 95.4% of the quantized base's throughput on RTX 5090. With 2-bit weights and KV, StrataFlux raises Qwen3-8B's AIME 2024/2025 accuracy from 11.0% to 47.1% at unchanged GPU memory use. At 128K context, Llama-3.1-8B-Instruct with 3-bit weights and 2-bit KV achieves comparable perplexity and speed to vLLM's 4-bit AWQ weights and 8-bit KV on RTX 5090, using approximately half the GPU memory. These results show that GPU-resident precision need not set the accuracy ceiling, nor offloading-tier latency dictate decoding speed.