PagedWeight: Efficient MoE LLM Serving with Dynamic Quality-Aware Weight Quantization
2026-07-17 • Machine Learning
Machine Learning
AI summaryⓘ
The authors address a problem in large language models called Mixture-of-Experts (MoE), where the balance between memory for model weights and memory for cached data (KV cache) is tricky. They introduce PagedWeight, a method that adjusts the precision of the model's expert weights dynamically during use to better manage memory and speed. This approach helps keep accuracy high while saving a lot of GPU memory and making the model run faster. Their method works better than previous techniques, offering a good balance between accuracy, memory use, and processing speed.
Mixture-of-Experts (MoE)Large Language Models (LLMs)KV-cacheGPU memoryQuantizationFP16 precisionThroughputLatencyModel weightsMemory management
Authors
Yuchen Yang, Yifan Zhao, Anisha Dasgupta, Sasa Misailovic
Abstract
Mixture-of-Experts (MoE) is a popular class of large language models (LLMs), offering high efficiency and accuracy. However, in KV-cache-intensive serving scenarios, MoEs often exhibit a tension between the GPU memory requirements of the model weights and the growing KV cache. We propose PagedWeight, a novel management method for MoE LLM serving that dynamically quantizes MoE model's weights at runtime and balances expert-weight precision with the KV cache sizes. PagedWeight exposes and effectively navigates the complex tradeoff between the model's task accuracy, memory consumption, and throughput/latency. Across several memory-sensitive MoE serving scenarios, PagedWeight improves the quality-memory tradeoff over several existing quantization baselines. PagedWeight achieves FP16-equivalent accuracy with up to 72.0% GPU memory savings and 1.94$\times$ throughput improvement, and improves quality over quantization methods by up to 39.3% at a similar memory budget with at most 4.1% throughput loss.