PagedMoE improves large model speed by smart caching on SSD
PagedMoE: Cross-Precision Expert Caching for Token-Level Dynamic Precision in SSD-Resident MoE Inference
Hardware Architecture
Summary
Running huge AI models on regular computers is slow because they need to load big pieces of information from slow storage often. The authors found a way to store and reuse parts of these pieces at different detail levels so the computer fetches less data. This method, called PagedMoE, picks how detailed each piece should be and reuses cached data cleverly, making the whole process faster without losing much quality. It works well on very large language models and helps them run faster using less memory.
What this means in practice
- •For machine learning engineers: Increase throughput of large sparse language model inference on limited DRAM by smarter expert caching with dynamic precision.
- •For data center operators: Reduce SSD I/O bottlenecks during large model inference by deploying PagedMoE to optimize expert data reads.
- •For video game developers: Improve on-device AI assistant responsiveness by cutting model data fetches and computations using dynamic precision caching.$Commercial implications: Enables better performance and responsiveness for AI-powered game features by efficient use of limited memory and storage.
Authors
Yutong Lin, Yingdi Shan, Shaoyuan Chen, Yuanyong Chen, Yongwei Wu, Mingxing Zhang
Abstract
Large Mixture-of-Experts (MoE) models are promising for local inference because sparse routing allows the routed-expert pool to reside on SSD while DRAM caches recently used experts. However, under practical DRAM cache capacities, substantial SSD I/O remains on the critical path despite I/O-computation overlap. Token-level dynamic expert precision offers a natural way to reduce this data demand by allocating bits according to expert importance and quantization sensitivity. Nevertheless, treating each bit-width of the same expert as an independent object prevents cross-precision cache reuse, causing overlapping data to be fetched repeatedly. The key insight is that precision should instead denote the refinement depth of one shared expert state. Based on this insight, we present PagedMoE, which stores each expert once using a prefix-compatible tree codebook and bitplane layout. Lower-precision representations are valid prefixes of higher-precision ones, allowing cached data to be reused or extended by fetching only missing bitplanes. PagedMoE selects the required prefix length before I/O using current routing weights and offline expert-specific error profiles, and directly executes cached 1-8-bit prefixes without materializing full-precision experts. At an average routed-expert precision of 2.5 bits, PagedMoE achieves quality close to GPTQ-INT4 on Qwen3.5-397B-A17B and to MXFP4 on most evaluated GLM5.2 tasks. With a 24 GiB expert cache, it reaches 8.89 tokens/s on Qwen3.5, 1.33x the throughput of the same dynamic requests using independent precision objects and 1.56x that of uniform 4-bit execution.