FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution
2026-08-17 • Distributed, Parallel, and Cluster Computing
Distributed, Parallel, and Cluster Computing
AI summaryⓘ
The authors present FreeToken, a system designed to run very large AI models directly on personal computers rather than relying on big data centers. It adapts to different hardware setups by smartly distributing the work and memory based on what each machine has available. FreeToken can handle continuously changing tasks and supports various large models, enabling devices like laptops and workstations to run models much bigger than before. This makes advanced AI more practical for everyday users on their own machines.
Mixture of Experts (MoE)model servingedge computinginferenceGPUCPUmemory managementopen-weight modelsdeep learningruntime optimization
Authors
Shuo Yang, Xiaoze Fan, Melissa Pan, Haocheng Xi, Zhe Wang, Shanlin Sun, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu, Ion Stoica
Abstract
Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack, including model layout and loading, expert residency, CPU--GPU execution, agentic state reuse, and runtime memory management, around two realities of local AI: agent workloads continuously change their execution pattern, and edge hardware exposes heterogeneous resources whose balance differs from machine to machine. Rather than committing to a fixed offloading strategy, FreeToken continuously maps computation and model state onto the resources actually available. FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. We release the system at flashml.ai.