Flash-backed mixture-of-experts inference characterized for iPhone performance
Paging the Experts: A Reproducible Characterization of Flash-Backed MoE Inference on iPhone
PerformanceMachine Learning
Summary
Running large AI models on phones is tricky because different parts of the model (called experts) take up a lot of memory. The authors made a system called Routide that smartly stores these experts on the phone and loads only some parts into memory when needed. They tested how different memory strategies affect performance and accuracy, finding that the right memory management depends on the workload and cache policy. They also observed differences between running the model on a Mac versus a phone, showing exact matches aren’t always possible. Overall, they identify limits and factors important for deploying these models on mobile devices.
What this means in practice
- •For mobile app developers: Run large-scale mixture-of-experts AI models efficiently on iPhones by managing expert data storage and memory budgets.
- •For embedded system engineers: Design memory and cache policies for edge devices running sparse AI workloads with large model checkpoints stored externally.
Authors
Musa Shams
Abstract
Sparse activation reduces mixture-of-experts computation without eliminating the need to store all experts. We present Routide, a Swift/MLX runtime that executes the text path of a pinned public Qwen3.6-35B-A3B quantized checkpoint while keeping expert weights in iPhone storage and a byte-budgeted subset in memory. We characterize cache-policy sensitivity, numerical comparison boundaries, and measurement limits. Across five recorded 128-token workloads, fixed-route replay gives 0.00% demand hits with a 512 MiB LRU cache, 18.80% with seeded random eviction at the same budget, and 38.58% with 576 MiB LRU. The apparent capacity cliff is therefore a policy/workload interaction, not a universal memory requirement. Same-runtime Mac controls preserve generated sequences across eviction and asynchronous prefetch, including 2,560 exact token comparisons and 10,334 speculative loads. In contrast, complete resident-Python versus recorded-phone sequences disagree on all five tested cases, precluding a general numerical equivalence claim. Two separately scoped iOS 27 memory protocols observe sampled process-footprint peaks of 1.87-2.32 GiB on short prompts and 2.39-2.73 GiB on one longer prompt. We retain a thermal stopping event, negative timing comparisons, and a single qualified whole-device power estimate. These results establish bounded feasibility and identify limitations that a deployment claim must not hide.