Deep learning model over 700 billion parameters runs on 24 GB laptop
DeepInfer-Mac: Running a 750B-Class MoE on a 24 GB Laptop
Performance
Summary
Running very large AI language models usually needs huge computers with lots of memory. This paper shows how a model with about 750 billion parameters can run on a laptop with only 24 GB of memory by storing most data on the hard drive and loading parts as needed. The authors found ways to organize and load the data efficiently, though it still takes a moment to generate each word. Their work explores how to balance storage limits and speed for large models on small devices.
What this means in practice
- •For mobile device developers: Run extremely large language models locally on laptops without needing large memory systems by managing data storage and loading efficiently.
- •For ai inference engineers: Optimize memory and storage strategies to enable large-scale model inference on devices with limited RAM by reorganizing expert weights and caching.
Authors
Haorui Li
Abstract
DeepInfer-Mac runs the 750B-class DeepSeek-V4.1-Flash model on a MacBook Air with 24 GB of unified memory. A mixed low-bit conversion occupies 338 GiB on the internal SSD. About 7.6 GiB of weights remain resident; routed experts and Engram rows are loaded when needed. This separation makes the model executable, but it leaves each generated token waiting for a succession of storage reads. We organize expert weights so that reads fill reusable GPU buffers directly, and couple cache allocation with the lifetime of those buffers. In end-to-end runs, the system generates English, Chinese and code at 0.21-0.72 tokens per second, with a largest sampled process footprint of 16.73 GiB across the tested requests. Matched decode comparisons show a 21.0% throughput gain from next-layer prefetching over resident-expert overlap alone, at the cost of additional reads. Longer prompts remain costly. We describe the design, provide the implementation and measurement scripts, and identify the limits of this operating point: a model far larger than memory can run locally, while storage and prefill still govern the user's wait.