Justfit runs large language models on 24 giB laptops efficiently
JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management
Artificial IntelligencePerformance
Summary
Handling large language models on laptops is hard because they use a lot of memory to keep track of the context and state. The authors introduce JustFit, a system that carefully manages the parts of the model and its memory use while running. This allows a 24 GiB laptop to handle nearly 7 times more input text than previous setups, while still solving complex reasoning problems. JustFit achieves this by loading and unloading model data just in time, without depending on specific model compression techniques.
What this means in practice
- •For software developers: Run large AI models locally on laptops with limited memory to enable offline coding and reasoning tasks.
- •For mobile device engineers: Optimize deployment of AI models on resource-constrained devices by managing execution state and memory dynamically.
Authors
Yuhua Chen
Abstract
Capable open-weight models make local coding and reasoning attractive, but their context and execution state strain laptop memory. We present JustFit, an MLX-based inference runtime that combines KVExec for compressed KV execution, PhaseSwap for component residency, and StateTrans for state-preserving serving transitions. These mechanisms fuse reconstruction and coordinate just-in-time materialization and release, independently of model-weight quantization. In full-execution capacity tests on a 24 GiB M4 Pro MacBook running Qwen3.8-27B MXFP4, three independent runs complete 196,608 input and 16,384 output tokens, increasing completed single-request context from the mlx-vlm baseline's 30,720 positions to 212,992 (6.93x); a separate two-request run retains 229,376 positions in aggregate. In separate performance tests, a 32K-input, 64-output probe reaches 19.11 tokens/s, and a repeated 32K+6K workload has a median peak process footprint of 16,374 MiB. The integrated runtime answers 29 of 30 AIME 2026 problems correctly, showing how compact state and lifetime-aware execution expand local serving capacity while supporting extended generated reasoning.