Phone training boosts specialist language models with minimal memory
Training Specialist LLMs on Your Phone: Forward-Only Optimization via Block-Spectral Coordinates
Machine Learning
Summary
Big language AI models usually need lots of memory and power, so they’re hard to train on phones. This paper shows a way to specialize these models using only forward passes, avoiding the heavy memory use of backpropagation. The method adjusts small parts of the model based on how well it completes tasks, making it smart for specific uses on your device. The authors demonstrate that this can improve success on phone tasks dramatically and even train a sizable model overnight.
What this means in practice
- •For mobile app developers: Train specialized language models on smartphones to personalize AI features without heavy memory needs or labeled data.
- •For embedded system engineers: Adapt AI models in devices with limited computing power by updating only small weight components through forward passes.
Authors
Liyan Tan, Yequan Zhao, Ziyue Liu, Yupeng Su, Ruijie Zhang, Zheng Zhang
Abstract
Large language models (LLMs) on personal devices keep user data local, but limited memory restricts their general capability. Since most user-specific tasks do not require the breadth of frontier models, a practical solution is to adapt a shared on-device model into lightweight specialists. However, backpropagation is prohibitive in memory, and users rarely provide labeled data. We present MobileSpec, a system that trains specialists with forward passes alone and learns from automatically checked task outcomes, as in reinforcement learning with verifiable rewards (RLVR). Because forward-only updates are built from random perturbations, training is efficient only when these perturbations resemble the weight changes of RLVR, which largely preserve the singular values of pretrained weights and rotate their singular vectors. The core algorithm of MobileSpec, FuRA-S, produces such perturbations while training only the block-spectral coordinates of pretrained weights, namely the singular values of their column blocks. As we show theoretically and empirically, rescaling these blocks independently shifts their relative contributions to each weight matrix and thereby rotates its singular vectors. Since this update can be rewritten as a transformation of the layer input, FuRA-S also adapts INT4 models without modifying their weights. With specialists of only 1-5 MB that update 0.015-0.05% of the parameters, MobileSpec raises the success rate of a phone-use agent from 6-23% up to 89-100% and trains a personalized 8B model on an iPhone 16 Pro overnight. On RLVR benchmarks, FuRA-S is competitive with full-parameter evolution strategies (ES) and outperforms ES over LoRA adapters and quantized ES on INT4 models.