HyCo reduces server time for mobile-cloud language decoding
HyCo: Hybrid Drafting for Cost-Efficient Mobile-Cloud Speculative Decoding
Computation and LanguageDistributed, Parallel, and Cluster ComputingMachine Learning
Summary
Processing language on mobile devices while using powerful cloud models can be slow or expensive due to repeated computations. The authors introduce HyCo, a method that smartly splits work between a small on-device model and a larger cloud model to speed up text prediction. This reduces the time the server spends generating each word without losing quality. Tests showed meaningful improvements in speed and efficiency on real mobile phones connected via 5G networks.
What this means in practice
- •For mobile app developers: Reduce server load and speed up text generation for apps using language models across mobile-cloud setups.
- •For cloud service engineers: Optimize GPU usage in cloud servers by efficiently offloading parts of language model computations to clients' mobile devices.
Authors
Hanbin Bae, Heejin Choi, Joohyung Lee, Byoung Jin Choi, Sangjun Park, Hoon-Young Cho
Abstract
Device-cloud speculative decoding reduces server computation, but growing a deep draft tree with a small language model (SLM) requires repeated passes on a mobile neural processing unit (NPU). We present HyCo, which assigns depth to the server model's native multi-token prediction (MTP) head, executed on the device, and alternative branches to one SLM pass. The server returns the hidden state, input embedding, and key-value entries needed by the head instead of executing its sequential steps. Quantization, a restricted vocabulary, and a 1K-token window fit both drafters to the NPU. An ε-threshold rule additionally relaxes exact preservation of the verifier's sampling distribution. In single-client runs with a Galaxy S26 Ultra over LTE/5G, HyCo reduces server GPU time per output token relative to server-side MTP self-speculation by 38.6% on Spec-Bench and 29.2% on 32K-token summarization. With the same acceptance rule, three Spec-Bench runs per system show 9.0 percentage points more saving and 1.9x delivery speed than an SLM-only drafter. HyCo differs from Cloud+MTP by +0.2 judge-score points (paired 95% CI: -0.9 to +1.3); exploratory translation favors SpecEdge-ε. Without relaxation, HyCo exact saves 27.7% and 11.9% on the two datasets.