Workload-Driven Optimization for On-Device Real-Time Subtitle Translation
2026-07-10 • Computation and Language
Computation and LanguageArtificial Intelligence
AI summaryⓘ
The authors studied how to translate English subtitles to Traditional Chinese directly on devices in Taiwan, focusing on short inputs, fast response, and user privacy. They improved an existing model by reducing the vocabulary size, which helped make the translation faster without losing much accuracy. Their adapted model, called LocalSubs, competed well against Google Translate, especially with shorter subtitle lines. Early tests on Apple M2 hardware showed the smaller vocabulary sped up processing, but the speed results are still preliminary.
on-device translationEnglish-to-Traditional-Chinesesubtitle translationvocabulary sizetokenizerembedding calibrationfine-tuninginference latencyGGUF quantizationbenchmarking
Authors
Tsz-To Wong
Abstract
This report studies on-device English-to-Traditional-Chinese subtitle translation for Taiwan under short inputs, short outputs, batch-size-one inference, low latency, and privacy constraints. These conditions limit the value of optimizations designed for long-context or high-throughput language-model serving. Starting from LMT-60-0.6B, preliminary profiling suggests that vocabulary projection becomes a more important decode-time cost after GGUF quantization reduces the relative cost of Transformer blocks. We replace the original 151k-token vocabulary with a 64k-token subtitle-domain tokenizer, migrate the embedding space, and adapt the model through embedding calibration followed by full supervised fine-tuning. On a fixed 500-example subset of the OpenSubtitles2024 test set, the LocalSubs achieves a 59.2% tie-excluded win rate against Google Translate under GPT-4o pairwise judging. Performance is strongest on short cues and declines as cue length increases. Preliminary Apple M2 Metal measurements on a 64k-vocabulary model show a 1.63$\times$ speedup over a 151k-vocabulary profiling baseline. The raw benchmark configuration is incomplete, so the latency result is treated as preliminary.