Multilingual text recognition improves with script-aware model and dataset
All-in-One Multilingual Scene Text Recognition with Script-aware Mixture-of-Experts
Computer Vision and Pattern Recognition
Summary
Recognizing text from pictures in many languages is hard because there isn't enough data for most languages and existing models are either too big or not accurate enough. The authors created a huge synthetic dataset covering 10 scripts and 229 languages to help train better models. They also built a new model that knows about different writing scripts and uses specialized parts for each one, sharing knowledge where helpful. Their approach works better and is smaller than previous methods, boosting accuracy on multiple tests.
What this means in practice
- •For mobile app developers: Improve multilingual text detection in camera apps by integrating a more accurate and lightweight script-aware text recognizer.
- •For document digitization teams: Enhance recognition accuracy for multilingual scanned documents by deploying a text recognizer that handles various scripts within a single model.
Authors
Xingsong Ye, Yongkun Du, Jiaxin Zhang, Zhixian Li, Chong Sun, Chen Li, Jing Lyu, Lianwen Jin, Zhineng Chen
Abstract
Multilingual scene text recognition (STR) remains challenging due to the scarcity of training data for most languages and the difficulty of serving diverse scripts within a single model. Existing solutions either deploy one recognizer per language, inflating cost and introducing error accumulation, or rely on massive vision-language models (VLMs) that are expensive and still inaccurate on many scripts. In this work, we pursue an all-in-one multilingual recognizer that is simpler than per-language experts, lighter than VLMs, and more accurate than both. First, we construct TextMuSS-10M, a large-scale synthetic scene text dataset spanning 10 scripts and 229 languages. It provides balanced and sufficient supervision where real data is unavailable. Second, we propose ScriptMoE, a script-aware Mixture-of-Experts (MoE) architecture. It shares a single visual encoder and replaces the dense decoder with a sparse MoE block, which consists of an image-level router dispatches each image to the top-2 script-aligned experts and a shared expert absorbs cross-script knowledge. Extensive experiments on our assembled TextMuSS-Bench (10 scripts, 10,899 images) show that ScriptMoE achieves the highest accuracy of 82.06%, outperforming the strongest STR baseline by 1.31%. On the CC-OCR end-to-end multilingual task, replacing only the recognizer in PP-OCRv5 with ScriptMoE lifts F1 score from 65.71% to 80.89%, slightly surpassing the best VLM (80.73%) at a fraction of the parameter count.