Text spotter combines two training methods for better reading and locating
SupGRPO: Enhancing GRPO with Matching-based Online SFT for Text Spotting
Computer Vision and Pattern Recognition
Summary
Reading text in pictures requires knowing what the letters are and exactly where they appear. The authors found that two popular computer training methods each help only part of the problem: one is better at recognizing characters, the other better at locating them. They invented a way to combine these two methods that improves both skills at the same time. They tested this on a set of tricky artistic text images and showed their combined method works better. Their code and new dataset will be shared online for others to use.
What this means in practice
- •For mobile app developers: Develop apps that can accurately read and locate complex or artistic text in natural scenes using the combined training method.
- •For augmented reality developers: Create AR experiences that detect and display text precisely even in challenging artistic or decorative fonts using the improved text spotting approach.
Authors
Xudong Xie, Yuzhe Li, Jing Shi, Zhifei Zhang, Curtis Wigington, Zhaowen Wang
Abstract
Text spotting requires both accurate text recognition and precise spatial localization. Current specialised spotters excel at predicting tight bounding boxes in natural scenes, but falter on complex or artistic text, whereas multimodal large language models (MLLMs) possess strong recognition capabilities yet remain weak at localisation. To equip the text spotter with general and powerful recognition capabilities and to maximize its localization ability, we explore two MLLM-based fine-tuning methods: Supervised Fine-Tuning (SFT) and reinforcement learning fine-tuning based on Group Relative Policy Optimisation (GRPO). An interesting finding is that SFT is less effective than GRPO at enhancing recognition, while GRPO is less effective than SFT at enhancing detection. To compensate for each other's shortcomings, we introduce a joint training strategy, SupGRPO, which simultaneously optimizes the model using both SFT and GRPO. SupGRPO employs the specially designed reward functions and develops a matching-based online SFT applied solely to coordinate tokens. It both mitigates the reward sparsity problem of GRPO and avoids the instance order dependency problem of SFT. To evaluate particularly challenging cases, we curate ATS, a dataset for artistic text spotting. Experiments demonstrate that SupGRPO improves both text recognition and detection, and attains superior performance. Our code and dataset will be released at https://github.com/Psycho-9/SupGRPO.