SupGRPO improves text spotting by combining recognition and localization training
SupGRPO: Enhancing GRPO with Matching-based Online SFT for Text Spotting
Computer Vision and Pattern Recognition
Summary
Finding text in pictures is tricky because you need to both see where the text is and understand what it says. The authors noticed that two ways of teaching computers to do this each had strengths and weaknesses: one was better at reading text, while the other was better at finding it. They created a new method called SupGRPO that combines the best parts of both approaches to help computers locate and read text more accurately, especially on fancy or artistic writing. They also made a special test set with artistic text to show how well their method works.
text spottingtext recognitionspatial localizationmultimodal large language models (MLLMs)supervised fine-tuning (SFT)reinforcement learningGroup Relative Policy Optimization (GRPO)reward functionsartistic textdataset
Authors
Xudong Xie, Yuzhe Li, Jing Shi, Zhifei Zhang, Curtis Wigington, Zhaowen Wang
Abstract
Text spotting requires both accurate text recognition and precise spatial localization. Current specialised spotters excel at predicting tight bounding boxes in natural scenes, but falter on complex or artistic text, whereas multimodal large language models (MLLMs) possess strong recognition capabilities yet remain weak at localisation. To equip the text spotter with general and powerful recognition capabilities and to maximize its localization ability, we explore two MLLM-based fine-tuning methods: Supervised Fine-Tuning (SFT) and reinforcement learning fine-tuning based on Group Relative Policy Optimisation (GRPO). An interesting finding is that SFT is less effective than GRPO at enhancing recognition, while GRPO is less effective than SFT at enhancing detection. To compensate for each other's shortcomings, we introduce a joint training strategy, SupGRPO, which simultaneously optimizes the model using both SFT and GRPO. SupGRPO employs the specially designed reward functions and develops a matching-based online SFT applied solely to coordinate tokens. It both mitigates the reward sparsity problem of GRPO and avoids the instance order dependency problem of SFT. To evaluate particularly challenging cases, we curate ATS, a dataset for artistic text spotting. Experiments demonstrate that SupGRPO improves both text recognition and detection, and attains superior performance. Our code and dataset will be released at https://github.com/Psycho-9/SupGRPO.