Handwritten text recognition improves with new unified learning model
ExpertHTR: Unified Handwritten Text Recognition with Multi-Task Learning and Sparse Mixture-of-Experts
Computer Vision and Pattern RecognitionMachine Learning
Summary
Handwritten text recognition is hard because resources are small and vary a lot in language, script, and format. The authors created ExpertHTR, a system that learns from different types of handwriting data together by organizing annotations into a common structure and using multiple related tasks. They also introduced a way to efficiently activate parts of the model depending on the input. Testing on seven handwriting datasets showed ExpertHTR works better than many existing systems and achieves top results on a key benchmark.
What this means in practice
- •For document digitization teams: Develop systems that transcribe handwritten documents from diverse sources and formats more accurately using multi-task training with shared model components.
- •For archival service providers: Automate transcription of historical handwritten pages by leveraging shared representations and mixtures of experts to handle varied scripts and annotations.
Authors
Dang Hoai Nam, Nguyen Duy Hieu, Quang Huu Hieu, Vo Nguyen Le Duy
Abstract
Handwritten text recognition resources are often small and distributed across collections that differ in language, script, document structure, and annotation format, making joint page-level training difficult. We propose ExpertHTR, a unified vision-language framework that addresses this problem through complementary supervision and conditional model capacity. Structural annotations from heterogeneous datasets are first organized through a common Page-Region-Line representation and used to construct four related training tasks for complete transcription, physical-line coverage, text localization, and localized recognition, without requiring additional manual labels. Building on a jointly trained dense model, ExpertHTR introduces a sparse Mixture-of-Experts architecture with an always-active shared branch and conditionally routed full-MLP experts. Sparsegen allows the number of active routed experts to vary with the hidden representation, while routing regularization reduces persistent concentration on a small subset of experts. Experiments on seven heterogeneous handwriting benchmarks show that complementary supervision consistently improves training with page transcription alone, while joint multi-source training provides further gains on most datasets. The proposed sparse expert model further improves the dense baseline on six of the seven sources. The final unified model also substantially outperforms the evaluated general-purpose OCR and vision-language systems on most benchmarks and achieves state-of-the-art performance on the IAM paragraph-level benchmark, while specialized HTR systems remain stronger on several challenging collections.