Evaluating OCR accuracy for Arabic English and French documents

Evaluating Cloud and Self-Hosted OCR Systems for Arabic, English, and French Documents: Bounding Box Detection, Text Recognition, and Mathematical Expression Recognition

Computer Vision and Pattern Recognition

Summary

Choosing the right OCR (Optical Character Recognition) system to read documents in multiple languages like Arabic, English, and French can be tricky because you need to balance how accurate it is with how you can use it. The authors compared cloud-based and self-hosted OCR tools on three tasks: detecting where text is on a page, reading the text, and recognizing math expressions. They found that some systems were almost equally good at reading text but none clearly excelled at reading math. They also discovered that perfectly good scores sometimes hid mistakes in Arabic specifics, so careful checking beyond just overall accuracy is important.

What this means in practice

  • For document management teams: Select OCR systems for archives containing Arabic, English, and French documents allowing better trade-offs between accuracy and deployment needs.
  • For software integrators: Implement reliable OCR pipelines by including uncertainty estimates and notation-specific checks to avoid hidden recognition errors in multilingual text and math.

Authors

Jumana Alsubhi, Ahmed Alhusayni, Abdulrahman Gharawi, Israa Hamdine, Khadija Alansari, Alshaymaa Allahim, Lamees Alhumaid, Ahmad Shabana

Abstract

Choosing an OCR system for a multilingual archive requires balancing accuracy against deployment constraints. This study compares cloud and self-hosted systems on three tasks: bounding box detection on 48 Arabic, English, and French pages, text recognition on 315 pages across nine document types, and mathematical expression recognition on 83 English and Arabic samples. Detection was assessed by page-level visual grades, text by character error rate (CER) against a Gemini-generated reference transcription, and mathematics by scores against known answers. Main estimates are accompanied by 95% intervals and paired comparisons. Google Document AI and Surya produced Good line boxes on 47 of 48 (98%) and 46 of 48 (96%) pages. Datalab, Azure Document Intelligence, and Document AI had mean CERs of 0.145-0.152, with no difference detected in unadjusted paired comparisons. Surya had the lowest mean CER among self-hosted systems (0.199). No mathematical test had a statistically clear leader. Strict checks showed that near-perfect page scores could conceal a replaced Arabic variable or Eastern Arabic digits rewritten in Western form. Reliable evaluation therefore requires uncertainty estimates, structural checks, and notation-specific tests in addition to aggregate recognition scores.
View PDFOpen arXiv