Vision language models struggle to read braille from images
Learning to Read Braille: Benchmarking and Improving Braille Recognition Capabilities of VLMs
Computer Vision and Pattern RecognitionComputation and Language
Summary
Reading Braille from images is an important challenge for making information accessible, but current vision-language models don't do well at it. The authors collected a set of real Braille images to test different models and found that general models perform poorly, while tools designed specifically for reading text do better. They created a synthetic set of Braille images to train a model, which improved recognition on simpler Braille but still had trouble with more complex types. Their work shows that more focused training can help models understand Braille from photos better, but there is still a gap between training on fake and real images.
What this means in practice
- •For assistive technology developers: Develop better Braille reading features that can automatically transcribe photos of Braille documents using improved vision-language models trained on synthetic data.
- •For mobile app developers: Build mobile apps that capture and convert Braille images into text, improving accessibility for visually impaired users with more accurate recognition models.
Authors
Kartik Jangra, Atharva Gawane, Siddharth Pulipaka, Hemachandran Karah, Raj Dabre
Abstract
Despite rapid progress in visual understanding and document transcription, modern vision-language models (VLMs) struggle with a fundamental accessibility task: reading Braille from images. We introduce BrailleBench, a manually curated benchmark of 259 real-world Grade-1 and Grade-2 Braille images. We evaluate proprietary, open-weight, OCR-specialized, and tool-assisted multimodal systems and find that standalone general-purpose models perform poorly at directly transcribing Braille images, whereas tool-assisted systems perform much better. We design a synthetic Braille generation pipeline with diverse layouts and image perturbations. A 2B VLM model trained on our synthetic data reaches 87% exact match on Grade-1 and 61% on Grade-2 BrailleBench, outperforming standalone VLMs and achieving the highest Grade-2 exact match score among the evaluated systems. We further show that high accuracy on synthetic validation data does not guarantee high accuracy on real photographs in our experiments: models trained on prior synthetic data reach up to 96% on their synthetic validation set but at most 3.26% on Grade-1 BrailleBench. Our results identify an accessibility-relevant capability gap in multimodal systems and show that targeted synthetic supervision can improve real-world Braille transcription.