Large language models struggle to verbalize math formulas accurately
How Well Can LLMs Verbalize Mathematical Formulas? A Multi-Domain Evaluation
Computation and LanguageArtificial Intelligence
Summary
It can be hard for computers to read math formulas out loud in a clear, accurate way. This paper tests how well popular language models do at turning math formulas into words across different science subjects. The results show some success, but accuracy depends a lot on the formula's length, complexity, and extra context. Certain math symbols and variable names cause more mistakes. Overall, the models can help but still make errors in verbalizing math.
What this means in practice
- •For text-to-speech developers: Improve math formula verbalization in speech interfaces for visually impaired users by using better language models informed by this evaluation.
- •For assistive technology engineers: Design audio-based learning tools that read scientific content aloud more accurately by incorporating findings about formula types and context effects.
Authors
Peter Volbach, Tim Schlippe
Abstract
Large Language Models (LLMs) have shown strong capabilities across natural language tasks. But their ability to generate accurate written verbalizations of mathematical formulas has not been systematically evaluated. Such verbalization is important for text-to-speech systems supporting audio-based learning and visually impaired users. We evaluated five state-of-the-art LLMs across six scientific domains and constructed Read-Aloud Math, a balanced multi-domain corpus with LaTeX formulas, manually validated simple and extended transcriptions, multiple variants, textual context, domain labels, and formula complexity levels. Using multi-reference BLEU, GPT-5.2 achieved the highest score for simple transcriptions (75.9% without context, 64.9% with context), although without context it did not significantly outperform Grok 4.1 Fast or Claude Sonnet 4.5. Claude Sonnet 4.5 achieved the highest score for extended transcriptions without context (36.5%), and Grok 4.1 Fast with context (48.8%). For extended transcriptions, Statistics and Mathematics scored highest across the two context settings, while Physics scored lowest. Formula length was significantly negatively associated with BLEU for simple transcriptions without context, but not with context, whereas BLEU increased significantly with formula length for extended transcriptions in both settings. Grouping symbols were most frequently involved in transcription differences, while variable names, capitalization, and Greek letters were associated with lower BLEU scores. Overall, LLM-based formula verbalization is promising but remains sensitive to transcription type, textual context, and formula characteristics.