Summary
It is hard for computer models that understand both images and language to solve math problems in Bengali, especially those involving geometric diagrams. The authors created a large set of Bengali geometry problems with both pictures and detailed text descriptions to test these models. They found that models do about the same when given only text descriptions as when given both images and text, showing the descriptions can stand in for diagrams in tests. However, models often fail when asked to confirm that the text matches the image, mixing up spatial relationships. Training the models on this dataset helps, but they still lag behind the best models used without extra training. Overall, this work gives a new way to study how models handle math problems combining images and language in Bengali.
What this means in practice
- •For software developers: Improve tools that solve Bengali geometry problems by testing models’ ability to understand both diagrams and text together.
- •For education technology teams: Build educational apps that adapt Bengali math tutoring based on model evaluation using comprehensive geometry problem sets combining images and descriptions.
Authors
Khan Raiyan Ibne Reza, Sanjana Aktar Maria, Sumaiya Tabassum Nimi, Md Adnan Arefeen
Abstract
Evaluation of vision-language models (VLMs) for multimodal mathematical reasoning remains limited for low-resource languages and for geometry problems that require reading a diagram and a question together. We introduce ChitraMiti-12.8k, a synthetic benchmark of 12,874 Bengali planar geometry problems paired with structured 15-attribute descriptions, and NCTB-500, a complementary set of 500 diagrams manually extracted from Bengali school textbooks. Using a three-phase protocol that separates diagram-only, diagram-plus-description, and description-only inputs, we show across five open-weight and closed-source VLMs that description-only performance is statistically indistinguishable from diagram-plus-description performance, establishing structured descriptions as a sufficient textual proxy for controlled evaluation. Despite this, models remain poor at cross-modal verification, frequently misled by a swapped spatial relation even when they answer the unmodified item correctly. We further evaluate supervised adaptation on ChitraMiti-12.8k, finding that fine-tuning improves performance on both ChitraMiti-1k and NCTB-500, although a substantial gap to the strongest zero-shot model remains. Together, ChitraMiti-12.8k, NCTB-500, and our evaluation protocol offer a standardized way to study Bengali multimodal geometry reasoning and, more broadly, whether VLMs actually check their text against what they see. Our dataset and code are publicly available on Hugging Face at https://huggingface.co/datasets/RaiyanKhaan/ChitraMiti.