Papers for

education technology teams

Papers whose findings have a practical use for this group, as judged from the abstract. Open a paper to read what it means in practice.

Bengali vision language models struggle with geometry diagrams and text

ChitraMiti: Benchmarking Visual Grounding and Modality Reliance in Bengali Geometric Reasoning

Abstract: Evaluation of vision-language models (VLMs) for multimodal mathematical reasoning remains limited for low-resource languages and for geometry problems that require reading a diagram and a question together. We introduce ChitraMiti-12.8k, a synthetic benchmark of 12,874 Bengali planar geometry problems paired with structured 15-attribute descriptions, and NCTB-500, a complementary set of 500 diagrams manually extracted from Bengali school textbooks. Using a three-phase protocol that separates diagram-only, diagram-plus-description, and description-only inputs, we show across five open-weight and closed-source VLMs that description-only performance is statistically indistinguishable from diagram-plus-description performance, establishing structured descriptions as a sufficient textual proxy for controlled evaluation. Despite this, models remain poor at cross-modal verification, frequently misled by a swapped spatial relation even when they answer the unmodified item correctly. We further evaluate supervised adaptation on ChitraMiti-12.8k, finding that fine-tuning improves performance on both ChitraMiti-1k and NCTB-500, although a substantial gap to the strongest zero-shot model remains. Together, ChitraMiti-12.8k, NCTB-500, and our evaluation protocol offer a standardized way to study Bengali multimodal geometry reasoning and, more broadly, whether VLMs actually check their text against what they see. Our dataset and code are publicly available on Hugging Face at https://huggingface.co/datasets/RaiyanKhaan/ChitraMiti.

Fri 11 SeptComputer Vision and Pattern Recognition
The gist
It is hard for computer models that understand both images and language to solve math problems in Bengali, especially those involving geometric diagrams. The authors created a large set of Bengali geometry problems with both pictures and detailed text descriptions to test these models. They found that models do about the same when given only text descriptions as when given both images and text, showing the descriptions can stand in for diagrams in tests. However, models often fail when asked to confirm that the text matches the image, mixing up spatial relationships. Training the models on this dataset helps, but they still lag behind the best models used without extra training. Overall, this work gives a new way to study how models handle math problems combining images and language in Bengali.
Open → 2609.12509v1