Large language models show varied success on English and Bengali summaries
Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali
Computation and LanguageArtificial Intelligence
Summary
Generating short summaries from texts is hard, especially across different languages. The authors compared several large language models to see how well they summarize in English and Bengali. They found that models doing well in English often do not perform as strongly in Bengali, a less resourced language. Also, using just one way to judge summary quality can give misleading results, so a broad evaluation is needed.
What this means in practice
- •For multilingual ai developers: Improve summary systems by selecting or tuning models specifically for under-resourced languages like Bengali rather than relying only on English results.
- •For content moderation teams: Use better fact-checking metrics to verify summaries generated by AI in multiple languages, ensuring more reliable information dissemination.
Authors
Khan Mahi Al Atahar, Nazifa Tasneem, Ajwad Abrar
Abstract
Abstractive summarization remains challenging across languages, where linguistic quality and factual consistency may diverge, particularly in low-resource settings. We present a standardized zero-shot benchmark of eight closed-source and open-weight LLMs on size-matched English and Bengali XL-Sum test sets. Our evaluation combines lexical and semantic metrics with reference-free factuality verification and multidimensional LLM-as-a-Judge assessment, providing a comprehensive analysis of both surface-level quality and factual reliability in multilingual summarization. Results reveal substantial cross-lingual variation: GPT-5.6 Luna leads several English lexical metrics but does not maintain the same advantage in Bengali, where LLaMA 3.3 70B achieves the highest ROUGE and BLEU scores. Gemini 2.5 Flash obtains the highest observed Bengali factuality score. Furthermore, evaluation methods produce different model rankings, highlighting the limitations of relying on a single metric for multilingual summarization assessment. These findings demonstrate that strong English summarization performance does not necessarily transfer to Bengali and emphasize the need for comprehensive evaluation frameworks for low-resource languages.