Transformer language models vary in recognizing Bangla medical terms
Language Specificity vs. Domain Diversity: Benchmarking Transformers for Bangla Medical NER
Machine Learning
Summary
Identifying medical terms in Bangla text is hard because there isn’t much labeled data and the language changes a lot in medical settings. The authors tested several AI language models, including Bangla-specific and multilingual ones, to see which understands medical words best. They found that a multilingual transformer called XLM-RoBERTa worked better than the Bangla-only model, suggesting that diverse training data helps more than just focusing on one language. They also showed that fine-tuned models perform better than just asking large language models without extra training.
What this means in practice
- •For hospital data teams: Integrate improved Bangla medical NER to extract structured clinical information from patient records accurately using fine-tuned multilingual transformers.
- •For health app developers: Develop Bangla medical text processing features in mobile health applications that recognize medicines and specialists with high reliability.
Authors
Rakib Abdullah, Md. Maruful Islam Maruf
Abstract
Medical Named Entity Recognition (NER) for low-resource languages remains a challenging task due to high linguistic variability and a scarcity of domain-specific annotated corpora. This work presents a comprehensive empirical benchmark evaluating three fine-tuned transformer encoders-BanglaBERT, multilingual BERT (mBERT), and XLM-RoBERTa-against GPT-4o mini under zero-shot and few-shot prompting configurations for Bangla medical NER. In contrast to prior studies that evaluated large language models on limited subsets of only 50 samples, we conduct a large-scale evaluation across the full test set of 3,179 samples, providing statistically robust and reproducible baselines. Our fine-tuned XLM-RoBERTa model achieves an F1- score of 0.5959, establishing a new state-of-the-art and surpassing the previously reported best result of 0.5848. Crucially, we demonstrate that the language-specific BanglaBERT model consistently underperforms its multilingual counterparts with an F1-score of 0.4937, indicating that pretraining domain diversity can outweigh language specificity in highly specialized clinical settings. Furthermore, we present a detailed per-entity-type analysis for this task, revealing that Medicine and Specialist categories are recognized with high reliability, achieving F1- scores above 0.83, while the Symptom category remains the most challenging with an F1-score of 0.4367 despite being the most frequent training class. Finally, fine-tuned transformer models outperform the optimal prompting configuration by a factor of 3.76, confirming that prompt-only pipelines remain inadequate for structured clinical entity extraction in low-resource language environments.