Domain-specific pretraining improves transformers on arabic english social media
Domain-specific Pretraining Profile and Transformer Performance: Evidence from Modeling Digital Pragmatics in Arabic-English Code-switching
Computation and Language
Summary
This study shows that teaching AI models with data specific to Arabic and English social media conversations helps them better understand the meaning behind code-switched posts, where people mix both languages. The authors compared different AI models and found that one called MARBERT, trained mainly on Arabic social media, outperforms others in recognizing the intentions behind these mixed-language posts. This means focusing on domain-specific learning helps AI understand nuanced language use better than just covering many languages generally.
What this means in practice
- •For social media analysts: Build tools to accurately classify user intent in mixed Arabic-English social media posts using domain-tuned Transformer models.
- •For multilingual chatbot developers: Enhance chatbot understanding of Arabic-English code-switching by implementing models pretrained on domain-relevant social media data.
Authors
Fahad Al Hussen, King Saud University, Riyadh, Saudi Arabia, Mohammed Q. Shormani, Ibb University, Ibb, Yemen
Abstract
This study highlights the role of domain-specific pretraining profile (DSPP) in Transformer performance for modeling digital pragmatics in Arabic-English code-switched discourse. It evaluates MARBERT and XLM-R(oBERTa), with BERT serving as a general-purpose baseline. The models were evaluated on their ability to classify context-sensitive pragmatic functions in code-switched social-media discourse. 11695 unique X posts were collected via Python and utilized for the study. The study employs a quantitative and qualitative NLP approach, following a supervised pipeline. Findings unveil that MARBERT consistently surpasses XLM-R with validation Macro F1 increasing from 0.39 to 0.84 and validation loss decreasing from 0.55 to 0.19. On an independent test set, it achieved 0.96 accuracy, 0.83 macro precision, 0.87 macro recall, and 0.85 Macro F1, while XLM-R achieved 0.92 test accuracy but a substantially lower Macro F1 of 0.52. This was also supported by class-level performance where MARBERT outperforms XLM-R considerably with F1 improvements ranging from +0.33 to +0.60, demonstrating a clear advantage in modeling Arabic digital pragmatics. The study concludes that Transformer performance depends more on DSPP than multilingual coverage alone, as the latter does not guarantee optimal performance on a highly specialized pragmatic classification task.