Arabic language models tested with new detailed error analysis dataset

E-CONAN (Entailment, CONtradition And Neutral) Diagnostics Dataset Investigating Linguistic Phenomena in Arabic Natural Language Understanding

Computation and LanguageArtificial IntelligenceHuman-Computer InteractionMachine LearningLogic in Computer Science

Summary

Understanding human language is complicated for computers because of things like word meanings and logic. The authors noticed there wasn’t a clear way to categorize and analyze errors in Arabic language understanding systems. They created a new Arabic dataset called E-CONAN that organizes errors into broad and detailed categories so developers can better see where their systems struggle. They tested several language AI models and found some did better on world facts while others handled sentence structure better. This dataset helps improve Arabic natural language tools by pinpointing their weaknesses.

What this means in practice

  • For arabic nlu developers: Assess and improve Arabic natural language understanding models by analyzing their errors with a detailed, linguistically informed dataset.
  • For multilingual ai teams: Use the proposed error categorization hierarchy to compare and refine cross-lingual language models’ performance with consistent linguistic criteria.

Authors

Khloud AL Jallad, Nada Ghneim, Ghaida Rebdawi

Abstract

Natural Language Understanding (NLU) plays a crucial role in various applications, yet its performance suffers from weaknesses in handling the complexities of human languages, ranging from lexical ambiguity to high-level reasoning difficulties. Analyzing errors across diverse linguistic phenomena is crucial for NLU improvement, as it will help humans get insights to comprehensively assess models' limitations and capabilities, so optimizing models' generalization. Notably, several benchmarks contain diagnostics datasets designed for investigation and fine-grained error analysis. When highlighting the gaps in the state-of-the-art, we noted that there is no naming convention for macro and micro categories or even a standard set of linguistic phenomena that should be covered. To overcome this gap, we propose an initial hierarchy for Cross-Lingual NLU error analysis. Moreover, we propose a methodology to create an NLI hierarchical framework and applied a case study on Arabic NLU. Moreover, this paper introduces E-CONAN diagnostics dataset, a freely available dataset manually-annotated with coarse-grained and fine-grained categories based on our proposed Arabic hierarchy. E-CONAN dataset helps NLU designers better understand their models by doing error analysis and in-depth investigation. We used E-CONAN to investigate the performance of 9 pretrained language models and 5 LLMs. Results indicate that LLMs outperform pretrained models in world knowledge and commonsense reasoning macro-category, and underperform pretrained models in syntactic macro-category. Moreover, the hardest phenomena for all models is Reasoning, and the easiest phenomena for all pretrained models is Syntactic, and the easiest for LLMs is Lexico-Syntactic.
View PDFOpen arXiv