Arabic language model improves faster with curated data and partial retraining
AQL-Lite: Enabling Efficient Knowledge Acquisition for Arabic LLMs via Data-Centric Continual Pretraining
Computation and LanguageMachine Learning
Summary
Training Arabic language models is hard because Arabic has many dialects and online data can be messy. The authors created AQL-Lite, a way to clean and select good Arabic text and then update parts of a large language model efficiently. This method improves how well the model understands Arabic by using less time and computing power than training everything. They showed big improvements on tests using less data and training time compared to older methods.
What this means in practice
- •For arabic nlp developers: Improve Arabic language model quality efficiently by using curated data and partial model retraining.
- •For ai engineering teams: Reduce training time and computation costs when adapting multilingual models to specific languages like Arabic.
- •For customer support platform builders: Build better Arabic chatbots by adapting language models with less training data and faster updates.$Commercial implications: Enables faster development of improved Arabic conversational AI products for customer service providers.
Authors
Houssam Eddine Boukhalfa, Mourad Bouache, Mourad Dahmane
Abstract
Training Arabic large language models faces challenges from noisy web data, dialectal variation, and the high cost of continual pretraining (CPT). We introduce AQL-Lite (AQL: Arabic Quality Language), a time- and resource-efficient framework for adapting a multilingual LLM to Modern Standard Arabic (MSA). AQL-Lite couples a quality-first curation pipeline (structural cleaning, heuristic and model-based filtering, MSA selection, and deduplication) with a partial-parameter CPT recipe that trains RS-LoRA adapters across the network together with the last eight transformer blocks and the embedding matrix, under layer-wise learning rates. At a matched 1B-token budget, CPT on AQL-curated data raises a four-benchmark Arabic mean by 3.51 points over CPT on raw web text, which yields no gain over the base model. In a controlled 5B-token comparison, AQL-Lite recovers 94% of the gain of full-parameter CPT (mean 73.19 vs. 73.58) in 52% of the wall-clock time and outperforms adapter-only baselines on every benchmark, and a component ablation shows that adapters, unfrozen blocks, and embedding training each contribute. Scaled to 40B MSA tokens and instruction-tuned on one million examples, the resulting 12B model scores 75.3 on the seven-benchmark Open Arabic LLM Leaderboard v2 (OALL v2) suite. These results suggest that partial-parameter CPT on a carefully curated corpus is a practical route to efficient MSA adaptation.