French language model reveals tokenizer impact on grammar tests
Right Tool, Right Job: Native-Language Evaluation, Tokenizer Sensitivity, and Methodological Findings from a French-Only BabyLM
Computation and Language
Summary
Training a French-only language model, the authors found it performs well on native French grammar tests and some language tasks. They noticed some tasks improve while others get worse when combining French data with adaptations from English models. The study highlights that how words are split into pieces (tokenizer) can strongly affect scoring in tests, especially for small-scale models. They suggest better evaluation methods for such language models using native language tests and careful testing procedures.
What this means in practice
- •For natural language developers: Create French language models with optimal tokenizers and native test benchmarks to improve grammatical understanding.
- •For machine translation teams: Use cross-lingual adaptations while monitoring task-specific gains and losses to enhance translation and comprehension accuracy in French.
Authors
Adam Zachary Wasserman, David Beauchemin
Abstract
We submit MéTRON-FR, a 125M GPT-2 pretrained on 92.47M words of French, to the BabyLM 2026 Strict track. It scores 85.97 +/- 0.17% on QFrBLiMP (a native Quebec-French benchmark of grammatical minimal pairs) and 62.80% on the BabyLM-weighted leaderboard. A cross-lingual GLUE (General Language Understanding Evaluation) protocol that combines French task-data translation with rank-16 LoRA (Low-Rank Adaptation) produces a sharp task-type gradient: relational tasks gain measurably, while world-knowledge tasks regress. Bilingual Lexicon Induction aligns the French embeddings to GPT-2 at p@1 = 68.84 +/- 8.61%, 18X above chance, suggesting cross-lingual alignment tracks acquired grammatical competence rather than training duration. An ablation study shows that single-token zero-shot scoring is dominated by tokenizer and template artifacts at the child scale, motivating tokenizer-swap sensitivity, placebo-controlled prompting, and native-language minimal-pair benchmarks as standard diagnostics.