Multilingual character-level baby language models improve with synthetic data
Qwenfant: Train Big Character-level BabyLMs on Concise Parallel Text
Computation and Language
Summary
Training language models with lots of text helps them learn to understand and generate language. This paper shows that using text created and summarized by larger AI can improve the quality of training data. Changing the model to use characters instead of words or subwords and making the model bigger also helps. Tuning the model with a special technique called LoRA unlocked its full potential. The final model named Qwenfant performed best in a multilingual language modeling challenge.
What this means in practice
- •For natural language processing teams: Build more effective small-scale multilingual language models using synthetic summarized and translated data along with character-level modeling.
- •For machine translation developers: Use refined synthetic parallel texts with character-based language models to improve multilingual text understanding and generation.
Authors
Lukas Edman, Alexander Fraser
Abstract
This work concerns our submission for the Multilingual track of the 2026 BabyLM Challenge. We improve on our baseline using data that has been summarized and translated by an LLM, showing that higher-quality data can come from synthetic sources. We further improve with 2 simple changes: using a larger model, and using a character-level vocabulary. Both of these changes require LoRA-style fine-tuning for realizing their full potential. We dub our final Qwen3-based model Qwenfant, which, at the time of submission, ranks 1st in the Multilingual track.