Romanian first language model family trains from scratch and shows promise
Neurograf v1: An End-to-End Romanian-First Language Model Family
Computation and LanguageMachine Learning
Summary
Languages with less digital text often don’t have their own AI language models because it’s hard to train them well. The researchers created a large set of Romanian texts and built several language models from small to large based only on that data. Bigger models learned more Romanian words and facts, and one scored about 70% on a Romanian quiz. They found challenges like keeping accent marks and making sure the data was good, but showed it’s possible to build useful Romanian language AI from scratch.
What this means in practice
- •For natural language processing engineers: Build Romanian language models and assistants with a unified training pipeline specialized for this language.
- •For machine learning platform teams: Train and evaluate language models efficiently for languages with limited data using the described corpus and tokenization approach.
Authors
Ionut Adrian Rotaru
Abstract
We present Neurograf v1, an experimental end-to-end Romanian-first language-model pipeline developed to investigate the feasibility of training language models from scratch for a lower-resource European language. The pipeline comprises RoSfera, a 635 GB Romanian corpus containing 175.7 million documents; an 80k byte-level BPE tokenizer; five decoder-only models ranging from 160M to 6B parameters, trained on 131 billion tokens; three instruction-tuned assistants; and two contrastively trained embedding models. Training and evaluation were conducted using computing infrastructure at ELI-NP/IFIN-HH. The model family exhibits consistent scaling of validation loss and increasing Romanian factual and cultural knowledge with model size. On the 1,000-question WWTBM benchmark, Neurograf-6B achieves 69.89% accuracy on Romanian-context questions, while Neurograf-3.2B reaches 67.38%. Compute-optimal analysis yields a parameter-scaling exponent of approximately 0.423. Neurograf v1 serves as an experimental demonstrator rather than a production system. The study identifies practical challenges involving diacritic preservation, corpus quality, data composition, and tokenization efficiency, while demonstrating the feasibility of developing Romanian-first base models, instruction-following assistants, and semantic embedding models within a unified training pipeline.