Hungarian language models show performance and evaluation weaknesses
Made in Hungary: Comments on the performance of generative language models
Computation and Language
Summary
Three separate attempts have been made to build AI language models that work well specifically for the Hungarian language, since existing big models often focus on English. The authors examined these projects and found problems: the ways they tested the models may not be reliable, some data they trained on might have leaked answers, and their training methods lack important controls. Also, the adapted models sometimes forgot what they learned before, lowering their overall skill. These findings show how careful testing and data preparation are important when making language models, especially given how costly it is to develop them.
What this means in practice
- •For language technology developers: Improve testing and data handling methods to increase Hungarian language model reliability and performance.
- •For ai safety teams: Use insights about data contamination and forgetting to better protect model integrity and monitor capability loss.
Authors
Mátyás Osváth, Enikő Héja, Noémi Ligeti-Nagy
Abstract
In recent years, three initiatives have emerged to develop generative language models in Hungary. The motivation behind them is the same. For Hungarian, no model with the given capability existed, or existing English-centric models offered limited proficiency. A detailed examination of the corresponding studies, however, reveals several methodological limitations. First, the reliability of the evaluation protocols is questionable. Contrary to the findings of Csibi et al. [2026], evaluation under the recommended inference settings shows that Qwen3-4B achieves higher scores than Racka-4B, its Hungarian-adapted version. Data contamination is evident in the work of Yang et al. [2025d] and Szentmihályi et al. [2025], potentially biasing the reported results. Second, the training pipelines fall short of current best practices in corpus curation and data mixture, which risks wasting substantial compute on low-quality data. The lack of controlled ablations prevents reliable assessment of these choices. Third, none of the three papers assessed forgetting or capability loss. Testing the adapted models on a subset of the original benchmarks indicates performance decline in all three cases, especially Racka-4B. These observations emphasize the importance of rigorous experimental design in language model development, given the significant computational and financial costs involved.