Largest synthetic pretraining data boosts Indian language models
Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs
Computation and Language
Summary
Building good language tools for India's many languages is hard because there isn't enough training data and ways to test them well. The authors created a very large synthetic dataset by translating lots of English documents into 22 Indian languages with high quality, and they improved low-quality translations automatically. They showed training with this dataset plus real Indian text helps language models understand and reason better than before. They also made a new test set to check how well models work in these languages across different tasks like chatting, coding, and translating.
What this means in practice
- •For machine learning engineers: Train Indian language models using the large synthetic corpus and new benchmarks to improve performance across diverse Indic languages and tasks.
- •For software localization teams: Evaluate and improve language support for Indian languages in chatbots, translation tools, and code generation using the Indic-LLM-Eval-Suite.
Authors
Abhinav Khattar, Aishwarya Chhabra, Ameya Velingker, Anusha Kamath, Brandon Norick, Kanishk Singla, Kiran Praveen, Markus Kliegl, Mohammad Qazim Bhat, Mostofa Patwary, Niket Agarwal, Rajpreet Thethy, Ramani Akkati, Raviraj Joshi, Sai Nishanth Padala, Shuoyang Ding, Sriharsha Niverty, Utkarsh Vaidya, Vitaly Lavrukhin, Yev Meyer, Anoop Kunchukuttan, Mitesh Khapra, Mohammed Safi Ur Rahman Khan, Rudra Murthy, Sumanth Doddapaneni, Thanmay Jayakumar, Ujjwal Sharma
Abstract
Building capable language models for India requires scalable multilingual data and robust evaluation. We introduce Nemotron-Pretraining-Indic-v1, a synthetic pretraining corpus of approximately 2.83 trillion tokens covering all 22 scheduled Indian languages in native script and twelve in romanized form. It is the largest openly released Indic pretraining corpus. Using our GPU-accelerated pipeline, we translate curated English documents, assess translation quality, and repair low-quality outputs while preserving technical content and document structure and maintaining source traceability. Through continued-pretraining experiments across twelve languages, we show that combining our synthetic corpus with natural Indic text improves knowledge and reasoning over a public-corpus baseline. Additional experiments show how data volume, script coverage, and Indic training share affect English performance. We also introduce the Indic-LLM-Eval-Suite, a multilingual benchmark suite for evaluating Indic language models across chat, instruction following, reasoning, knowledge, code, and translation. The suite combines existing Indic benchmarks with task-aware translations of English benchmarks, using automated quality assessment, targeted correction, and native-speaker review in a human-in-the-loop pipeline. We use the suite to comprehensively benchmark contemporary open-weight Indic and multilingual models across capabilities and languages, revealing substantial variation across tasks and languages. Together, these resources provide a scalable foundation for developing and evaluating more capable Indic language models.