NVIDIA and AI4Bharat Release Massive Indic Pretraining Dataset
A joint initiative by NVIDIA, IIT Madras, and AI4Bharat introduces a 2.8 trillion token pretraining corpus and open data generation pipeline across 22 Indian languages.
A collaborative effort between NVIDIA, the Indian Institute of Technology (IIT) Madras, and its AI4Bharat lab has introduced a large-scale open resource for Indic language model pretraining. Titled Nemotron-pretraining-Indic-v1, the release spans approximately 2.8 trillion tokens across 22 Indian languages outlined in India's Constitution. According to the creators, these languages represent the primary speech of roughly 97 percent of the country's population.
Mitesh Khapra, head of AI4Bharat, has previously highlighted a core bottleneck in regional language technology: a shortage of robust training data for Indian languages. To help bridge this gap, the project not only publishes the massive text corpus but also makes its underlying, GPU-accelerated generation pipeline publicly accessible. This setup allows engineering teams to adapt prompts, leverage LLM workflows, and produce custom datasets for their specific domain requirements.
Structure of the Pretraining Corpus
The newly released corpus contains roughly 2.6 trillion native-script tokens and over 224 billion romanized tokens, utilizing the Gemma-4 tokenizer. The data covers 34 distinct language-script configurations. This includes native scripts for all 22 languages—such as Assamese, Bengali, Gujarati, Hindi, Kannada, and Tamil—alongside romanized text configurations for 12 of those languages.
To construct this resource, the teams utilized source material from 15 subsets of Nemotron-CC-v2, which encompasses web content, code, mathematics, and question-answer materials. They processed this content using NVIDIA's open-source NeMo Data Designer framework. The generation pipeline separates translation from script conversion, utilizing automated quality assessments inspired by Multidimensional Quality Metrics to ensure technical elements like equations, URLs, and code blocks remain intact. Outputs scoring below a strict threshold undergo automated repair processes before inclusion.
What it means for developers
For developers building multilingual AI applications, this release provides both direct data access and a reproducible synthetic generation blueprint. Teams can easily explore and stream splits directly using the Hugging Face datasets library without needing to download the entire multi-terabyte repository upfront. Developers looking to experiment with these capabilities or integrate cutting-edge models can try top AI models cheaply through one API at https://apixoai.online. Furthermore, developers can adapt the open-source translation plugins and use the provided source IDs to audit provenance, ensuring transparency in model training runs.
Model Performance Improvements
Evaluations detailed in the Nemotron 3.5 Super VL technical report demonstrate the practical impact of incorporating this data. During continued pretraining stages where a 10 billion token subset of Indic text was introduced, benchmark scores improved across multiple tasks including MILU, Indic MMLU-Pro, Indic ARC, and Indic GSM8K. Additionally, the final model variants exhibited performance gains across knowledge, reasoning, coding, and translation metrics when compared to previous iterations.
Source: Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline — Hugging Face blog. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
