NVIDIA Unveils a 2.8 Trillion-Token Indic Pretraining Corpus and the Open Pipeline Behind It

NVIDIA, IIT Madras and the AI4Bharat lab have described a large synthetic pretraining dataset for Indian languages: Nemotron-pretraining-Indic-v1, about 2.8 trillion tokens covering 22 scheduled languages, together with an open, GPU-accelerated pipeline for generating more. The announcement on Hugging Face is dated October 16, 2026. The idea is simple to state: high-quality English web data is abundant, Indic data is not, so translate the good English data at scale, score every translation, and keep only what passes.

This matters for anyone building multilingual products or training models for India, because the bottleneck there has long been data rather than compute. Mitesh Khapra of AI4Bharat is quoted in the post noting that Indian language technology trails English because of limited data. It also fits a pattern we have followed in NVIDIA's open releases, from the Nemotron 3.5 Super VL model that this data was used to improve, to the 34-language Nemotron reranker.

TL;DR: what was released

Token counts include accepted rewrites. NVIDIA is explicit that they are not unique English document counts and not a training-run budget, so do not read 2.8T as "2.8T tokens of distinct knowledge".

What is in the corpus?

Indic pretraining corpus composition by native-script and romanized token counts Image: NVIDIA / AI4Bharat

The corpus is split into two groups of languages, handled by two different workflows.

Twelve dual-script languages come in both native script and romanized (Latin-script) form: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu and Urdu. Romanized text matters because many Indian users type on phones in Latin letters, so a model that only knows Devanagari or Tamil script misses real usage. Configuration tags follow a language-script-region pattern, for example hi-Deva-IN for Hindi in Devanagari and hi-Latn-IN for romanized Hindi.

Ten native-script-only languages are lower-resource: Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali and Sindhi. Data for these is scarce enough that even synthetic text is a meaningful contribution.

The source material is English, taken from the Nemotron-CC-v2 family, with source IDs retained so each translated document can be traced to its origin. That provenance is useful if you later want to filter by topic, such as math or code, or remove a subset.

How does the pipeline work?

Both workflows are built on NeMo Data Designer, NVIDIA's framework for composing synthetic data steps, and run on GPUs.

The twelve-language workflow

1. Translate full documents with Gemma-4-31B-it, preserving code, equations, URLs and formatting. Keeping structure intact is what makes translated math and code usable as pretraining text.
2. Assess each translation against its English source using an MQM-inspired rubric (the same family of error-annotation schemes used in machine translation evaluation), covering meaning, fluency, terminology, style and formatting.
3. Keep or repair. Outputs scoring at least 75 out of 100 are kept. Lower scores are rewritten using the assessor's feedback and checked again. According to the post, this repair loop recovered 438.4 billion tokens that would otherwise have been discarded.
4. Romanize a subset by converting accepted native-script translations to Latin script, then checking the result against the native input.

The ten-language workflow

Here a dedicated document-level model, Indic Translate (8B parameters, from Bodhan AI), does the translation, and Gemma-4-31B-it acts as judge with the same 75/100 threshold. This path produces native-script data only and has no feedback-based rewriting. Two special cases are worth noting: long Santali documents are translated with overlapping windows, and Manipuri is translated via Bengali script and then converted to Meitei Mayek.

The generation-and-judging design is the reusable part. If you work on a language NVIDIA did not cover, the plugin lets you swap in your own translator and judge. The plugin lives in the NVIDIA-NeMo DataDesignerPlugins repository under the Nemotron Bharat translation folder, with a configuration reference in its docs.

Did it make a model better?

Continued pretraining before and after scores on Indic benchmarks Image: NVIDIA / AI4Bharat

NVIDIA tested the data in a continued-pretraining run for Nemotron 3.5 Super VL. The configured budget was 100B training tokens, of which 10B (10 percent) were synthetic plus natural Indic text. Across five benchmarks, scores rose by between 4.98 and 9.82 percentage points:

• MILU
• Indic MMLU-Pro
• Indic ARC-Challenge
• Indic ARC-Easy
• Indic GSM8K (Odia excluded because of context limits)

Compared with Nemotron 3 Super, the released model scores higher on five of six Indic task groups covering knowledge, reasoning, math, code and translation, with WMT24++ reported in chrF. It also improved on all four IndiVibe dimensions: usefulness, fluency, language and script fidelity, and verbosity. NVIDIA labels these preliminary conditional preference scores, not head-to-head win rates.

There is a cost: IFEval 1.5, an instruction-following benchmark, dropped 2.04 points. That is a typical trade-off when shifting a data mix toward another language, and it is the kind of regression to check before adopting a model in an agent workflow.

The most important caveat is attribution. The gains reflect the combined synthetic and natural Indic stage, not the synthetic corpus alone. The post does not claim that synthetic data alone produces the whole improvement, so a reader cannot conclude that 2.8T synthetic tokens beat the same amount of real text.

What are the limitations?

NVIDIA lists several, and they are the part to read most carefully:

• Translation is not native writing. Translated English carries English topics, idioms and assumptions. It does not replace text originally written in Indian languages and does not ensure coverage of local culture.
• The judge is also the translator in the 12-language workflow, since Gemma-4-31B-it does both jobs. Models tend to rate their own style favorably, so scores may be optimistic.
• Native-speaker validation is incomplete for the ten-language workflow.
• Deduplication is by source ID only, not semantic, so near-duplicate content can remain.
• License is unstated in the post. Review the dataset card before using it commercially.

What does this mean for builders?

If you build for Indian users, there are three practical angles.

Fine-tuning and evaluation. The benchmark list above (MILU, Indic MMLU-Pro, Indic ARC, Indic GSM8K) is a ready-made checklist for testing any model you plan to deploy in Hindi, Tamil or Bengali. A companion report, "Scaling Synthetic Pre-training Data and Evaluation for Indic LLMs", is on Hugging Face, and the AI4Bharat and NVIDIA evaluation suite has its own blog.

Data generation for your own language. The recipe is general: translate good source data, judge with a rubric, repair, keep. The same shape applies to any low-resource language, and it echoes zero-seed approaches such as the one in our coverage of synthetic training data APIs.

Romanized text. Few public corpora include romanized Indic text at scale. If your product handles chat or search input typed in Latin letters, this is the rare dataset that addresses it, though NVIDIA only romanized a subset.

It also adds to a broader picture of India as an AI market. Voice and speech work such as Rumik-OSS-1, an open Indic text-to-speech model, shows the same push for open Indic tooling, while labor questions are explored in our piece on India's entry-level hiring and AI. For NVIDIA's wider open-model strategy, see how the company framed open models and courseware for European science.

How to try it

1. Open the dataset collections linked from the announcement post and read the dataset card for the license and the exact configuration names.
2. Pick the language-script configs you need, such as hi-Deva-IN or hi-Latn-IN, rather than downloading everything.
3. For model comparison, start from the NVIDIA Nemotron 3.5 Super VL checkpoint and the Nemotron 3 Super baseline, and rerun the five Indic benchmarks on your own prompts.
4. To generate data for another language, install the Nemotron Bharat plugin and replace the translator and judge models.

Bottom line

The release is notable less for the headline number than for the open method: a documented, scored, repairable translation pipeline that others can reuse. The measured gains are real but modest in scope and tied to a mixed data stage, and the authors are candid that translated data cannot substitute for text written natively in these languages. Treat it as a strong starting layer, then add native sources.

Related reading

• Nemotron 3.5 Super VL: 120B multimodal open model
• Nemotron 3.5 Rerank 8B across 34 languages
• Rumik-OSS-1: open Indic TTS with emotion
• Synthetic training data from a zero-seed API
• India entry-level hiring and AI
• NVIDIA open models for European science

Details reflect NVIDIA's October 16, 2026 post; dataset availability and licensing may change, so verify on the dataset card.