Large-Scale Multilingual Training Datasets for Large Language Models

As part of its mission to strengthen India’s AI ecosystem, AI Kosh, in collaboration with Meta and Sarvam AI, is making available 12 billion high-quality synthetic Indic training tokens along with a fine-tuned 17B Llama-4 MoE model and the reproducible fine-tuning pipeline used to train it.

The release includes 14 structured datasets designed for training and fine-tuning large language models in 10 major Indian languages: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, and Telugu.

The datasets are generated through large-scale synthetic data generation pipelines to augment scarce high-quality training data in low-resource Indian languages. They are structured across three real-world linguistic styles: formal language for academic and official use cases, code-mixed conversational formats such as Hinglish and Tanglish, and transliteration across Roman and native scripts.

The datasets cover a broad range of high-impact tasks, including reasoning, instruction following, conversational AI, code generation, agentic workflows, multilingual translation, creative writing, and summarization. This breadth enables the development of multilingual AI systems that can operate reliably across domains such as scientific research support, healthcare communication, educational copilots, agricultural advisory systems, citizen services, and enterprise knowledge platforms.

In addition to the datasets, the initiative includes a fine-tuned 17B Llama-4 MoE model with 17B active parameters across 16 experts and 109B total parameters. Trained on a subset of the released dataset, the model demonstrates measurable improvements in Indic language fluency, instruction following, domain-specific reasoning, and cross-lingual performance.

The model release includes training configuration details, fine-tuning scripts, preprocessing utilities, evaluation setup, and reproducibility documentation. Together, these assets provide a reference implementation for open, large-scale multilingual model training and lower the barrier for researchers, startups, enterprises, and public institutions to build high-performance AI systems aligned with India’s linguistic diversity.