Indian Flag
Government Of India
A-
A
A+
UpvoteUpvote
1
EKA Pretraining Indic Corpus v1

EKA Pretraining Indic Corpus v1

EKA Pretraining Indic Corpus v1 — a multilingual Indic dataset under Project EKA for India’s 120B foundation model.
4.75
AI Score infoDownload score report
  • Upvotes1upvotes
  • Downloads127downloads
  • Views4,602views
  • File size458.23 GB
  • Shares0Shares
  • Last updatedupdated 15 Dec 2025
UpvoteUpvote
1
COVERAGE Global
LICENSE Attribution-Non-Commercial 4.0 International (CC BY-NC 4.0)
TYPE Unstructured
SECTOR Sector Agnostic
UPDATES Static

About Dataset

Overview

Dataset name: EKA_PretrainingIndicCorpus_v1

Release: Version 1 — initial public release

Languages: All Indic languages including English

Format: CSV (uri and text)

Scale: Multi-billion-token text corpus aggregated until October 2025

License: Open Source

Supported by: IndiaAI Mission, Ministry of Electronics & IT, Government of India

Contributors: Indian Institute of Technology Gandhinagar and Soket AI Labs under Project EKA


About Project EKA

EKA is India’s open-source AI initiative aimed at building a sovereign, large-scale foundation model ecosystem rooted in Indic languages and values.

It represents a national effort to develop AI systems that are open, multilingual, ethical, energy-efficient, and community-driven.


Through EKA, we are constructing a transparent and inclusive foundation model stack — from datasets and benchmarks to training infrastructure — ensuring that India’s AI capabilities are built in India, for India, and for the world.


Key pillars of Project EKA include:

• Developing foundational models across languages and modalities

• Building high-quality, bias-aware datasets representative of India’s linguistic and cultural diversity

• Promoting open research and collaboration across academia and industry

• Advancing energy-efficient and climate-aware AI training

• Ensuring responsible, ethical, and safe AI deployment


About Soket AI

Soket AI is an Indian frontier AI lab focused on building next-generation multilingual foundation models and ethical AI systems.

We design models, datasets, and APIs that enable enterprises, developers, and researchers to build human-aligned intelligence efficiently and responsibly.

Soket’s mission is to ensure that AI made in India reflects India’s diversity, languages, and values while contributing to the global AI commons.

As one of the core builders of Project EKA, Soket AI drives dataset creation, model training, and infrastructure development for large-scale Indic models.


About IIT Gandhinagar

The Indian Institute of Technology Gandhinagar (IITGn) leads the academic and research partnership for Project EKA.

The institute contributes its expertise in data curation, computational linguistics, and large-scale machine learning, ensuring that the corpus meets rigorous research and linguistic standards.


Data Sources

The corpus is aggregated from multiple open and public Indic domains, including:

• National and regional news media

• Educational and research publications

• Government and institutional releases

• Open-web text sources

• Social and cultural materials available in the public domain

• Domain-specific texts from agriculture, health, education, and technology sectors


All content has been collected with the objective of maximising linguistic coverage, topical diversity, and cross-domain representativeness.


Purpose and Use

The EKA Indic Large Text Corpus serves as the foundational dataset for:

• Pretraining and fine-tuning multilingual Indic large language models (LLMs)

• Developing cross-lingual transfer, translation, and summarization systems

• Creating domain-specific Indic NLP models for education, agriculture, and governance

• Benchmarking and evaluating Indian-language LLMs

• Enabling open and inclusive AI research across academia and startups


This corpus will power the training of the upcoming 120 B parameter EKA Foundation Model, designed to represent India’s linguistic and cultural landscape at scale.


Roadmap

This is the first release in a multi-stage dataset roadmap.

Future versions will expand and refine the corpus with:

• Additional Indic languages and regional dialects

• Domain-balanced data distributions

• Improved metadata coverage

• Multimodal (speech and image) extensions

• Benchmarks and evaluation splits for downstream tasks

• Continuous integration with EKA’s model training pipelines


The long-term goal is to build a multi-trillion-token dataset powering India’s open-source foundation models.


Note: A sample of the dataset is available to download as a single archive file. The entire dataset can be fetched from individual CSV files.

Tags

  • text
  • llm
  • indic
  • pretraining

Linked Dataset Linked Dataset

bhasha‑wiki-Urdu
bhasha‑wiki-Urdu
-Information•Updated 1 year(s) ago
SOKET LABS TECHNOLOGY AND RESEARCH PRIVATE LIMITED
  • See Upvoters0
  • Downloads10
  • Views298
  • File Size0

bhasha‑wiki-Tamil
bhasha‑wiki-Tamil
-Information•Updated 1 year(s) ago
SOKET LABS TECHNOLOGY AND RESEARCH PRIVATE LIMITED
  • See Upvoters0
  • Downloads18
  • Views361
  • File Size0

bhasha‑wiki-Hindi
bhasha‑wiki-Hindi
-Information•Updated 1 year(s) ago
SOKET LABS TECHNOLOGY AND RESEARCH PRIVATE LIMITED
  • See Upvoters0
  • Downloads38
  • Views379
  • File Size0

bhasha‑wiki-Gujrati
bhasha‑wiki-Gujrati
-Information•Updated 1 year(s) ago
SOKET LABS TECHNOLOGY AND RESEARCH PRIVATE LIMITED
  • See Upvoters2
  • Downloads17
  • Views588
  • File Size0

bhasha‑wiki-English
bhasha‑wiki-English
-Information•Updated 1 year(s) ago
SOKET LABS TECHNOLOGY AND RESEARCH PRIVATE LIMITED
  • See Upvoters0
  • Downloads34
  • Views228
  • File Size0

bhasha‑wiki-Bengali
bhasha‑wiki-Bengali
-Information•Updated 1 year(s) ago
SOKET LABS TECHNOLOGY AND RESEARCH PRIVATE LIMITED
  • See Upvoters0
  • Downloads17
  • Views484
  • File Size0

bhasha‑wiki-Kannada
bhasha‑wiki-Kannada
-Information•Updated 1 year(s) ago
SOKET LABS TECHNOLOGY AND RESEARCH PRIVATE LIMITED
  • See Upvoters0
  • Downloads17
  • Views363
  • File Size0


Linked Model Linked Model

EKA-1-26B
EKA-1 is a 26-billion-parameter foundation model built from scratch by Soket AI with support from IndiaAI, MeitY. Designed for mathematics, coding, reasoning and agentic workflows across 100+ languages, it offers a 64K-token context window and flexible deployment on-premises, in air-gapped environments or in private clouds.
SOKET LABS TECHNOLOGY AND RESEARCH PRIVATE LIMITED
  • See Upvoters0
  • Downloads0
  • Views2
  • File Size0

EKA-1-26B
EKA-1 is a 26-billion-parameter foundation model built from scratch by Soket AI with support from IndiaAI, MeitY. Designed for mathematics, coding, reasoning and agentic workflows across 100+ languages, it offers a 64K-token context window and flexible deployment on-premises, in air-gapped environments or in private clouds.
SOKET LABS TECHNOLOGY AND RESEARCH PRIVATE LIMITED
  • See Upvoters6
  • Downloads0
  • Views2,911
  • File Size30.49 GB


Related Datasets by Organization Related Datasets by Organization

Panchayati Raj and Drinkin
Panchayati Raj and Drinkin
4.2Information•Updated 10 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views2
  • File Size282.56 KB

Agent Communication
Agent Communication
4.5Information•Updated 11 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views19
  • File Size4.08 MB

INDIAN INSTITUTE OF SCIENCE (IISC), BANGALORE
  • See Upvoters2
  • Downloads218
  • Views2,893
  • File Size0

IndicDLP
IndicDLP
5.0Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads388
  • Views2,605
  • File Size0

Lahaja
Lahaja
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads17
  • Views156
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads57
  • Views353
  • File Size0

Svarah
Svarah
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads33
  • Views235
  • File Size0

IndicVoices-R
IndicVoices-R
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads60
  • Views549
  • File Size0

Shrutilipi
Shrutilipi
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads26
  • Views257
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads199
  • Views3,164
  • File Size0


Related Datasets by Sector Related Datasets by Sector

Panchayati Raj and Drinkin
Panchayati Raj and Drinkin
4.2Information•Updated 10 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views2
  • File Size282.56 KB

Agent Communication
Agent Communication
4.5Information•Updated 11 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views19
  • File Size4.08 MB

INDIAN INSTITUTE OF SCIENCE (IISC), BANGALORE
  • See Upvoters2
  • Downloads218
  • Views2,893
  • File Size0

IndicDLP
IndicDLP
5.0Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads388
  • Views2,605
  • File Size0

Lahaja
Lahaja
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads17
  • Views156
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads57
  • Views353
  • File Size0

Svarah
Svarah
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads33
  • Views235
  • File Size0

IndicVoices-R
IndicVoices-R
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads60
  • Views549
  • File Size0

Shrutilipi
Shrutilipi
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads26
  • Views257
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads199
  • Views3,164
  • File Size0


Related Datasets by Task Type Related Datasets by Task Type

Panchayati Raj and Drinkin
Panchayati Raj and Drinkin
4.2Information•Updated 10 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views2
  • File Size282.56 KB

Agent Communication
Agent Communication
4.5Information•Updated 11 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views19
  • File Size4.08 MB

INDIAN INSTITUTE OF SCIENCE (IISC), BANGALORE
  • See Upvoters2
  • Downloads218
  • Views2,893
  • File Size0

IndicDLP
IndicDLP
5.0Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads388
  • Views2,605
  • File Size0

Lahaja
Lahaja
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads17
  • Views156
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads57
  • Views353
  • File Size0

Svarah
Svarah
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads33
  • Views235
  • File Size0

IndicVoices-R
IndicVoices-R
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads60
  • Views549
  • File Size0

Shrutilipi
Shrutilipi
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads26
  • Views257
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads199
  • Views3,164
  • File Size0