Indian Flag
Government Of India
A-
A
A+
UpvoteUpvote
0

Pralekha

Pralekha is a large-scale parallel document dataset spanning across 11 Indic languages and English
0
AI Score infoDownload score report
  • Upvotes0upvotes
  • Downloads19downloads
  • Views147views
  • File size0
  • Shares0Shares
  • Last updatedupdated 05 Aug 2025
UpvoteUpvote
0
COVERAGE National
LICENSE Attribution 4.0 International (CC BY- 4.0)
TYPE Structured
SECTOR Sector Agnostic
UPDATES Static

About Dataset

Pralekha covers 12 languages—Bengali (ben), Gujarati (guj), Hindi (hin), Kannada (kan), Malayalam (mal), Marathi (mar), Odia (ori), Punjabi (pan), Tamil (tam), Telugu (tel), Urdu (urd), and English (eng). It includes a mixture of high- and medium-resource languages, covering 11 different scripts. The dataset spans two broad domains: News Bulletins (Indian Press Information Bureau (PIB)) and Podcast Scripts (Mann Ki Baat (MKB)), offering both written and spoken forms of data. All the data is human-written or human-verified, ensuring high quality.

While this accounts for alignable (parallel) documents, In real-world scenarios, multilingual corpora often include unalignable documents. To simulate this for CLDA evaluation, we sample unalignable documents from Sangraha Unverified, selecting 50% of Pralekha’s size to maintain a 1:2 ratio of unalignable to alignable documents.

For Machine Translation (MT) tasks, we first randomly sample 1,000 documents from the alignable subset per English-Indic language pair for each development (dev) and test set, ensuring a good distribution of varying document lengths. After excluding these sampled documents, we use the remaining documents as the training set for training document-level machine translation models.

Tags

  • Parallel Corpus
  • Machine Translation
  • multilingual NLP
  • Indic Languages
  • document-alignment

Related Datasets Related Datasets

Panchayati Raj and Drinkin
Panchayati Raj and Drinkin
4.2Information•Updated 10 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views2
  • File Size282.56 KB

Agent Communication
Agent Communication
4.5Information•Updated 11 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views19
  • File Size4.08 MB

INDIAN INSTITUTE OF SCIENCE (IISC), BANGALORE
  • See Upvoters2
  • Downloads218
  • Views2,893
  • File Size0

IndicDLP
IndicDLP
5.0Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads388
  • Views2,605
  • File Size0

Lahaja
Lahaja
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads17
  • Views156
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads57
  • Views353
  • File Size0

Svarah
Svarah
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads33
  • Views235
  • File Size0

IndicVoices-R
IndicVoices-R
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads60
  • Views549
  • File Size0

Shrutilipi
Shrutilipi
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads26
  • Views257
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads199
  • Views3,164
  • File Size0


Related Datasets by Organization Related Datasets by Organization

Panchayati Raj and Drinkin
Panchayati Raj and Drinkin
4.2Information•Updated 10 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views2
  • File Size282.56 KB

Agent Communication
Agent Communication
4.5Information•Updated 11 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views19
  • File Size4.08 MB

INDIAN INSTITUTE OF SCIENCE (IISC), BANGALORE
  • See Upvoters2
  • Downloads218
  • Views2,893
  • File Size0

IndicDLP
IndicDLP
5.0Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads388
  • Views2,605
  • File Size0

Lahaja
Lahaja
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads17
  • Views156
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads57
  • Views353
  • File Size0

Svarah
Svarah
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads33
  • Views235
  • File Size0

IndicVoices-R
IndicVoices-R
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads60
  • Views549
  • File Size0

Shrutilipi
Shrutilipi
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads26
  • Views257
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads199
  • Views3,164
  • File Size0


Related Datasets by Sector Related Datasets by Sector

Panchayati Raj and Drinkin
Panchayati Raj and Drinkin
4.2Information•Updated 10 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views2
  • File Size282.56 KB

Agent Communication
Agent Communication
4.5Information•Updated 11 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views19
  • File Size4.08 MB

INDIAN INSTITUTE OF SCIENCE (IISC), BANGALORE
  • See Upvoters2
  • Downloads218
  • Views2,893
  • File Size0

IndicDLP
IndicDLP
5.0Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads388
  • Views2,605
  • File Size0

Lahaja
Lahaja
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads17
  • Views156
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads57
  • Views353
  • File Size0

Svarah
Svarah
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads33
  • Views235
  • File Size0

IndicVoices-R
IndicVoices-R
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads60
  • Views549
  • File Size0

Shrutilipi
Shrutilipi
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads26
  • Views257
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads199
  • Views3,164
  • File Size0


Related Datasets by Task Type Related Datasets by Task Type

Panchayati Raj and Drinkin
Panchayati Raj and Drinkin
4.2Information•Updated 10 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views2
  • File Size282.56 KB

Agent Communication
Agent Communication
4.5Information•Updated 11 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views19
  • File Size4.08 MB

INDIAN INSTITUTE OF SCIENCE (IISC), BANGALORE
  • See Upvoters2
  • Downloads218
  • Views2,893
  • File Size0

IndicDLP
IndicDLP
5.0Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads388
  • Views2,605
  • File Size0

Lahaja
Lahaja
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads17
  • Views156
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads57
  • Views353
  • File Size0

Svarah
Svarah
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads33
  • Views235
  • File Size0

IndicVoices-R
IndicVoices-R
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads60
  • Views549
  • File Size0

Shrutilipi
Shrutilipi
-Information•Updated 14 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads26
  • Views257
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads199
  • Views3,164
  • File Size0