Indian Flag
Government Of India
A-
A
A+
UpvoteUpvote
0
Northeast Languages Test Set

Northeast Languages Test Set

A multilingual evaluation test set for Northeast Indian languages containing 500 deduplicated sentences per language across Assamese, Garo, Khasi, Kokborok, Meitei, Mizo, Nagamese, Nyishi, and Pnar for benchmarking low-resource language models.
0
AI Score infoDownload score report
  • Upvotes0upvotes
  • Downloads7downloads
  • Views79views
  • File size0
  • Shares0Shares
  • Last updatedupdated 02 Jul 2026
UpvoteUpvote
0
COVERAGE Northeast India
LICENSE Attribution 4.0 International (CC BY- 4.0)
FORMAT Primary
TYPE Structured
SECTOR Sector Agnostic
UPDATES Static

About Dataset

Northeast Languages Test Set is a multilingual evaluation dataset developed by MWire Labs for benchmarking language models on low resource Northeast Indian languages. The dataset contains 500 deduplicated test sentences per language across Assamese Garo Khasi Kokborok Meitei Mizo Nagamese Nyishi and Pnar totaling 4500 evaluation samples. The dataset was created for multilingual language model evaluation and was used in the NE BERT paper for perplexity benchmarking across Northeast Indian languages. All test sentences were extracted from newer corpora not included in NE BERT training data and were deduplicated against the training corpus to reduce train test leakage. The dataset supports evaluation of multilingual NLP systems language models tokenizers embedding models and cross lingual transfer methods for low resource Indian languages. The included languages span Tibeto Burman Indo Aryan and Austroasiatic language families representing one of the most linguistically diverse regions of India. This resource is intended to support research benchmarking and development of AI systems for underserved Indian languages and regional language technologies.

Tags

  • Multilingual
  • low-resource
  • northeast-india

Related Datasets Related Datasets

Panchayati Raj and Drinkin
Panchayati Raj and Drinkin
4.2Information•Updated 9 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views2
  • File Size282.56 KB

Agent Communication
Agent Communication
4.5Information•Updated 10 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views17
  • File Size4.08 MB

INDIAN INSTITUTE OF SCIENCE (IISC), BANGALORE
  • See Upvoters2
  • Downloads218
  • Views2,882
  • File Size0

IndicDLP
IndicDLP
5.0Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads388
  • Views2,599
  • File Size0

Lahaja
Lahaja
-Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads17
  • Views151
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads57
  • Views350
  • File Size0

Svarah
Svarah
-Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads33
  • Views231
  • File Size0

IndicVoices-R
IndicVoices-R
-Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads60
  • Views546
  • File Size0

Shrutilipi
Shrutilipi
-Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads26
  • Views252
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads199
  • Views3,155
  • File Size0


Linked Model Linked Model

NE-BERT
NE-BERT is Northeast India's first domain-specific multilingual foundation model. Built on the ModernBERT architecture and trained on 8.3 million sentences, it supports 9 regional languages: Assamese, Khasi, Garo, Manipuri (Meitei), Mizo, Nyishi, Nagamese, Kokborok, and Pnar. It achieves State-of-the-Art performance on regional benchmarks and offers 1.6x faster inference, bridging the digital divide for low-resource languages.
MWIRE LABS
  • See Upvoters0
  • Downloads27
  • Views857
  • File Size0


Related Datasets by Organization Related Datasets by Organization

Panchayati Raj and Drinkin
Panchayati Raj and Drinkin
4.2Information•Updated 9 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views2
  • File Size282.56 KB

Agent Communication
Agent Communication
4.5Information•Updated 10 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views17
  • File Size4.08 MB

INDIAN INSTITUTE OF SCIENCE (IISC), BANGALORE
  • See Upvoters2
  • Downloads218
  • Views2,882
  • File Size0

IndicDLP
IndicDLP
5.0Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads388
  • Views2,599
  • File Size0

Lahaja
Lahaja
-Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads17
  • Views151
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads57
  • Views350
  • File Size0

Svarah
Svarah
-Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads33
  • Views231
  • File Size0

IndicVoices-R
IndicVoices-R
-Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads60
  • Views546
  • File Size0

Shrutilipi
Shrutilipi
-Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads26
  • Views252
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads199
  • Views3,155
  • File Size0


Related Datasets by Sector Related Datasets by Sector

Panchayati Raj and Drinkin
Panchayati Raj and Drinkin
4.2Information•Updated 9 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views2
  • File Size282.56 KB

Agent Communication
Agent Communication
4.5Information•Updated 10 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views17
  • File Size4.08 MB

INDIAN INSTITUTE OF SCIENCE (IISC), BANGALORE
  • See Upvoters2
  • Downloads218
  • Views2,882
  • File Size0

IndicDLP
IndicDLP
5.0Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads388
  • Views2,599
  • File Size0

Lahaja
Lahaja
-Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads17
  • Views151
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads57
  • Views350
  • File Size0

Svarah
Svarah
-Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads33
  • Views231
  • File Size0

IndicVoices-R
IndicVoices-R
-Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads60
  • Views546
  • File Size0

Shrutilipi
Shrutilipi
-Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads26
  • Views252
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads199
  • Views3,155
  • File Size0


Related Datasets by Task Type Related Datasets by Task Type

Panchayati Raj and Drinkin
Panchayati Raj and Drinkin
4.2Information•Updated 9 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views2
  • File Size282.56 KB

Agent Communication
Agent Communication
4.5Information•Updated 10 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters0
  • Downloads0
  • Views17
  • File Size4.08 MB

INDIAN INSTITUTE OF SCIENCE (IISC), BANGALORE
  • See Upvoters2
  • Downloads218
  • Views2,882
  • File Size0

IndicDLP
IndicDLP
5.0Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads388
  • Views2,599
  • File Size0

Lahaja
Lahaja
-Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads17
  • Views151
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads57
  • Views350
  • File Size0

Svarah
Svarah
-Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads33
  • Views231
  • File Size0

IndicVoices-R
IndicVoices-R
-Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads60
  • Views546
  • File Size0

Shrutilipi
Shrutilipi
-Information•Updated 13 day(s) ago
AI4BHARAT
  • See Upvoters0
  • Downloads26
  • Views252
  • File Size0

AI4BHARAT
  • See Upvoters0
  • Downloads199
  • Views3,155
  • File Size0