Northeast Languages Test Set

About Dataset

Northeast Languages Test Set is a multilingual evaluation dataset developed by MWire Labs for benchmarking language models on low resource Northeast Indian languages. The dataset contains 500 deduplicated test sentences per language across Assamese Garo Khasi Kokborok Meitei Mizo Nagamese Nyishi and Pnar totaling 4500 evaluation samples. The dataset was created for multilingual language model evaluation and was used in the NE BERT paper for perplexity benchmarking across Northeast Indian languages. All test sentences were extracted from newer corpora not included in NE BERT training data and were deduplicated against the training corpus to reduce train test leakage. The dataset supports evaluation of multilingual NLP systems language models tokenizers embedding models and cross lingual transfer methods for low resource Indian languages. The included languages span Tibeto Burman Indo Aryan and Austroasiatic language families representing one of the most linguistically diverse regions of India. This resource is intended to support research benchmarking and development of AI systems for underserved Indian languages and regional language technologies.