PahariLI - Pahari Language Identification Corpus

A labeled text corpus for automatic identification of four Pahari languages: Dogri, Garhwali, Kumaoni, and Nepali. Designed to support language identification and multilingual NLP research.

About Dataset

This dataset is a labeled text corpus developed for automatic language identification of four Northern Indo-Aryan (Pahari) languages: Dogri, Garhwali, Kumaoni, and Nepali. Each sentence in the dataset is annotated with the ISO 639-3 code corresponding to its language, enabling supervised machine learning and benchmark evaluation. The dataset serves as an important resource for developing language identification systems, multilingual language technologies, and computational tools for Pahari languages. This dataset was identified and facilitated for onboarding as part of the Dataset Onboarding Support Team (DOST) initiative led by CivicDataLab (CDL), partnering with the Gates Foundation in collaboration with BHASHINI. CivicDataLab provided technical support for dataset discovery, validation, metadata preparation and onboarding facilitation. All dataset ownership and intellectual property rights remain with the original author(s).

Purpose of Dataset

The Purpose Of This Dataset Is To Support The Development And Evaluation Of Automatic Language Identification Systems For Pahari Languages. It Can Be Used For Language Identification, Text Classification, Multilingual Nlp, Machine Learning Research, Corpus Linguistics, Low-resource Language Technology Development, And Computational Studies Involving Dogri, Garhwali, Kumaoni, And Nepali.

Dataset Metadata

License

Apache 2.0

Geographical coverage

India

Sector

Science, Technology and Research

Author

rachanagusain

Source Organisation

Digital India BHASHINI Division

Uploaded by

Nikil Augustine

Data Quality Score (Beta)

Dataset type

Structured

Frequency

Time Granularity

Year range

N.A.

Date & Time

06/07/26 11:36:56

Visibility

Open

Hosted / Redirected

Redirected

Data Type

Hybrid

If Redirection which source

Github

Data Collection Method

The Dataset Was Prepared By Collecting And Curating Textual Data In Four Pahari Languages: Dogri, Garhwali, Kumaoni, And Nepali. Sentences Were Organized Into Language-specific Categories And Labeled Using Their Respective Iso 639-3 Language Codes. The Corpus Was Divided Into Training And Test Sets Containing 44,014 And 11,004 Labeled Sentences Respectively, Enabling Supervised Machine Learning And Benchmark Evaluation For Automatic Language Identification Tasks.