ORGANISATION

Assamese Wikipedia Corpus

A text corpus derived from Assamese language Wikipedia (as.wikipedia.org),It contains cleaned plain text extracted from Wikipedia articles.

About Dataset

This dataset is a collection of textual content extracted from Assamese Wikipedia, providing a large-scale digital corpus for the Assamese language. As one of the largest openly available digital text resources for Assamese, the dataset serves as an important resource for natural language processing, language modeling, information retrieval, machine learning, and computational linguistics research. The dataset also contributes to the preservation and advancement of digital resources for a low-resource Indian language. This dataset was identified and facilitated for onboarding as part of the Dataset Onboarding Support Team (DOST) initiative led by CivicDataLab (CDL), partnering with the Gates Foundation in collaboration with BHASHINI. CivicDataLab provided technical support for dataset discovery, validation, metadata preparation and onboarding facilitation. All dataset ownership and intellectual property rights remain with the original author(s).

Purpose of Dataset

The Purpose Of This Dataset Is To Support The Development Of Language Technologies And Digital Resources For The Assamese Language. It Can Be Used For Language Modeling, Natural Language Processing, Information Retrieval, Text Classification, Named Entity Recognition, Question Answering, Creating Assamese Specific Tokenizers, Machine Translation, Knowledge Extraction, Corpus Linguistics, Multilingual Ai Research, And Preservation Of Assamese Language Resources.

Dataset Metadata

License

Attribution 4.0 International (CC BY- 4.0)

Geographical coverage

Assam

Sector

Science, Technology and Research

Author

marsh-mellow

Source Organisation

Digital India BHASHINI Division

Uploaded by

Nikil Augustine

Data Quality Score (Beta)

Dataset type

Structured

Frequency

Time Granularity

Static

Year range

N.A.

Date & Time

06/07/26 10:27:48

Visibility

Open

Hosted / Redirected

Redirected

Data Type

Hybrid

If Redirection which source

Hugging Face

Data Collection Method

The Dataset Was Prepared By Extracting Articles From Assamese Wikipedia Using The Mwclient Python Library And Wikipedia Api. Mediawiki Markup Was Removed Using Mwparserfromhell, And The Text Was Filtered To Retain Only Assamese Characters, Numerals, Whitespace, And Essential Punctuation. Pages With Insufficient Content Or Predominantly Punctuation Were Excluded. The Cleaned Articles Were Stored In Json Lines (.Jsonl) Format, With Each Record Containing The Text Of A Single Assamese Wikipedia Article For Use In Nlp And Language Modeling Tasks.