Varta - a large-scale multilingual and headline-generation dataset

About Dataset

Varta is a diverse, challenging, large-scale, multilingual, and high-quality headline-generation dataset containing 41.8 million news articles in 14 Indic languages and English. Languages are - Assamese, Bhojpuri, Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Oriya, Punjabi, Tamil, Telugu, and Urdu. It enables research in language understanding, text generation, summarization, and multilingual AI applications.The citation of the dataset - Aralikatte, Rahul, Ziling Cheng, Sumanth Doddapaneni, and Jackie Chi Kit Cheung. “Vārta: A Large-Scale Headline-Generation Dataset for Indic Languages.” arXiv preprint arXiv:2305.05858, 2023. Available at - https://arxiv.org/abs/2305.05858. This dataset was identified and facilitated for onboarding as part of the Dataset Onboarding Support Team (DOST) initiative led by by CivicDataLab (CDL), partnering with the Gates Foundation in collaboration with BHASHINI. CivicDataLab provided technical support for dataset discovery, validation, metadata preparation and onboarding facilitation. All dataset ownership and intellectual property rights remain with the original author(s)."