Indian Flag
Government Of India
A-
A
A+
UpvoteUpvote
0

Thore Bhasha-Setu

Tokenization: Using a custom SentencePiece tokenizer trained on the entire Indic corpus to efficiently handle the morphology of Indian languages.
  • Upvotes0upvotes
  • Downloads17downloads
  • Views475views
  • File size170.07 KB
  • Shares0Shares
  • Last updatedupdated 28 Oct 2025
UpvoteUpvote
0
LICENSE Attribution 4.0 International (CC BY- 4.0)
SECTOR Sector Agnostic

About Model

Dataset Strategy


The model's performance hinges on a high-quality, diverse dataset. The "Thore Bhasha-Setu 1B" dataset would be a curated collection of approximately 200 billion tokens.

  • Data Composition:

  • Massive Monolingual Corpus (60%): Sourced from web crawls of regional news sites (e.g., Dainik Jagran, Anandabazar Patrika), digital libraries, Indic Wikipedia, and publicly available books. This builds the model's core understanding of each language's grammar and vocabulary.

  • Parallel Corpus (20%): High-quality translated text. This includes government documents (press releases, legal texts available in multiple languages), professionally translated news articles, and movie subtitles. This is crucial for translation tasks. The Bhashini project's Samanantar corpus would be a key resource here.

  • Code-Mixed & Conversational Data (15%): Sourced from social media (Twitter, Reddit), chat logs (anonymized), and movie scripts. This dataset is essential for the model to understand how Indians naturally communicate.

  • Instructional & QA Corpus (5%): A curated set of question-answer pairs, summarization tasks, and classification examples to teach the model how to follow instructions and perform specific downstream tasks.

  • Data Preprocessing:

  1. Cleaning: Removal of HTML tags, boilerplate text, and duplicate entries.

  2. Language Identification: Rigorous classification to ensure data is correctly labeled.

  3. Normalization: Standardizing text, especially handling variations in script and transliteration.

  4. Tokenization: Using a custom SentencePiece tokenizer trained on the entire Indic corpus to efficiently handle the morphology of Indian languages.

Metadata Metadata

Attribution 4.0 International (CC BY- 4.0)

Sector Agnostic

Alok Kumar

N.A.

Thore Network PVT LTD

Alok Kumar

N.A.

N.A.

N.A.

N.A.

N.A.

N.A.

N.A.

Open

04/10/25 05:21:22

N.A.

170.07 KB

Tags

  • Bhashini

Related Models by Organization Related Models by Organization

Project MAARG: AI-Powered Road Safety for India
An AI model that predicts real-time road accident risk by analyzing live traffic, weather, and historical data to enhance public safety.
THORE NETWORK PVT LTD
  • See Upvoters1
  • Downloads75
  • Views1,063
  • File Size7.99 KB


Related Models by Sector Related Models by Sector

spk cond tts pflow Hindi
This 150 M-parameter non-autoregressive speech generative model, developed by BharatGen, is designed for speaker-conditioned Text-to-speech in Hindi.
BHARATGEN
  • See Upvoters0
  • Downloads3
  • Views121
  • File Size3.46 GB

spk cond tts pflow Marathi
This 150 M-parameter non-autoregressive speech generative model, developed by BharatGen, is designed for speaker-conditioned Text-to-speech in Marathi.
BHARATGEN
  • See Upvoters0
  • Downloads0
  • Views498
  • File Size3.49 GB

spk cond tts pflow Telugu
This 150 M-parameter non-autoregressive speech generative model, developed by BharatGen, is designed for speaker-conditioned Text-to-speech in Telugu.
BHARATGEN
  • See Upvoters0
  • Downloads0
  • Views84
  • File Size3.38 GB

spk cond tts pflow Bengali
This 150 M-parameter non-autoregressive speech generative model, developed by BharatGen, is designed for speaker-conditioned Text-to-speech in Bengali.
BHARATGEN
  • See Upvoters0
  • Downloads1
  • Views82
  • File Size3.49 GB

spk cond tts pflow Tamil
This 150 M-parameter non-autoregressive speech generative model, developed by BharatGen, is designed for speaker-conditioned Text-to-speech in Tamil.
BHARATGEN
  • See Upvoters0
  • Downloads4
  • Views138
  • File Size3.47 GB

Tamil vb
This 150 M-parameter non-autoregressive speech generative model, developed by BharatGen, is designed for speaker-conditioned Text-to-speech in Tamil.
BHARATGEN
  • See Upvoters0
  • Downloads2
  • Views548
  • File Size3.17 GB

Marathi vb
This 150 M-parameter non-autoregressive speech generative model, developed by BharatGen, is designed for speaker-conditioned Text-to-speech in Marathi.
BHARATGEN
  • See Upvoters0
  • Downloads2
  • Views505
  • File Size3.18 GB

Hindi vb
This 150 M-parameter non-autoregressive speech generative model, developed by BharatGen, is designed for speaker-conditioned Text-to-speech in Hindi.
BHARATGEN
  • See Upvoters0
  • Downloads4
  • Views1,079
  • File Size3.18 GB

Telugu vb
This 150 M-parameter non-autoregressive speech generative model, developed by BharatGen, is designed for speaker-conditioned Text-to-speech in Telugu.
BHARATGEN
  • See Upvoters0
  • Downloads2
  • Views485
  • File Size3.17 GB

Bengali vb
This 150 M-parameter non-autoregressive speech generative model, developed by BharatGen, is designed for speaker-conditioned Text-to-speech in Bengali.
BHARATGEN
  • See Upvoters0
  • Downloads1
  • Views317
  • File Size3.17 GB


Related Models by Task Type Related Models by Task Type

AI4Bharat - IndicBART-XXEN: Multilingual to English Text Generation Model
A model based on the BART architecture, trained to translate text from various Indian languages to English.
AI4BHARAT
  • See Upvoters0
  • Downloads31
  • Views532
  • File Size0

COILD Multilingual Translation
COILD Multilingual Translation Model (COILD-Mul-MT)
DIGITAL INDIA BHASHINI DIVISION
  • See Upvoters0
  • Downloads0
  • Views36
  • File Size0