CoRover-BharatGPT-mini

About Model

BharatGPT mini is a Transformer-based language model pretrained on a large corpus of publicly available text data using a self-supervised learning approach. This means the model was trained without any human-labelled annotations—learning directly from raw text using an automatic mechanism to generate training signals. During pretraining, BharatGPT mini was optimised for the causal language modelling task: given a sequence of tokens, the model learns to predict the next token in the sequence. More specifically, it takes a sequence of continuous text as input and is trained to predict the next word or subword by shifting the target sequence one position to the right. A masking mechanism ensures that predictions for token i are based only on tokens from positions 1 to i, without peeking at future tokens. This preserves the autoregressive nature of language modelling. Through this training process, BharatGPT mini develops a deep internal understanding of language patterns, grammar, and semantics. While it can be fine-tuned for various downstream tasks such as classification, summarisation, or question answering, it performs best in text generation tasks, which align with its original training objective.