NE-LID Language Identification for Northeast India

NE-LID is a production-ready language identification system designed specifically for Northeast India, covering eleven languages: Assamese, Bodo, English, Garo, Hindi, Khasi, Kokborok, Meitei, Mizo, Nagamese, and Nyishi. These languages span multiple families and scripts including Latin, Devanagari, and Bengali-Assamese, making general multilingual systems unreliable in this region. The model is trained on a curated dataset of 22,000 sentences (2,000 per language) with a stratified 70/15/15 split. NE-LID achieves 99.09% overall accuracy on a balanced test set (300 samples per language), significantly outperforming existing multilingual systems such as GlotLID (73.12%), OpenLID (42.03%), IndicLID (39.30%), and LangDetect (24.33%). The system is built using fastText with character n-grams (2–5), enabling robust handling of script diversity and low-resource conditions. Transformer-based approaches were evaluated but performed poorly (9–37% accuracy), confirming the suitability of character-level modeling for this task. NE-LID runs entirely on CPU with an average inference time of 0.084 milliseconds (~12,000 predictions per second), making it suitable for real-time applications and large-scale deployment. The system serves as foundational infrastructure for multilingual AI in Northeast India, enabling reliable language routing for machine translation, ASR pipelines, chatbots, digital governance platforms, and content moderation systems.