NE-Embed is a multilingual text embedding model for Northeast Indian languages, enabling semantic search, retrieval, and RAG across 10 languages including Khasi, Garo, Meitei, Bodo, Mizo, Assamese, Nyishi, Kokborok, Pnar, and Nagamese. Fine-tuned on LaBSE with 201,738 parallel pairs.
NE-Embed is a multilingual text embedding model purpose-built for Northeast Indian languages. General multilingual models like LaBSE achieve under 15% retrieval accuracy on languages like Garo, Meitei, Nyishi, in zero-shot settings. NE-Embed addresses this through bi-encoder fine-tuning with MultipleNegativesRankingLoss on 201,738 balanced parallel pairs across 10 languages, capped at 25,000 pairs per language to prevent high resource attractor bias, a known failure mode where languages like Assamese, Bodo dominates retrieval in mixed-language indices. NE-Embed achieves strong gains: Bodo R@1 55.8%→99.8%, Garo 13.2%→90.8%, Khasi 28.6%→95.6%, Nyishi 10.2%→75.0%. Cross-language retrieval interference (CLRI) drops from 88%+ to under 10% for most supported languages. The model is part of the NE-Stack; MWire Labs' foundational language AI infrastructure for Northeast India.
Attribution 4.0 International (CC BY- 4.0)
MWirelabs
Feature Extraction
PyTorch
Open
Sector Agnostic
27/06/26 21:03:22
0
Attribution 4.0 International (CC BY- 4.0)
© 2026 - Copyright AIKosh. All rights reserved.