ORGANISATION

EkaIndicMTEB

Eka-IndicMTEB, is a evaluation dataset comprising Indian Multilingual Medical Terms designed to evaluate embedding models on medical terminology across multiple Indic languages and scripts.

About Dataset

Eka-IndicMTEB, is a evaluation dataset comprising Indian Multilingual Medical Terms designed to evaluate embedding models on medical terminology across multiple Indic languages and scripts. It contains 2,532 doctor-verified queries, capturing the linguistic and domain-specific diversity of the Indian healthcare ecosystem. The dataset includes medical entities spanning symptoms, diagnoses, procedures, medications, and related concepts, enriched with real-world linguistic variations such spelling errors, special characters, abbreviations, and colloquial expressions. The dataset covers multilple languages including English, Hindi, Bengali, Tamil, Telugu, Kannada, Marathi, and Malayalam.

Purpose of Dataset

Eka-IndicMTEB addresses a critical gap in multilingual medical AI evaluation by offering: A Shared Evaluation Framework: Researchers can now benchmark multilingual medical embeddings against a standardized, clinically-validated dataset spanning multiple Indian languages. Insight into Model Strengths and Weaknesses: The benchmark systematically reveals how models handle India's linguistic diversity, identifying specific failure modes and success patterns across different language families and medical domains. Guidance for Model Development: Performance analysis across varied query types provides actionable insights for targeted model improvements. This benchmark is invaluable for researchers developing cross-lingual medical information retrieval systems, and AI teams building multilingual clinical decision support tools. Healthcare organizations deploying language-agnostic medical chatbots or semantic search systems will find this dataset essential for validating performance across India's diverse linguistic landscape. Academic institutions working on low-resource medical NLP can leverage this benchmark to identify gaps and measure progress in Indian language healthcare AI.