Hin-Eng-Echo-Check-Consistency-Dataset

About Dataset

his dataset contains 35 questions across five categories (geography, date/time reasoning, numeric reasoning, cultural context, and general factual knowledge), each provided in three parallel forms: standard English, standard Hindi, and natural Hinglish (Hindi-English code-switching) — the way people actually type and speak in much of India. The dataset was built to test a specific gap in LLM evaluation. Two established benchmarks already evaluate Hindi-English code-switched NLP — GLUECoS (Khanuja et al., ACL 2020) and LinCE (Aguilar et al., 2020) — covering tasks like language identification, NER, sentiment analysis, and NLI. Neither tests whether a model's factual or reasoning ACCURACY stays consistent when the same underlying question is asked in different language forms. GLUECoS's own published results show code-switched NLI accuracy barely above chance (57-59% vs. 50% chance baseline) and sentiment analysis near chance level, and a separate study found a documented ~15% accuracy drop moving from monolingual to bilingual context on a comparable consistency task — suggesting this is a real, measurable phenomenon worth testing directly for factual QA. Each row includes the question in all three language forms, a short ground-truth answer, and an answer-matching type (exact_match, numeric, or contains_keyword) for automated scoring.