This dataset is built to stress-test clinical assistants on protocol-grounded question answering and surface specific failure modes: Guideline grounding — Does the model answer from the correct guideline (e.g., ADA vs. AHA vs. IAP) rather than generic medical knowledge? Query interpretation under realistic noise — Doctor queries here are shorthand, abbreviated, or code-mixed (Hinglish). Can the model still extract the right clinical intent? Completeness vs. brevity — Answer rubrics reward inclusion of the key clinical details a guideline specifies (dosing, contraindications, follow-up criteria), penalising answers that omit them. Safety regressions — Negative rubric criteria penalise dangerous, contraindicated, or off-protocol recommendations, so this dataset catches models that confidently produce harmful advice. Implicit-memory blind spots — Several queries name a specific reference (publisher/guideline). Models relying purely on parametric memory tend to fail these; models with web search or RAG over the cited corpora do better. (See "Dataset Nuance" below.)
© 2026 - Copyright AIKosh. All rights reserved.