CrazyGoldFish APMS Phase 2 Validation Dataset Dec 2025

About Dataset

This dataset contains the per-question and per-copy evaluation records from CrazyGoldFish Technologies' App-phase validation of AI-assisted subjective grading, conducted in December 2025 at a Government Model School in Srikakulam, Andhra Pradesh (dually designated APMS and PM Shri). The validation tested whether AI-assisted evaluation of handwritten subjective answers can be adopted safely and effectively in schools, with teachers retaining final authority. Scope: 23 copies, 801 questions, 4 teachers, 4 assessments across two subjects (Science and English). All values traceable to this annexure dataset. Confidence intervals reported in the accompanying validation report were computed via bootstrap resampling at copy level. Headline metrics derived from this dataset (as published in the validation report): - OCR extraction success 95.8 percent; question-to-answer mapping success 99.6 percent - Total-score reliability ICC(A,1) 93 percent (95 percent CI 0.805 to 0.976) - Mean Absolute Error 3.91 marks between AI and teacher totals (95 percent CI 2.61 to 5.41) - On 215 mismatch cases adjudicated against intended marking: AI closer 60.9 percent (131 cases), Teacher closer 32.6 percent (70), Both defensible 6.5 percent (14) - Adjudicated correctness on mismatches: AI 89.7 percent vs Teacher 82.8 percent (delta plus 6.9 percentage points) - Practical tolerance: 87 percent of copies within plus or minus 6 marks of teacher totals; 91.3 percent within plus or minus 10 percent of total - High-severity deviation tail 8.7 percent (95 percent CI 0.0 percent to 21.7 percent), governed via flagging, escalation, and 100 percent outlier review Sheets included: - Data_Dictionary: column definitions across all sheets - Remarks_Taxonomy: 16-category mismatch driver coding - Question_Level: 801 rows of per-question AI-vs-teacher comparisons with adjudication verdict and remark code - Copy_Level: 23 rows of per-copy aggregates - Teacher_Summary: per-teacher rollup - Batch_Summary: per-assessment rollup - Question_Pivot: per-question reliability statistics with quadratic-weighted-kappa (QWK) - Subject_Summary: Science vs English breakdown - QWK_Lookup: per-question QWK values Anonymisation protocol: All personally identifiable information has been removed prior to publication. Teacher identifiers have been replaced with codes (Teacher_A through Teacher_D). Student names have been replaced with sequential codes (Student_001 through Student_019; 19 unique students producing 23 copies across multiple assessments). Composite identifiers (CopyID, CopyKey) have been rebuilt to use the anonymised codes consistently across sheets. Free-text remark fields were scanned and any residual proper-noun references redacted. The school identifier is retained as Government Model School, Srikakulam, Andhra Pradesh, consistent with the disclosure in the MeitY IndiaAI Impact Summit 2026 Compendium (Case Study 8). A scratch working sheet was dropped from the published version. Methodology context: The validation followed a four-objective protocol: (1) validate operational feasibility of the app workflow in offline and low-bandwidth contexts; (2) validate extraction, mapping, and AI-vs-teacher reliability; (3) establish evidence on mismatch drivers and question-level hotspots; (4) define a governed end-to-end workflow with complete audit trail. Adjudication-as-ground-truth was the central methodological move: where AI and teacher scores disagreed, cases were manually adjudicated against the intended marking scheme to determine who was closer to the rubric. Recommendation from the validation report (verbatim): Proceed with scaling in Hybrid Mode with defined safeguards. Rationale: strong total-score reliability, higher AI adjudicated correctness on mismatches, and teacher authority preserved via overwrite and edit controls. Remaining risks are concentrated and controllable through upload quality gates, hotspot routing, and outlier escalation. Acknowledgments: District Collector; Principal, Government Model School; Teaching Staff, Government Model School. Companion artefacts: The validation report PDF (5 pages, December 2025) is available alongside the use case publication on AIKosh. The methodological framework, Adjudication as Ground Truth, is documented in a research paper currently under Q2 journal submission. CrazyGoldFish operates a 60-day open-data transparency protocol under which raw data, marking rubrics, and adjudication notes underpinning each published study are made available for independent review. CrazyGoldFish Technologies is incubated at IIT Mandi Catalyst.