AI4Bharat- Hercules Hindi LORA: Evaluation Model

About Model

Hercule is a cross-lingual evaluation model from the CIA Suite, designed to assess multilingual LLMs, with a focus on Hindi. It evaluates multilingual outputs using English reference responses and aligns closely with human judgments, outperforming zero-shot models like GPT-4 on the RECON test set. Fine-tuned on the INTEL dataset, it excels in low-resource scenarios and supports zero-shot evaluation for unseen languages. Built on Llama-3.1-8B-Instruct, it employs a structured 1-5 scoring system and supports lightweight fine-tuning with LoRA.