AI4Bharat- Hercules Telugu LORA: Evaluation Model

About Model

Hercule is a cross-lingual evaluation model from the CIA Suite, designed to assess multilingual Large Language Models (LLMs), with a focus on Telugu. It addresses the challenge of evaluating multilingual outputs using English reference responses. Fine-tuned on the INTEL dataset, Hercule aligns closely with human evaluations and outperforms zero-shot proprietary models like GPT-4 on the RECON test set. It is particularly effective in low-resource language scenarios and supports zero-shot evaluation for unseen languages. The model employs a reference-based evaluation system, providing structured feedback with a 1-5 scoring rubric. Built on Llama-3.1-8B-Instruct, Hercule leverages lightweight fine-tuning methods like LoRA for efficient multilingual assessment.