Indian Flag
Government Of India
A-
A
A+
ORGANISATION
FERMAT

FERMAT

FERMAT is a benchmark designed to test the multimodal reasoning and auto-evaluation capabilities of VLMs using real-world handwritten math problems.

About Dataset

FERMAT has 244 handwritten math solutions, carefully annotated across core mathematical domains -  🔢 Arithmetic | 📏 Algebra | 📐 Geometry | 📏 Mensuration | 🎲 Probability | 📊 Statistics | 📐 Trigonometry | 📈 Calculus.

Each solution features realistic student mistakes categorized along four key axes:

  1. 🧑‍💻 Computational Errors
  2. 🤔 Conceptual Misunderstandings
  3. ✍️ Notation Errors
  4. 📑 Presentation Issues

Additionally, some solutions contain superficial variations that don't actually affect correctness (e.g., "16 cm" vs. "16.0 cm")—perfect for testing the subtlety of your models!

Activity Overview Activity Overview

  • Downloads0
  • Redirect 12
  • File Size 0
  • Views 96

Tags Tags

  • Benchmark
  • vision-language model
  • natural language processing (NLP)
  • Document Processing
  • visual-document-understanding

License Control License Control

Attribution 4.0 International (CC BY- 4.0)