ORGANISATION

HumanEval

Hand-written programming problems with unit tests, used as a standard benchmark for evaluating code-generation models.

About Dataset

HumanEval is a benchmark dataset released by OpenAI consisting of 164 hand-written Python programming problems, each accompanied by a function signature, docstring, and a set of unit tests to verify correctness. The problems are designed to test a model's ability to synthesize working code from natural-language descriptions, rather than simply completing patterns seen during training. Its hand-crafted nature helps reduce the risk of test-set contamination from web-scale pretraining data.

Purpose of Dataset

Humaneval Is Used As A Standard Benchmark For Evaluating The Functional Correctness Of Code-generation Models Via The Pass@k Metric. It Is Widely Adopted Across The Research Community To Compare The Coding Capability Of Large Language Models And To Track Progress In Program Synthesis. Researchers Also Use It To Study Prompting Strategies, Sampling Techniques, And Reasoning Approaches For Code Generation.