Indian Flag
Government Of India
A-
A
A+
/IITGN ORGANISATION
UpvoteUpvote
0
PythonSaga

PythonSaga

PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLM
4.75
AI Score infoDownload score report
  • Upvotes0upvotes
  • Downloads5downloads
  • Views167views
  • File size346.61 KB
  • Shares0Shares
  • Last updatedupdated 15 Dec 2025
UpvoteUpvote
0
COVERAGE Global
LICENSE Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)
TYPE Structured
SECTOR Science, Technology and Research
UPDATES Daily

About Dataset

Driven by the surge in code generation using large language models (LLMs), numerous benchmarks have emerged to evaluate these LLMs' capabilities. We conducted a large-scale human evaluation of HumanEval and MBPP, two popular benchmarks for Python code generation, analyzing their diversity and difficulty. Our findings unveil a critical bias towards a limited set of programming concepts, neglecting most of the other concepts entirely. Furthermore, we uncover a worrying prevalence of easy tasks that can inflate model performance estimations. To address these limitations, we propose a novel benchmark, PythonSaga, featuring 185 hand-crafted prompts in a balanced representation of 38 programming concepts across diverse difficulty levels. The robustness of our benchmark is demonstrated by the poor performance of existing Code-LLMs. The code and dataset are openly available to the NLP community at https://github.com/PythonSaga/PythonSaga.


Citation:

If you use this dataset, please cite the following work:
@inproceedings{yadav-etal-2024-pythonsaga,
    title = "{P}ython{S}aga: Redefining the Benchmark to Evaluate Code Generating {LLM}s",
    author = "Yadav, Ankit  and
      Beniwal, Himanshu  and
      Singh, Mayank",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-emnlp.996/",
    doi = "10.18653/v1/2024.findings-emnlp.996",
    pages = "17113--17126"
}

Tags

  • benchmark dataset
  • Code-LLMs

Related Datasets Related Datasets

Chat-Dataset
Chat-Dataset
4.8Information•Updated 13 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters2
  • Downloads9
  • Views79
  • File Size777.41 KB

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads13
  • Views130
  • File Size27.06 GB

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads8
  • Views75
  • File Size43.39 GB

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads10
  • Views79
  • File Size26.69 GB

SPICOR TTS 3 Sourashtra Corpus
SPICOR TTS 3 Sourashtra Corpus
-Information•Updated 1 month(s) ago
DIGITAL INDIA BHASHINI DIVISION
  • See Upvoters0
  • Downloads13
  • Views108
  • File Size0

SPICOR TTS_1.0 Indian English Corpus
SPICOR TTS_1.0 Indian English Corpus
-Information•Updated 1 month(s) ago
DIGITAL INDIA BHASHINI DIVISION
  • See Upvoters0
  • Downloads2
  • Views51
  • File Size0

SPICOR TTS_2.0 Gujarati Corpus
SPICOR TTS_2.0 Gujarati Corpus
-Information•Updated 1 month(s) ago
DIGITAL INDIA BHASHINI DIVISION
  • See Upvoters0
  • Downloads1
  • Views36
  • File Size0

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads1
  • Views43
  • File Size3.82 GB

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads1
  • Views40
  • File Size13.78 GB

SETS-THIRAL-PRINCE-EM-Protected-TI-FPGA
SETS-THIRAL-PRINCE-EM-Protected-TI-FPGA
4.8Information•Updated 1 month(s) ago
SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads3
  • Views39
  • File Size4.79 GB


Related Datasets by Organization Related Datasets by Organization

Chat-Dataset
Chat-Dataset
4.8Information•Updated 13 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters2
  • Downloads9
  • Views79
  • File Size777.41 KB

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads13
  • Views130
  • File Size27.06 GB

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads8
  • Views75
  • File Size43.39 GB

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads10
  • Views79
  • File Size26.69 GB

SPICOR TTS 3 Sourashtra Corpus
SPICOR TTS 3 Sourashtra Corpus
-Information•Updated 1 month(s) ago
DIGITAL INDIA BHASHINI DIVISION
  • See Upvoters0
  • Downloads13
  • Views108
  • File Size0

SPICOR TTS_1.0 Indian English Corpus
SPICOR TTS_1.0 Indian English Corpus
-Information•Updated 1 month(s) ago
DIGITAL INDIA BHASHINI DIVISION
  • See Upvoters0
  • Downloads2
  • Views51
  • File Size0

SPICOR TTS_2.0 Gujarati Corpus
SPICOR TTS_2.0 Gujarati Corpus
-Information•Updated 1 month(s) ago
DIGITAL INDIA BHASHINI DIVISION
  • See Upvoters0
  • Downloads1
  • Views36
  • File Size0

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads1
  • Views43
  • File Size3.82 GB

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads1
  • Views40
  • File Size13.78 GB

SETS-THIRAL-PRINCE-EM-Protected-TI-FPGA
SETS-THIRAL-PRINCE-EM-Protected-TI-FPGA
4.8Information•Updated 1 month(s) ago
SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads3
  • Views39
  • File Size4.79 GB


Related Datasets by Sector Related Datasets by Sector

Chat-Dataset
Chat-Dataset
4.8Information•Updated 13 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters2
  • Downloads9
  • Views79
  • File Size777.41 KB

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads13
  • Views130
  • File Size27.06 GB

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads8
  • Views75
  • File Size43.39 GB

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads10
  • Views79
  • File Size26.69 GB

SPICOR TTS 3 Sourashtra Corpus
SPICOR TTS 3 Sourashtra Corpus
-Information•Updated 1 month(s) ago
DIGITAL INDIA BHASHINI DIVISION
  • See Upvoters0
  • Downloads13
  • Views108
  • File Size0

SPICOR TTS_1.0 Indian English Corpus
SPICOR TTS_1.0 Indian English Corpus
-Information•Updated 1 month(s) ago
DIGITAL INDIA BHASHINI DIVISION
  • See Upvoters0
  • Downloads2
  • Views51
  • File Size0

SPICOR TTS_2.0 Gujarati Corpus
SPICOR TTS_2.0 Gujarati Corpus
-Information•Updated 1 month(s) ago
DIGITAL INDIA BHASHINI DIVISION
  • See Upvoters0
  • Downloads1
  • Views36
  • File Size0

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads1
  • Views43
  • File Size3.82 GB

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads1
  • Views40
  • File Size13.78 GB

SETS-THIRAL-PRINCE-EM-Protected-TI-FPGA
SETS-THIRAL-PRINCE-EM-Protected-TI-FPGA
4.8Information•Updated 1 month(s) ago
SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads3
  • Views39
  • File Size4.79 GB


Related Datasets by Task Type Related Datasets by Task Type

Chat-Dataset
Chat-Dataset
4.8Information•Updated 13 day(s) ago
AIKOSH OPENSOURCE COMMUNITY
  • See Upvoters2
  • Downloads9
  • Views79
  • File Size777.41 KB

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads13
  • Views130
  • File Size27.06 GB

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads8
  • Views75
  • File Size43.39 GB

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads10
  • Views79
  • File Size26.69 GB

SPICOR TTS 3 Sourashtra Corpus
SPICOR TTS 3 Sourashtra Corpus
-Information•Updated 1 month(s) ago
DIGITAL INDIA BHASHINI DIVISION
  • See Upvoters0
  • Downloads13
  • Views108
  • File Size0

SPICOR TTS_1.0 Indian English Corpus
SPICOR TTS_1.0 Indian English Corpus
-Information•Updated 1 month(s) ago
DIGITAL INDIA BHASHINI DIVISION
  • See Upvoters0
  • Downloads2
  • Views51
  • File Size0

SPICOR TTS_2.0 Gujarati Corpus
SPICOR TTS_2.0 Gujarati Corpus
-Information•Updated 1 month(s) ago
DIGITAL INDIA BHASHINI DIVISION
  • See Upvoters0
  • Downloads1
  • Views36
  • File Size0

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads1
  • Views43
  • File Size3.82 GB

SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads1
  • Views40
  • File Size13.78 GB

SETS-THIRAL-PRINCE-EM-Protected-TI-FPGA
SETS-THIRAL-PRINCE-EM-Protected-TI-FPGA
4.8Information•Updated 1 month(s) ago
SOCIETY FOR ELECTRONIC TRANSACTIONS AND SECURITY (SETS), UNDER OFFICE OF PSA TO GOVT. OF INDIA
  • See Upvoters0
  • Downloads3
  • Views39
  • File Size4.79 GB