> ML_BENCHMARK // HUMANEVAL-CODE-GENERATION-BENCHMARK_v1.0
HumanEval (Python Code Generation Benchmark)
Code Generation & Functional Correctness · task-code-generation · near-saturation
Code Generation & Functional Correctnessnear-saturationdeterministic-cpu
Evaluation Protocol
164 handcrafted Python programming problems evaluating docstring-to-code generation against unit test test-cases.
Baseline & Metrics
Canonical Baseline:Codex: 28.8% | Claude 3.5 Sonnet: 92.0% | DeepSeek-V3: 82.6%
Evaluated Metrics:
pass@1pass@10pass@100
Contamination & Leakage Risks
Widespread memorization across GitHub-trained language models; pass@1 may reflect verbatim memorization of test cases.
Reproducibility Concerns
Requires sandboxed execution environment to prevent arbitrary code execution on host.
CONNECTED DATASETHumanEval: Handcrafted Python Coding BenchmarkCode Generation & Functional Correctness · 164 handwritten Python programming problems with docstrings, solutions, and unit test cases
View Dataset Spec