Skip to main content

> ML_BENCHMARK // SWE-BENCH-SOFTWARE-ENGINEERING_v1.0

SWE-bench (Software Engineering GitHub Issue Benchmark)

Autonomous Coding Agents · task-code-generation · active-headroom

Autonomous Coding Agentsactive-headroomhigh-vram-dependent
Official LeaderboardBack to Benchmarks

Evaluation Protocol

Agent receives an entire GitHub repository and natural language issue description, generates git patch diff executed inside Docker container.

Baseline & Metrics

Canonical Baseline:GPT-4 (naive): 1.7% | SWE-agent + Claude 3.5 Sonnet: ~49.0%
Evaluated Metrics:
Resolved Rate (% of issues passing full test regressions)

Contamination & Leakage Risks

Issues before pretraining cutoff may be partially memorized; test cases must strictly execute uncommitted tests.

Reproducibility Concerns

Execution depends on deterministic Docker build environments and exact Python dependency versions.

CONNECTED DATASETSWE-bench: Real-World GitHub Software Engineering IssuesSoftware Engineering & Autonomous Agents · 2,294 real-world GitHub issues and pull request test patches across 12 popular open-source Python repos
View Dataset Spec