SWE-Bench: The Gold Standard for Evaluating LLM Software Engineering

Why HumanEval Is Not Enough

HumanEval and MBPP test LLMs on self-contained coding problems with simple function signatures. Real software engineering involves understanding a large existing codebase, reading issue descriptions and bug reports, making targeted changes to multiple files, and passing an existing test suite you did not write. SWE-Bench (arXiv:2310.06770) by Jimenez et al. tests exactly this.

The Benchmark Construction

SWE-Bench collected 2,294 issues from 12 popular Python repositories:

• pytest, sympy, matplotlib, scikit-learn, requests, astropy, flask, pylint, django, seaborn, sphinx-gallery, xarray

For each issue, the benchmark captures:

• The GitHub issue description (bug report or feature request)
• The repository state at the time of the issue (the "broken" state)
• The gold-patch fix that was merged to resolve the issue
• The failing tests before the fix and passing tests after

The task: given the issue description and repository, produce a patch that makes the failing tests pass.

Evaluation: Pass@1 With Existing Tests

Evaluation is binary: the model's patch either causes the test suite to pass (success) or not (failure). There is no partial credit and no LLM-as-judge - real pytest determines correctness. This is a significantly harder and more objective evaluation than asking a human or LLM to judge code quality.

Score Progression Across Models

The jump from GPT-4's 1.7% to modern agents' 50%+ illustrates how much better AI coding agents have become in just two years.

# Running SWE-Bench evaluation (simplified)
git clone https://github.com/princeton-nlp/SWE-bench
cd SWE-bench
pip install -e .

# Run inference with your model/agent
python run_inference.py \
    --model_name "claude-3-5-sonnet" \
    --dataset_path "princeton-nlp/SWE-bench_Verified" \
    --output_dir ./predictions/

# Evaluate predictions
python evaluation/harness.py \
    --predictions_path ./predictions/ \
    --log_dir ./logs/ \
    --testbed /tmp/testbed \
    --skip_existing

SWE-Bench Verified: A Cleaner Subset

The original 2,294 instances include some ambiguous or underspecified issues where even humans disagree on the correct fix. SWE-Bench Verified is a 500-instance subset that was validated by human contractors from Upwork - each issue was confirmed to be unambiguous, solvable, and correctly specified. This subset is now the primary leaderboard for fair comparison.

What Good Agents Do Differently

Low-performing approaches generate patches without understanding the codebase. High-performing systems (Claude 3.7, Agentless):

• Read and understand the full repository structure first
• Identify the specific files and functions related to the issue
• Write localized targeted changes rather than large rewrites
• Run the test suite locally to verify the fix before submitting
• Handle file navigation, import resolution, and test interpretation

Why SWE-Bench Matters for AI Coding Tools

SWE-Bench scores correlate strongly with real-world usefulness for AI coding assistants. A model that solves 50% of SWE-Bench instances can meaningfully help with production bugs and feature requests. The benchmark has become the de facto standard for evaluating AI software engineering capability.

Further Reading

• SWE-Bench paper arXiv:2310.06770
• SWE-Bench leaderboard
• SWE-bench GitHub