SWE-Bench: The Gold Standard for Evaluating LLM Software Engineering

Updated
9 min read
SWE-Bench: The Gold Standard for Evaluating LLM Software Engineering
TL;DR

SWE-Bench tests LLMs on 2,294 real GitHub issues from popular Python repositories, evaluating whether the model can write code that passes the existing test suite - a far harder and more realistic evaluation than HumanEval.

Why HumanEval Is Not Enough

HumanEval and MBPP test LLMs on self-contained coding problems with simple function signatures. Real software engineering involves understanding a large existing codebase, reading issue descriptions and bug reports, making targeted changes to multiple files, and passing an existing test suite you did not write. SWE-Bench (arXiv:2310.06770) by Jimenez et al. tests exactly this.

The Benchmark Construction

SWE-Bench collected 2,294 issues from 12 popular Python repositories:

  • pytest, sympy, matplotlib, scikit-learn, requests, astropy, flask, pylint, django, seaborn, sphinx-gallery, xarray

For each issue, the benchmark captures:

  • The GitHub issue description (bug report or feature request)
  • The repository state at the time of the issue (the "broken" state)
  • The gold-patch fix that was merged to resolve the issue
  • The failing tests before the fix and passing tests after

The task: given the issue description and repository, produce a patch that makes the failing tests pass.

Evaluation: Pass@1 With Existing Tests

Evaluation is binary: the model's patch either causes the test suite to pass (success) or not (failure). There is no partial credit and no LLM-as-judge - real pytest determines correctness. This is a significantly harder and more objective evaluation than asking a human or LLM to judge code quality.

Score Progression Across Models

5 rows · Click column to sort
GPT-4 (2023 baseline)1.7%
Claude 3.5 Sonnet (2024)49.0%
Agentless (open-source)50.8%
Claude 3.7 Sonnet (2025)~70%
Top open-source agents~55%

The jump from GPT-4's 1.7% to modern agents' 50%+ illustrates how much better AI coding agents have become in just two years.

bash
# Running SWE-Bench evaluation (simplified)
git clone https://github.com/princeton-nlp/SWE-bench
cd SWE-bench
pip install -e .

# Run inference with your model/agent
python run_inference.py \
    --model_name "claude-3-5-sonnet" \
    --dataset_path "princeton-nlp/SWE-bench_Verified" \
    --output_dir ./predictions/

# Evaluate predictions
python evaluation/harness.py \
    --predictions_path ./predictions/ \
    --log_dir ./logs/ \
    --testbed /tmp/testbed \
    --skip_existing

SWE-Bench Verified: A Cleaner Subset

The original 2,294 instances include some ambiguous or underspecified issues where even humans disagree on the correct fix. SWE-Bench Verified is a 500-instance subset that was validated by human contractors from Upwork - each issue was confirmed to be unambiguous, solvable, and correctly specified. This subset is now the primary leaderboard for fair comparison.

What Good Agents Do Differently

Low-performing approaches generate patches without understanding the codebase. High-performing systems (Claude 3.7, Agentless):

  • Read and understand the full repository structure first
  • Identify the specific files and functions related to the issue
  • Write localized targeted changes rather than large rewrites
  • Run the test suite locally to verify the fix before submitting
  • Handle file navigation, import resolution, and test interpretation

Why SWE-Bench Matters for AI Coding Tools

SWE-Bench scores correlate strongly with real-world usefulness for AI coding assistants. A model that solves 50% of SWE-Bench instances can meaningfully help with production bugs and feature requests. The benchmark has become the de facto standard for evaluating AI software engineering capability.

Further Reading

#swe-bench#coding#evaluation#software-engineering#github-issues

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns, delivered to your inbox.

What should we test or explain next?

Comments

Have a question or something to add? Sign in to join the discussion.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

Software architect, full-stack engineer, and machine learning practitioner. Founder & CEO of Pristren, CTO at Venture Kick-backed PIN Sport Application GmbH (Basel, Switzerland), Software Development Lead at DailyFrame GmbH (building Dimetra patient transport software), and creator of Zlyqor. Focused on custom workflow automation, scalable web and mobile applications, and production AI/ML systems.