SWE-Bench: The Gold Standard for Evaluating LLM Software Engineering
SWE-Bench tests LLMs on 2,294 real GitHub issues from popular Python repositories, evaluating whether the model can write code that passes the existing test suite - a far harder and more realistic evaluation than HumanEval.
HumanEval and MBPP test LLMs on self-contained coding problems with simple function signatures. Real software engineering involves understanding a large existing codebase, reading issue descriptions and bug reports, making targeted changes to multiple files, and passing an existing test suite you did not write. SWE-Bench (arXiv:2310.06770) by Jimenez et al. tests exactly this.
The Benchmark Construction
SWE-Bench collected 2,294 issues from 12 popular Python repositories:
Evaluation is binary: the model's patch either causes the test suite to pass (success) or not (failure). There is no partial credit and no LLM-as-judge - real pytest determines correctness. This is a significantly harder and more objective evaluation than asking a human or LLM to judge code quality.
Score Progression Across Models
Model / System
SWE-Bench Verified (%)
GPT-4 (2023 baseline)
1.7%
Claude 3.5 Sonnet (2024)
49.0%
Agentless (open-source)
50.8%
Claude 3.7 Sonnet (2025)
~70%
Top open-source agents
~55%
The jump from GPT-4's 1.7% to modern agents' 50%+ illustrates how much better AI coding agents have become in just two years.
The original 2,294 instances include some ambiguous or underspecified issues where even humans disagree on the correct fix. SWE-Bench Verified is a 500-instance subset that was validated by human contractors from Upwork - each issue was confirmed to be unambiguous, solvable, and correctly specified. This subset is now the primary leaderboard for fair comparison.
What Good Agents Do Differently
Low-performing approaches generate patches without understanding the codebase. High-performing systems (Claude 3.7, Agentless):
Read and understand the full repository structure first
Identify the specific files and functions related to the issue
Write localized targeted changes rather than large rewrites
Run the test suite locally to verify the fix before submitting
Handle file navigation, import resolution, and test interpretation
Why SWE-Bench Matters for AI Coding Tools
SWE-Bench scores correlate strongly with real-world usefulness for AI coding assistants. A model that solves 50% of SWE-Bench instances can meaningfully help with production bugs and feature requests. The benchmark has become the de facto standard for evaluating AI software engineering capability.
Practical deep-dives on LLMs, developer tools, and AI engineering. No filler. Unsubscribe any time.
// written byFIG. AUTH-01
530
Mahmudul Haque Qudrati
CEO & ML Engineer
CEO and ML Engineer at Pristren. Builds AI-powered software for teams and writes about machine learning, LLMs, developer tools, and practical AI applications.
Claude 3.5 Sonnet Review: What It Does Better Than GPT-4o (and Where It Falls Short)
An honest, benchmark-driven comparison of Claude 3.5 Sonnet vs GPT-4o covering coding, document analysis, multimodal tasks, pricing, and real-world verdict.
