What it is: SWE-bench is the leading benchmark for AI coding agents, measuring whether they can solve real GitHub issues from open-source Python projects. Three variants exist (full, Verified, Pro). Top agents score 85–89% on Verified but only ~46% on Pro — the gap that defines real-world reliability.
Who it is for: Developers comparing AI coding agents and trying to interpret benchmark numbers they see in vendor announcements.
Best if: You want a one-page explanation of what SWE-bench scores actually mean and why the Verified vs Pro gap matters.
Skip if: You don’t write code and don’t read benchmark leaderboards. Want one practical AI workflow every morning? Subscribe to our free daily newsletter.
What is SWE-bench?
SWE-bench is a benchmark that measures how well AI coding agents can solve real GitHub issues from open-source Python projects. Originally released by Princeton researchers in 2023, it has become the most-cited measure of AI coding agent capability. The benchmark gives the agent a repository state plus an issue description; the agent must produce a patch that resolves the issue and passes the project’s existing tests. Three variants exist in 2026: SWE-bench (full) with thousands of tasks; SWE-bench Verified, a curated 500-task subset with cleaner specs; and SWE-bench Pro, a harder benchmark introduced after researchers found that frontier models could reproduce verbatim training-set patches on some Verified tasks.
Why does SWE-bench matter?
SWE-bench numbers are the headline metric in every AI coding-agent announcement. As of May 2026, top agents score around 85–89% on SWE-bench Verified (GPT-5.5 at 88.7%, Claude Opus 4.7 at 87.6%). The same models drop to roughly 46% on SWE-bench Pro — a 40-point gap that captures how much benchmark numbers overstate real-world reliability. When evaluating coding agents, look at both numbers, and remember that even Pro scores come from curated tasks with clean specs and existing tests — conditions your actual codebase rarely meets.
How does SWE-bench work?
Each SWE-bench task contains: a repository at a specific commit, an issue description (usually a real GitHub issue from the project’s history), and a hidden test that the resolving patch must pass. The agent has access to the codebase and standard tools. It produces a patch; the harness runs the test; pass or fail is recorded. SWE-bench Verified additionally requires that the task’s specs and tests are sufficient and unambiguous — cleaning out tasks that were impossible to solve even for a human without insider knowledge.
The leaderboard (marc0.dev/leaderboard) is updated regularly. Each vendor reports their own scores; reproduce-yourself is encouraged.
Related terms
Learn more on Beginners in AI
Sources and further reading
Last reviewed: May 2026. AI terminology evolves quickly — verify specifics on the official source pages above.
Get Smarter About AI Every Morning
Free daily newsletter — one term, one tool, one tip. Plain English.
Free forever. Unsubscribe anytime.
You may also like
- Why AI Coding Agents Fail
- Claude Code vs Cursor vs Copilot
- Harness Engineering for Beginners
- Claude
- ChatGPT
- Codex
- AI Glossary
Two ways to go further
The AI Prompt Library
1,000+ ready-to-use prompts for Claude, ChatGPT, and Gemini. Stop staring at a blank box.
Get it for $39 →2-Hour Live AI Crash Course
A private, beginner-friendly session across Claude, ChatGPT, Gemini, and the wider landscape.
Book for $125 →