What Is SWE-bench? — AI Glossary

What it is: SWE-bench is the leading benchmark for AI coding agents, measuring whether they can solve real GitHub issues from open-source Python projects. Three variants exist (full, Verified, Pro). Top agents score 85–89% on Verified but only ~46% on Pro — the gap that defines real-world reliability.
Who it is for: Developers comparing AI coding agents and trying to interpret benchmark numbers they see in vendor announcements.
Best if: You want a one-page explanation of what SWE-bench scores actually mean and why the Verified vs Pro gap matters.
Skip if: You don’t write code and don’t read benchmark leaderboards. Want one practical AI workflow every morning? Subscribe to our free daily newsletter.

What is SWE-bench?

SWE-bench is a benchmark that measures how well AI coding agents can solve real GitHub issues from open-source Python projects. Originally released by Princeton researchers in 2023, it has become the most-cited measure of AI coding agent capability. The benchmark gives the agent a repository state plus an issue description; the agent must produce a patch that resolves the issue and passes the project’s existing tests. Three variants exist in 2026: SWE-bench (full) with thousands of tasks; SWE-bench Verified, a curated 500-task subset with cleaner specs; and SWE-bench Pro, a harder benchmark introduced after researchers found that frontier models could reproduce verbatim training-set patches on some Verified tasks.

Why does SWE-bench matter?

SWE-bench numbers are the headline metric in every AI coding-agent announcement. As of May 2026, top agents score around 85–89% on SWE-bench Verified (GPT-5.5 at 88.7%, Claude Opus 4.7 at 87.6%). The same models drop to roughly 46% on SWE-bench Pro — a 40-point gap that captures how much benchmark numbers overstate real-world reliability. When evaluating coding agents, look at both numbers, and remember that even Pro scores come from curated tasks with clean specs and existing tests — conditions your actual codebase rarely meets.

How does SWE-bench work?

Each SWE-bench task contains: a repository at a specific commit, an issue description (usually a real GitHub issue from the project’s history), and a hidden test that the resolving patch must pass. The agent has access to the codebase and standard tools. It produces a patch; the harness runs the test; pass or fail is recorded. SWE-bench Verified additionally requires that the task’s specs and tests are sufficient and unambiguous — cleaning out tasks that were impossible to solve even for a human without insider knowledge.

The leaderboard (marc0.dev/leaderboard) is updated regularly. Each vendor reports their own scores; reproduce-yourself is encouraged.

Related terms

Learn more on Beginners in AI

Sources and further reading

Last reviewed: May 2026. AI terminology evolves quickly — verify specifics on the official source pages above.

Get Smarter About AI Every Morning

Free daily newsletter — one term, one tool, one tip. Plain English.

Free forever. Unsubscribe anytime.

You may also like

Two ways to go further

The AI Prompt Library

1,000+ ready-to-use prompts for Claude, ChatGPT, and Gemini. Stop staring at a blank box.

Get it for $39 →

2-Hour Live AI Crash Course

A private, beginner-friendly session across Claude, ChatGPT, Gemini, and the wider landscape.

Book for $125 →