SWE-bench
Era 5 · Generative AI, reasoning, and agents (2022–2026) › Agents
- Lineage
- Acting in the world
- Kind
- Implementation
Who
Princeton University, University of Chicago
What happened
Researchers at Princeton and the University of Chicago present SWE-bench: 2,294 real programming problems drawn from GitHub issues in 12 popular Python projects. The best model, Claude 2, solves 1.96 percent.
Why it mattered
It measures the resolution of real GitHub issues and becomes the benchmark of reference for coding agents. In November 2025 its Verified variant reaches 80.9 percent.
The full story
The coding benchmarks of the day asked for short, self-contained functions. SWE-bench posed something much closer to real work: the model is given a project's full codebase and the description of an issue, and it has to change the code to resolve it. The result is checked against the tests the developers themselves wrote when they fixed it.
The 2,294 problems came from issues, and the fixes that closed them, in 12 popular Python repositories. Many required coordinated changes across several functions, classes, or files, reading lots of code, and reasoning over several steps. The best models of 2023 solved only the simplest cases: Claude 2 reached 1.96 percent.
In August 2024 OpenAI released SWE-bench Verified, a subset of 500 problems reviewed by people, and the labs cited it in their launches, such as those of Claude 3.7 Sonnet and Gemini 2.5 Pro. In November 2025 Claude Opus 4.5 reached 80.9 percent.
Success brought problems. The models had read, during training, the repositories the tasks came from, and some reproduced the original fixes from memory. On February 23, 2026, OpenAI explained that it was dropping Verified: on reviewing 138 problems its models often failed, it found flaws in the tests or problem statements of 59.4 percent of them. As a replacement it recommended Scale AI's SWE-Bench Pro, which keeps part of its tasks private.
Leads to
The time horizon of agents (March 18, 2025)
Documented direct influence
METR also measures the time horizon on SWE-bench Verified, where it doubles in under three months, and compares it with its own trend
Threads that pass through here
- From answering to acting (step 3 of 9)
Sources
Original source
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv:2310.06770) (opens in a new tab). Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan, 2023. open license: CC BY 4.0
- Claude 3.7 Sonnet and Claude Code (Anthropic, February 24, 2025) (opens in a new tab). Anthropic, 2025
Scholarship and context
- Introducing SWE-bench Verified (OpenAI, August 13, 2024) (opens in a new tab). OpenAI, 2024
- Gemini 2.5 announcement (Google blog, March 25, 2025) (opens in a new tab). Koray Kavukcuoglu, 2025
- Introducing Claude Opus 4.5 (Anthropic, November 24, 2025) (opens in a new tab). Anthropic, 2025
- Why we no longer evaluate SWE-bench Verified (OpenAI, February 23, 2026) (opens in a new tab). OpenAI, 2026
- SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? (arXiv:2509.16941) (opens in a new tab). Xiang Deng et al. (Scale AI), 2025
How to cite
“SWE-bench.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/swe-bench
This entry was last reviewed on November 13, 2026.