The History of AI

SWE-bench

Era 5 · Generative AI, reasoning, and agents (2022–2026) › Agents

Lineage
Acting in the world
Kind
Implementation

Who

Princeton University, University of Chicago

What happened

Researchers at Princeton and the University of Chicago present SWE-bench: 2,294 real programming problems drawn from GitHub issues in 12 popular Python projects. The best model, Claude 2, solves 1.96 percent.

Why it mattered

It measures the resolution of real GitHub issues and becomes the benchmark of reference for coding agents. In November 2025 its Verified variant reaches 80.9 percent.

The full story

The coding benchmarks of the day asked for short, self-contained functions. SWE-bench posed something much closer to real work: the model is given a project's full codebase and the description of an issue, and it has to change the code to resolve it. The result is checked against the tests the developers themselves wrote when they fixed it.

The 2,294 problems came from issues, and the fixes that closed them, in 12 popular Python repositories. Many required coordinated changes across several functions, classes, or files, reading lots of code, and reasoning over several steps. The best models of 2023 solved only the simplest cases: Claude 2 reached 1.96 percent.

In August 2024 OpenAI released SWE-bench Verified, a subset of 500 problems reviewed by people, and the labs cited it in their launches, such as those of Claude 3.7 Sonnet and Gemini 2.5 Pro. In November 2025 Claude Opus 4.5 reached 80.9 percent.

Success brought problems. The models had read, during training, the repositories the tasks came from, and some reproduced the original fixes from memory. On February 23, 2026, OpenAI explained that it was dropping Verified: on reviewing 138 problems its models often failed, it found flaws in the tests or problem statements of 59.4 percent of them. As a replacement it recommended Scale AI's SWE-Bench Pro, which keeps part of its tasks private.

Leads to

  • The time horizon of agents (March 18, 2025)

    Documented direct influence

    METR also measures the time horizon on SWE-bench Verified, where it doubles in under three months, and compares it with its own trend

Threads that pass through here

Sources

Original source

How to cite

“SWE-bench.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/swe-bench

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline