The History of AI

Computer use in Claude

Era 5 · Generative AI, reasoning, and agents (2022–2026) › Agents

Lineage
Acting in the world
Kind
Implementation

Who

Anthropic

What happened

Anthropic introduces computer use in public beta in an upgraded version of Claude 3.5 Sonnet: the model operates a computer the way a person does, looking at the screen, moving the cursor, clicking, and typing.

Why it mattered

The first frontier model to operate a graphical interface in public beta. Any program designed for people becomes, in principle, a tool for AI.

The full story

Until then, for a model to use a program it needed an interface designed for machines, an API. Most software, however, is made for people: windows, buttons, and forms. Anthropic taught Claude to use that same interface.

The model receives screenshots and responds with actions: move the cursor to a point, click, type some text. By repeating that cycle it can complete multistep tasks in programs that were never designed for AI. Anthropic presented it as the first frontier model to offer this capability in public beta, available to developers through its API, Amazon Bedrock, and Google Cloud's Vertex AI.

The company itself warned that the feature was experimental and at times cumbersome and error-prone. In the screenshot-only category of OSWorld, Claude 3.5 Sonnet scored 14.9 percent, nearly double the next system's 7.8 percent. Companies such as Asana, Canva, Cognition, DoorDash, Replit, and The Browser Company began testing it on multistep tasks.

Three months later OpenAI presented Operator, an agent with a similar approach that works inside its own browser.

Three boxes in a row, screenshot, the model decides, and click, key, or text, joined by arrows. A dashed arrow runs from the action back to the screenshot, because the screen changes and the model looks again.

The model sees the screen as a person does, decides on an action, carries it out, and looks again.

Videos

Builds on

  • ReAct (October 6, 2022)

    Inherited, with no proven direct link

    Computer use follows the ReAct loop: the model looks at a screenshot, decides on an action, and looks again at the result

Leads to

  • Operator, Deep Research, and Claude Code (January 23, 2025)

    Independent convergence

    Operator drives a browser with screenshots, mouse, and keyboard, the same approach Anthropic had unveiled three months earlier

  • GPT-5.4 (March 5, 2026)

    Inherited, with no proven direct link

    GPT-5.4 brings computer control through screenshots, mouse, and keyboard into the general model

Threads that pass through here

Sources

How to cite

“Computer use in Claude.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/claude-computer-use

This entry was last reviewed on November 13, 2026.

Open in the interactive timeline