Computer use in Claude
Era 5 · Generative AI, reasoning, and agents (2022–2026) › Agents
- Lineage
- Acting in the world
- Kind
- Implementation
Who
Anthropic
What happened
Anthropic introduces computer use in public beta in an upgraded version of Claude 3.5 Sonnet: the model operates a computer the way a person does, looking at the screen, moving the cursor, clicking, and typing.
Why it mattered
The first frontier model to operate a graphical interface in public beta. Any program designed for people becomes, in principle, a tool for AI.
The full story
Until then, for a model to use a program it needed an interface designed for machines, an API. Most software, however, is made for people: windows, buttons, and forms. Anthropic taught Claude to use that same interface.
The model receives screenshots and responds with actions: move the cursor to a point, click, type some text. By repeating that cycle it can complete multistep tasks in programs that were never designed for AI. Anthropic presented it as the first frontier model to offer this capability in public beta, available to developers through its API, Amazon Bedrock, and Google Cloud's Vertex AI.
The company itself warned that the feature was experimental and at times cumbersome and error-prone. In the screenshot-only category of OSWorld, Claude 3.5 Sonnet scored 14.9 percent, nearly double the next system's 7.8 percent. Companies such as Asana, Canva, Cognition, DoorDash, Replit, and The Browser Company began testing it on multistep tasks.
Three months later OpenAI presented Operator, an agent with a similar approach that works inside its own browser.
Three boxes in a row, screenshot, the model decides, and click, key, or text, joined by arrows. A dashed arrow runs from the action back to the screenshot, because the screen changes and the model looks again.
Videos
Builds on
ReAct (October 6, 2022)
Inherited, with no proven direct link
Computer use follows the ReAct loop: the model looks at a screenshot, decides on an action, and looks again at the result
Leads to
Operator, Deep Research, and Claude Code (January 23, 2025)
Independent convergence
Operator drives a browser with screenshots, mouse, and keyboard, the same approach Anthropic had unveiled three months earlier
GPT-5.4 (March 5, 2026)
Inherited, with no proven direct link
GPT-5.4 brings computer control through screenshots, mouse, and keyboard into the general model
Threads that pass through here
- From answering to acting (step 4 of 9)
Sources
Scholarship and context
Further reading
How to cite
“Computer use in Claude.” The History of AI, revised November 13, 2026. https://thehistoryofai.com/en/milestone/claude-computer-use
This entry was last reviewed on November 13, 2026.