Claude Learns to Use a Computer: The Agent Era Begins

Anthropic made an announcement in October that I am still processing, and I want to write about it carefully because I think it marks a genuine turning point. They taught Claude to use a computer.

Not in the metaphorical sense we have been using for two years — "the model can write code that runs on a computer." In the literal sense. Claude 3.5 Sonnet, in its new configuration, can look at a screen, move a cursor, click, scroll, and type. It can navigate a browser, fill out forms, move files, operate software the way a human operator would. Anthropic released it as a public beta via their API, and the capability is called, plainly, computer use.

I have been testing it for a week. Here is what I found.

The Demonstration

The first thing I did was give Claude a task that would be trivial for a human and impossible for every AI system I have ever used: "Find my local farmer's market hours, then add this Saturday's date and the hours to my calendar."

Claude opened a browser. It navigated to a search engine. It typed the query. It read the results, clicked through to the market's website, found the hours, opened my calendar application, created an event, filled in the details, and saved it. The whole sequence took perhaps ninety seconds. I sat and watched my cursor move and my browser fill with activity that was not mine.

It was eerie. It was also unmistakably the future.

What "Computer Use" Actually Means

I want to be precise about what is happening under the hood, because the precision matters.

Claude is not running scripts. It is not calling predefined APIs. It is looking at pixels — receiving screenshots of the screen as it goes — and deciding, based on what it sees, where to move the cursor and what to click. This is agentic behaviour in the most literal sense: the model perceives an environment, reasons about a goal, takes an action, observes the result, and iterates. It is the same loop a human user runs, just executed by a language model.

This is harder than it sounds. Graphical interfaces are messy, inconsistent, and constantly changing. A button that was here yesterday may be somewhere else today. Claude handles this through perception and reasoning rather than brittle automation. When it fails — and it does fail, sometimes comically — it fails the way a person does: it misreads a label, clicks the wrong thing, recovers.

The Reliability Question

I should be honest about the current limits. Computer use is a beta, and it shows. Complex multi-step tasks succeed maybe half the time in my testing. Claude occasionally gets stuck in loops, misinterprets a loading screen, or clicks something it did not mean to. Anthropic themselves rate it around 14 percent on OSWorld, a benchmark for computer-use tasks, versus roughly 70 to 75 percent for skilled humans.

But — and this is the crucial point — the direction is clear, and the capability exists at all, which it simply did not a year ago. Every release will improve the reliability. The architecture works. What remains is refinement.

What This Unlocks

Here is where my thinking has landed.

Until now, an AI assistant could tell you things. It could draft, summarise, explain, suggest. But it could not do things. It could not complete a task in the world. Computer use changes that boundary. An AI that can perceive a screen and act on it can, in principle, execute any task a human user sitting at that computer could execute.

Now combine that with the threads I have been pulling on all year: persistence (the economics now work for an always-on assistant), memory (structured extraction makes durable recall reliable), and frontier reasoning (models are smart enough to plan multi-step actions). You can see the shape of something — an AI that stays with you, that knows you, and that can act on your behalf in the digital world. Not a chatbot. A presence that does things.

I am not there yet. But I can see the summit from here, and for the first time the path looks traversable.

Live curiously and give generously.