Home › Claude Ecosystem Guides

Claude Computer Use: Operating a Desktop Through Screenshots and Actions

Computer use lets Claude operate a graphical desktop the way a person does — it looks at the screen, decides on the next action, and moves the mouse or types, repeating until the task is finished.

Last reviewed: 2026-09-04 · Українською

What computer use actually means

Most integrations give a model a clean, structured door into a system: a defined endpoint with named fields. Computer use takes the opposite approach. It hands Claude the same messy interface a human sees — a rendered screen full of buttons, menus, text fields and scrollbars — and asks it to reach a goal through that surface alone.

This matters because a large share of real software has no automation door at all. Legacy internal tools, desktop applications, admin panels behind a login, third-party sites without a public interface — a person can operate them, but a script cannot. Computer use narrows that gap: if a human could finish the task by pointing and clicking, the model can attempt the same, without anyone building a bespoke connector first.

The perception–action loop

Under the hood, computer use is a loop. Your code captures a screenshot of the desktop and sends it to Claude. The model reads the image, reasons about what it sees, and returns one concrete action — move the pointer somewhere, click, type a string, scroll, or ask for a fresh screenshot. Your code executes that action against a real or virtual machine, captures the new screen, and sends it back. The cycle repeats until the goal is met or the model reports it is done.

The key shift is that Claude never touches the operating system directly. It only proposes an action; your runtime is the hand that carries it out. That separation is what makes the pattern controllable — you decide which actions are permitted, and you can inspect or block each one before it runs.

It is built on tool use

Computer use is not a separate magic mode — it is an application of ordinary tool use, the same mechanism that lets Claude call functions you define. Here the tools are the desktop primitives: capturing the screen and performing a mouse or keyboard action. The model's turn produces a request to run one of them; your handler runs it and feeds the result back as the next input.

Because it sits on the tool-use foundation, everything you already know about agentic loops carries over: you supply the loop that keeps calling the model, you set a step budget so it cannot run forever, and you keep the task description tight so each action is purposeful. The difference from a normal tool is only the payload — an image of a screen instead of a JSON row — and the fact that the model must localise a target visually before acting.

Where it earns its place

The honest test for computer use is: does a stable programmatic path exist? If a system offers an API or a scriptable interface, use that — it is faster, cheaper and far more reliable. Computer use is for the cases where no such path exists and building one is not worth it.

Typical fits include:

• Driving legacy or internal desktop apps that were never given an API;
• Cross-application chores that hop between a browser, a spreadsheet and a form;
• Filling in web forms on sites you do not control;
• Exploratory UI testing, where a model navigates a build like a first-time user;
• One-off migrations too small to justify a custom integration.

In each of these the value is the same: the work is visual and interactive, and wiring a proper integration would cost more than it saves.

Why it needs a sandbox and a human

Handing a model control of a real desktop is powerful and risky in equal measure, so the guardrails are not optional. Run computer use inside an isolated, disposable environment — a virtual machine or container with only the applications and data the task requires. That way a wrong click, a deleted file or a runaway loop stays contained and can be thrown away.

Consequential actions deserve a human in the loop. Anything that sends money, submits an irreversible form, deletes data or changes account settings should pause for explicit confirmation rather than proceed automatically. Give the environment the least access it can succeed with, and never let it hold live credentials or payment details it does not strictly need. Treat the automation as an assistant working under supervision, not an unattended operator.

A capability still maturing

Set expectations honestly. Computer use is slower and more error-prone than a direct integration: every step costs a screenshot round-trip, and the model can misread a cluttered screen, miss a target, or get stuck when a dialog it did not expect appears. Build in recovery — retries, a step ceiling, and a clear way to detect that the loop has stalled rather than letting it spin.

There is also a security angle specific to reading a screen. Text the model sees — a banner, a document, an email open in a window — can contain instructions aimed at the model itself. Because computer use acts on what is on screen, treat that on-screen content as untrusted data, never as commands, and keep the confirmation gate in front of anything sensitive. The capability is genuinely useful today for the right jobs, but it rewards a cautious, well-fenced deployment far more than an ambitious unattended one.

Frequently asked

What is Claude computer use?

It is a capability where Claude operates a graphical desktop like a person — it receives a screenshot, decides on a mouse or keyboard action, your code executes it, and the loop repeats until the task is done. It is built on tool use.

When should I use computer use instead of an API?

Only when no stable programmatic path exists. If a system offers an API or scriptable interface, that is always faster and more reliable; computer use is for GUI-only software, legacy apps and cross-app tasks where building an integration is not worth it.

Is computer use safe to run?

Only with guardrails. Run it in an isolated sandbox with least-privilege access, require human confirmation for consequential actions like payments or deletions, and treat on-screen text as untrusted data rather than instructions to obey.

Continue in this track