How do Computer-Use Agents work?

In this blog, we will learn about how computer-use agents work.

We will cover the following:

• What is a computer-use agent?
• Why do we need a computer-use agent?
• The perceive, think, act loop
• How does the agent see the screen?
• How does the agent decide what to do?
• How does the agent take actions?
• A step-by-step walkthrough with an example
• The system prompt and tools
• Safety and guardrails
• Limitations of computer-use agents
• Conclusion

I am Amit Shekhar, Founder @ Outcome School, I have taught and mentored many developers, and their efforts landed them high-paying tech jobs, helped many tech companies in solving their unique problems, and created many open-source libraries being used by top companies. I am passionate about sharing knowledge through open-source, blogs, and videos.

I teach AI and Machine Learning at Outcome School.

Let's get started.

What is a computer-use agent?

A computer-use agent is an AI program that can operate a computer the way a human does. It looks at the screen, moves the mouse, clicks buttons, and types on the keyboard.

In simple words, we give it a goal in plain language, and it uses the computer to complete that goal.

Let's decompose the name to make it clear:

Computer-Use Agent = Computer + Use + Agent

• Computer is the machine with a screen, a mouse, and a keyboard.
• Use means it actually operates that machine, just like we do.
• Agent means it acts on its own to reach a goal, step by step, without us guiding every click.

So, a computer-use agent is an AI that uses a computer on our behalf to get a task done.

Let's say we tell it, "Open the browser and book a table for two at 7 PM." The agent will find the browser icon, click it, type the website address, fill the form, and click the book button. We did not write any code for these clicks. We only gave the goal in plain words.

We have a detailed blog on AI Agent Explained that covers what an AI agent is from the ground up.

Why do we need a computer-use agent?

Before computer-use agents, automating a task on a computer was hard for us.

Let's say we wanted a program to book that table automatically. We had two old approaches, and both had problems.

Approach 1: Write a script for every step.

We write code that says click here, type this, click there. This works only if the screen never changes. The moment the website changes its design, the script breaks. The button moves, and our code clicks the wrong place.

The issue with this approach is that it is fragile and breaks easily. Let's see how the next approach solve this issue.

Approach 2: Use a special connection called an API.

An API is a direct doorway that a website or app provides for programs to talk to it. It is reliable and fast. But, here is the catch. Not every website or app gives us this doorway.

The issue with this approach is that most apps do not offer this doorway. So, we are stuck for those apps.

So, here comes the computer-use agent to the rescue.

The computer-use agent does not need a special doorway. It uses the same screen, mouse, and keyboard that we use. If a human can do the task by looking at the screen, the agent can do it too. This is the big idea, and it makes our life easy.

The perceive, think, act loop

Every computer-use agent runs on one simple cycle. We can call it the perceive, think, act loop.

In simple words, the agent keeps repeating three steps until the task is done:

• Perceive: Look at the current screen.
• Think: Decide the next single action.
• Act: Perform that action on the computer.

After acting, the screen changes. So, the agent looks again, thinks again, and acts again. The loop continues until the goal is reached. Do not worry, we will learn about each of these three steps in detail.

We can picture the flow as below:

        +-----------------------------+
        |                             |
        v                             |
   +----------+   +---------+   +----------+
   | Perceive |-->|  Think  |-->|   Act    |
   | (see the |   | (decide |   | (click,  |
   |  screen) |   |  next   |   |  type,   |
   |          |   | action) |   |  scroll) |
   +----------+   +---------+   +----------+
        ^                             |
        |     screen changes,         |
        +-----so look again ----------+

   The loop repeats until the goal is reached.

Here, we can see that the three steps form a circle. The agent moves from Perceive to Think to Act, and then the screen changes, so it goes back to Perceive and starts the next round. The loop keeps spinning until the goal is reached.

Let's understand this loop with a simple analogy.

Consider a person playing a video game. The person looks at the screen, decides to jump, and presses the jump button. The screen updates, so the person looks again and decides the next move. A computer-use agent works exactly like this person.

Let me tabulate the loop against the human player for your better understanding.

Now, we have understood the core loop. Let's understand each of these three steps in detail.

How does the agent see the screen?

The agent cannot see the screen the way we do with our eyes. So, the system takes a screenshot, which is a picture of the current screen, and sends it to the AI model.

In simple words, the screenshot is the eyes of the agent.

The AI model used here is a vision model. A vision model can look at an image and understand what is inside it. It can read the text on the screen, find the buttons, and understand the layout.

But, here is the catch. The model needs to know the exact position of things on the screen to click them.

A screen is measured in pixels, which are the tiny dots that make up the picture. Every spot on the screen has an address written as two numbers, the horizontal position and the vertical position. We write this address as (x, y).

So, when the model looks at the screenshot, it finds the button and reports its address, for example (640, 400). This address tells the computer exactly where to click.

Let's visualize this as below:

   Screen (measured in pixels)

   (0,0)                              (1280,0)
     +------------------------------------+
     |                                    |
     |          +-------------+           |
     |          |   Search    | <-- button at (640, 400)
     |          +-------------+           |
     |                                    |
     |  [browser]                         |
     |   icon at (50, 760)                |
     +------------------------------------+
   (0,800)                          (1280,800)

   x grows to the right -->
   y grows downward     v

Here, we can see that every spot on the screen has an address written as (x, y). The x value grows as we move to the right, and the y value grows as we move downward. The model reads the screenshot, finds the Search button, and reports its address as (640, 400) so the computer knows exactly where to click.

Some agents also read a hidden text list of everything on the screen, called the accessibility tree. In simple words, it is a plain list that says, "There is a button named Search at this position." This helps the agent find buttons without guessing. Many agents use both the screenshot and this list together.

This is how the agent sees and locates things on the screen.

If we want to go deep into how a single model reads an image and reasons over it in words - Multimodal AI and LLM Fundamentals - we cover it in depth in our AI and Machine Learning Program at Outcome School.

How does the agent decide what to do?

Now that the agent can see, it must decide the next action. This is the think step.

Here, a special kind of AI model comes into the picture. It is built on a Large Language Model, which is an AI trained to understand language and reason about it. We join this language ability with the vision model from the previous step. So, the same model can both see the screen and reason about the next step in words.

We give the model three things:

• The goal: what we want, in plain language.
• The current screenshot: what the screen looks like right now.
• The history: the actions already taken so far.

The model looks at all three and answers one question, "What is the single next action that moves me closer to the goal?"

The model thinks like this, "The goal is to book a table. The screen shows a browser. The browser is closed. So, my next action is to click the browser icon at (50, 760)."

The model does not decide all steps at once. It decides only the next step. This is important. Because after the action, the screen changes, and the new screenshot may be different from what the model expected. So, deciding one step at a time keeps the agent flexible.

This is how the agent thinks.

How does the agent take actions?

Now we reach the act step. The model has decided an action in words. But words alone cannot click a mouse. So, the system translates the model's decision into a real command for the computer.

The agent can perform a small set of basic actions. These are the same actions we perform every day:

• click(x, y) to click at a position.
• type("some text") to type text.
• scroll(direction) to scroll the page up or down.
• key("Enter") to press a special key.
• screenshot() to take a fresh picture of the screen.

The model outputs its choice in a clean, structured form so the system can read it easily. This form is called JSON, which is just a simple way of writing data using labels and values so a program can understand it. Let's see an example of what the model returns as below:

{
  "reasoning": "The browser is closed. I will open it first.",
  "action": "click",
  "x": 50,
  "y": 760
}

Here, we can see that the model returned three useful things. The reasoning explains why it chose this action, which helps us trust it. The action is click. The x and y are the address on the screen where the click must happen.

The system reads this and performs a real mouse click at (50, 760). The browser opens, a new screenshot is taken, and the loop continues.

This is how the agent turns a decision into a real action on the computer.

A quick note for you

No matter which tech domain you work in, get familiar with these topics:

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

We put it all together in one video:

AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, and Quantization

No need to stop reading - bookmark it and watch later when you get time. Future you will thank you.

Now, let's get back to the topic.

A step-by-step walkthrough with an example

The best way to learn this is by taking an example. Let's say our goal is, "Search for the weather in London."

Let's walk through the loop step by step.

Step 1: The system takes a screenshot. The screen shows a desktop with a browser icon. The model thinks, "I need a browser. I will click the browser icon." It returns a click action at the icon position. The system clicks. The browser opens.

Step 2: A new screenshot is taken. The model sees an empty browser with an address bar. It thinks, "I will click the address bar first." It returns a click on the address bar. The system clicks.

Step 3: A new screenshot is taken. The address bar is now active. The model thinks, "Now I will type the search." It returns a type action with the text weather in London. The system types it.

Step 4: A new screenshot is taken. The text is in the bar. The model thinks, "Now I will press Enter to search." It returns a key action for Enter. The system presses Enter.

Step 5: A new screenshot is taken. The search results show the weather of London. The model thinks, "The goal is complete." So, it returns a done action and stops.

Let me show this as a small table so we can see the loop clearly.

Here, we can notice the pattern. Look, decide one action, do it, look again. The agent never plans all five steps in advance. It reacts to each new screenshot. This is what makes it work even when the screen behaves in an unexpected way.

This way we can use the perceive, think, act loop to solve the interesting problem of operating any app.

The system prompt and tools

Now, the question is, how does the model know what actions it is allowed to take? The answer is the system prompt and the tools.

A system prompt is a set of starting instructions we give the model before the task begins. It tells the model who it is and how it must behave.

In simple words, the system prompt is the rulebook for the agent.

A simplified system prompt looks like below:

You are a computer-use agent.
You can see the screen through screenshots.
You can perform these actions: click, type, scroll, key, screenshot.
Always return one action at a time in JSON format.
Explain your reasoning before each action.
Stop when the goal is complete.

Here, we have told the model three important things. We told it what it is, a computer-use agent. We told it the actions it is allowed to use. We told it the format to reply in, which is the structured JSON we saw earlier.

The list of allowed actions is often called the tools of the agent. A tool is simply one capability the agent can use, like clicking or typing. The model picks the right tool for each step.

This is how the model learns its role and its powers before it begins.

To master the system prompt, Tool use in Agents, Agent Architecture, and the ReAct Pattern - and to build an AI Coding Agent from scratch - check out our AI and Machine Learning Program at Outcome School.

Safety and guardrails

A computer-use agent can do real things on a real computer. So, safety is very important.

Let's say the agent is filling a form and reaches a button that says delete account. We do not want the agent to click it without our permission.

So, here comes the idea of guardrails to the rescue. A guardrail is a safety rule that stops the agent from doing dangerous actions on its own.

A few common guardrails:

• Human approval: For risky actions like payments or deleting data, the agent pauses and asks us first.
• Blocked actions: Some actions, like changing system settings, are simply not allowed.
• Limited area: The agent often runs inside a safe, separate space so it cannot touch our important personal files.

These guardrails make sure the agent stays helpful and does not cause harm. We must always keep a human in control for sensitive tasks.

Limitations of computer-use agents

Now that we have learned how the agent works, it is time to understand its limits. Computer-use agents are powerful, but they are not perfect.

• They can be slow. Each step needs a screenshot and a model decision. Many steps mean many rounds, and that takes time.
• They can do mistakes. The model may misread the screen and click the wrong button.
• They can get stuck. If a screen looks confusing, the agent may keep trying the same action again and again.
• They cost more. Looking at images and reasoning at every step uses a lot of computing power.

This is why, for now, we use them mostly for tasks where a special doorway, the API, does not exist. When an API is available, we prefer it, because it is faster and more reliable.

The technology is improving very fast. The agents are getting better at seeing the screen, deciding actions, and recovering from mistakes.

Conclusion

Now we must have understood how computer-use agents work.

A computer-use agent looks at the screen, decides one action, and performs it, again and again, until the goal is reached. This is the perceive, think, act loop. It uses a vision model to see, a language model to reason, and a small set of actions like click and type to act.

The beauty of this idea is simple. If a human can do a task by looking at the screen and using the mouse and keyboard, the agent can do it too, without any special doorway into the app.

Prepare yourself for AI Engineering Interview: AI Engineering Interview Questions

That's it for now.

Thanks

Amit Shekhar
Founder @ Outcome School

You can connect with me on:

• X
• LinkedIn
• GitHub

Follow Outcome School on:

• X
• YouTube
• LinkedIn
• GitHub

Read all of our high-quality blogs here.

Subscribe to our newsletter to get our latest AI and Machine Learning blogs straight to your inbox.