One more buzzword is being left by the wayside in the chaotic/dreary/wonderful world of AI: Computer Use Agent (CUA). I started obsessing over them after watching OpenAI's Operator Agent demo. To me, that's the holy grail of autonomous AI - agents that can do work on your behalf while you do whatever heck humans were doing pre industrial revolution.

CUA models are a hybrid of LLMs and image models. They receive user instructions and a browser screenshot and then decide which browser action to take (click, scroll, type, etc.) to complete the task.

I want to launch CUA agents as a new sitenote.ai feature. The goal of sitenote is to 10x website conversion rates. But visitors have to do a lot of grunt work to convert. Filling out miles of forms and clicking through hordes of webpages to reach the checkout screen can take years off of ones life.

The fix? Provide a white-glove experience to customers using CUA agents.

  • Don't have time to find the next available massage slot on our site? We'll do it for you.
  • Don't have the patience to fill pages and pages of soulless forms? Don't worry about it!

So I came up with what I thought was a genius architecture over the Christmas break:

  • A user enters a task
  • A deep agent breaks it into tiny todos
  • The same deep agent hands over those tiny todos to a CUA sub-agent.
  • The CUA sub-agent finishes the tiny todo using a CUA model.
  • Step 3 and 4 repeats until task completion.

The CUA sub-agents are the real stars. Here's how they are supposed to work:

  • Get the job to be done from a deep agent.
  • Break it into sub steps.
  • Take a browser screenshot and send it and the sub step instructions to a CUA model.
  • The CUA model should come back with the next browser action that'll bring us closer to task completion.
  • Take CUA model's browser action recommendation (dropdown selection, button click, scroll etc. etc.) and convert it into executable code for the browser.
  • Once the code is executed, take a fresh screenshot and the next sub step and repeat the process recursively until you finish the job.
Illustration of how a CUA sub-agent works
Badly drawn illustration of how cua subagent worked

The above architecture mirrors how humans use computers. And that's its biggest flaw.

Let's talk about The Matrix movie for a bit (bear with me). Remember the scene when Neo started seeing the world in 0s and 1s? That was when he realized that he's in the Matrix - which is just a software - and he can change the code whatever way he wants. That enabled him to do things that weren't humanly possible before.

I had the same epiphany - but for my agent. Using the visual medium to navigate a website is such a human thing to do. But AI's capabilities aren't limited to the organs we got from evolution. Instead of looking at a page, it can read the raw JS/CSS/HTML code. Instead of dragging the cursor over a button to click it, AI can compose and execute a short JS snippet in real time to achieve the same outcome.

Armed with that insight, I rejigged the sub agent to read raw html code to ground itself on a web page. That made the agent faster and decreased token usage. But then I found an even better solution: Playwright MCP. That gave my agent standard tools to interact with a web page using code. Now, the agent is capable of doing a lot more and with a higher success rate.

Let's address the elephant in the room - how did I blow through $550 dollars on a single job application? Prompt engineering baby. Applying for a job was a perfect multi-step task to test the agent's capabilities. It has the right amount of complexity to challenge the agent but not too much to overwhelm it. So, that's what I used to optimize the agent prompt and the inference costs kept adding up.

Here's the agent in action (notice how gracefully it solves captcha): Watch on YouTube

Here's the main thing I want you to take away from this post: to use AI in transformational ways, you have to misplace your sense of up and down, of what's possible for a human and what's not. Only then can you tear through your horizon and do stuff that was impossible before.

P.S. Want to offload tasks to AI? DM me and I'll give you access to the alpha version of the CUA agent.