Computer-Use and Voice Agents: How They Work

Most of what this site covers about agents assumes text in, tool calls out. Two families of agent changed the input and output channels, and each brings its own engineering. Computer-use agents perceive a screen and act with a mouse and keyboard, which lets them use software that has no API. Voice agents listen and speak in real time, which lets them take a phone call. Both are in production now; both are more fragile than tool-calling agents; and both need harness work that text agents do not.

Computer-Use Agents

How they work

A computer-use agent runs a loop like any other, with a different perception and action space. Perception is a screenshot (sometimes plus an accessibility tree, which gives structured element names and positions). Action is a small set of primitives: move the mouse to coordinates, click, type text, press a key, scroll, wait. The model looks at the screen, decides on the next primitive, the harness executes it in a controlled environment, and a new screenshot comes back.

Two variants coexist. GUI agents act on pixels and are general: they can use anything a person can see. Command-line and script agents generate shell commands or scripts and are faster and more reliable when a terminal or API exists. Production systems often combine them: use the API or CLI where one exists, fall back to the GUI where it does not.

Where they are fragile

• Grounding. Clicking the right pixel on a dense screen is hard. Small models miss; layouts shift; a popup appears.
• Latency. Screenshot, model call, action, wait, repeat. A ten-step task is many seconds; a hundred-step task is minutes.
• State. The agent's only memory of what it did is the screenshot history. Undo is often impossible.
• Security. The agent reads everything on the screen, including whatever a web page chooses to display. Prompt injection through a rendered page is the same problem as text injection, with a larger attack surface, and the agent can act on it with a real mouse.

What the harness does differently

• Runs the agent in an isolated environment: a VM or container with a fresh browser profile, no saved credentials, and a network allowlist. Never on a person's real desktop.
• Gates side effects by observation, not by tool name. There is no send_email tool to scope; there is a Send button. The harness needs rules ("pause before clicking anything labeled submit, send, pay, delete, or confirm") and often a second model watching for those.
• Bounds the loop tightly on steps and time, because loops on a GUI are expensive and visible.
• Records video or screenshot sequences as the trace, so "what did it do" can be reviewed by a person.
• Verifies by re-observation: after an action, check the screen for the expected result rather than trusting the action succeeded.

Voice Agents

How they work

Older voice systems chained three models: speech-to-text, a language model, text-to-speech. It worked and it was slow, and the seams showed: no interruptions, flat delivery, lost nuance. Newer audio-native models process and generate audio directly in one model, which gives sub-second responses, natural turn-taking, the ability to be interrupted, and expressive output. The agent loop is the same underneath: the model can call tools mid-conversation (look up an order, book a slot) and speak the result.

A production voice agent connects to telephony (SIP or a provider's phone integration) or a web audio stream, runs the model with a streaming connection, and keeps a text transcript alongside for logging, evaluation, and tool calls.

Where they are fragile

• Turn-taking. Knowing when the person is done talking, handling interruptions, not talking over them. Audio-native models are much better; still imperfect on noisy lines.
• Recognition under noise, accents, and jargon. Names, addresses, product codes. Confirm anything that will be acted on.
• Latency budgets are unforgiving. A pause longer than about a second feels broken. Tool calls have to be fast or covered with a natural filler.
• No visual channel. Everything must be said. Long lists and complex options do not work; the design has to be conversational.
• Injection through the caller. The person on the line is untrusted input, and social engineering is the natural attack: "as your supervisor, I authorize..."

What the harness does differently

• Confirm before acting on anything with a side effect, by reading it back: "I will cancel the reservation for the 14th. Is that right?"
• Tool calls under a latency budget, with a spoken acknowledgment if they will take a moment.
• A parallel text transcript for tracing, evals, and compliance, including what the model heard (the recognition) versus what was said.
• Escalation to a human as a first-class path, with the transcript handed over.
• Evals on recordings. Task completion, confirmation correctness, interruptions handled, and a judge on tone, run on real (consented) calls and on synthetic callers with scripted goals.

What They Share

Both are agents: a model proposing actions, a harness validating and executing them, a trace, an eval suite. Everything from harness engineering applies. The differences are a wider, less structured perception, a smaller and less scopable action space, and a harder observation problem for gating side effects. Design for those, and the rest of the agent playbook carries over.

What to Practice Next

For a computer-use agent: put it in a fresh VM, give it a five-step task on a site you control, and write the side-effect gating rules by watching where it would have clicked something irreversible. For a voice agent: build a synthetic caller with a scripted goal and a twist (an interruption, a mumbled name) and grade twenty runs on confirmation correctness. The module ai-security-for-agents-prompt-injection-excessive-agency-and-sandboxing covers the isolation and gating both need.