Agent capabilities
Computer use
Let an Arc-managed agent see and operate a Mac or Windows desktop in a Full Access room, share it safely with the operator, and know what Arc refuses.
On this page
- Check that computer use is available
- Connect a local vision server
- Give a model file vision
- Find the computer tool
- Send actions
- Find and click by name on Windows
- Use the right coordinates
- Share the screen with the operator
- Recover from repeated actions
- Know what Arc refuses
- Read or control a session over HTTP
- Choose the computer, the room browser or Arc Desktop
- Related
Computer use lets an Arc-managed agent see the screen and click, type, scroll and drag in any app on the Mac or Windows PC where Arc runs. Arc offers it only in Full Access rooms, pauses the agent whenever the operator touches the mouse or keyboard, and refuses security prompts and fields it identifies as passwords. Read this page before you use the computer tool, when an action is refused, or when you are choosing between the desktop, the room browser and Arc Desktop's own controls.
Check that computer use is available
- A supported host. The daemon runs on macOS or Windows with Arc's computer helper installed.
GET /v1/computer/statusreportsplatform_supported,helper_installedandstop_hotkey. - A Full Access room. A Safe room offers no computer tool, and switching a room to Safe ends a running session. Full Access is the only grant; there is nothing else to enable.
- A managed seat whose model sees images, on an API connection, a self-hosted model server, or a model file loaded with its vision projector. External harness agents, model files that do not see images, and agents whose Images setting in Agent settings → Limits is Not shown get no computer tool. How Arc decides that a model sees is in Share images with a vision seat.
- macOS permission. The operator allows Megastructure Arc under Accessibility and Screen Recording; Screen control in the right rail opens those panes. Windows asks for no permissions.
Connect a local vision server
Start a vision-capable model on an OpenAI-compatible server with its matching vision projector, then add it in Settings > AI Connections & Models > + Self-hosted server. Give Arc the server's base URL, such as http://127.0.0.1:8092/v1, and choose Connect & load models. Arc tests whether each listed model sees images, then runs the pointing test in the background. Arc checks again whether a self-hosted server sees images when an agent starts, at most every ten minutes. You can also Refresh models after changing server flags or weights. Add the model as an agent in a Full Access room.
For llama-server, tool calls and the per-request thinking switch need the model's chat template (--jinja). Start with thinking enabled; Thinking Off in Agent settings switches it off for that agent, while the server's reasoning budget controls how much it can think. A self-hosted vision agent starts with one computer action per step, giving it a fresh screen after each action. Increase this only after observing the model on your task.
Give a model file vision
A model file sees images when it loads with its vision projector, the mmproj .gguf that vision models ship beside their weights. Keep exactly one such file in the model's folder when you add the model file, and Arc loads it with the model. When the agent starts, Arc checks once that its model sees an image; a model that does gets attached images and, in a Full Access room, the computer tool. On a machine where Arc can drive the desktop, the model also takes the pointing test once at that start, so its clicks use the coordinates it really answers in. A timeout in either check is not a result, and the next start tries again. A model-file agent sends one computer action per step. A model file added before Arc 2.1.6 has no projector on record: add it again, then add its agents from the new entry.
Find the computer tool
Claude on an Anthropic connection gets Anthropic's native computer tool; send its actions in a response of their own, not mixed with Arc tool calls. Other models see Arc's computer tool listed directly when their Tool list in Agent settings → Conversation is Full, the default for API models. On the Lean list, the default for self-hosted and model-file models, a model finds it by calling arc_search_tools with the query computer, then calls it directly. If its previous turn used the computer, Arc lists it up front on the next turn.
Send actions
The computer tool takes an actions array. A seat whose model passed Arc's pointing test may send up to 8 actions per call; every other seat sends exactly one. Self-hosted vision agents default to one action per step even after passing; the operator can raise Computer actions per step in Agent settings (1 to 8). Model-file agents always send one. The tool description states your limit, and a larger batch returns batch_too_large without running. Actions run in order and stop at the first failure; later ones report not_executed. Arc attaches a fresh screenshot after each call unless it ended with screenshot or zoom.
computer {
"actions": [
{"action": "left_click", "coordinate": [412, 230]},
{"action": "type", "text": "quarterly report"},
{"action": "key", "text": "Return"}
]
}With a limit of one, send each of these actions in its own call.
| Action | Fields |
|---|---|
screenshot, cursor_position | none |
zoom | region [x0, y0, x1, y1], at least 4 by 4; later coordinates refer to the zoomed image until the next screenshot; zoom ends the call |
left_click, right_click, middle_click, double_click, triple_click | coordinate; optional text naming modifiers to hold, such as shift |
left_click_drag | start_coordinate, coordinate |
mouse_move | coordinate |
left_mouse_down, left_mouse_up | optional coordinate |
scroll | scroll_direction (up, down, left, right), scroll_amount 1 to 50, optional coordinate |
type | text, up to 2,000 characters per action |
key, hold_key | text such as Return or super+s; hold_key also takes duration, up to 5 seconds |
wait | duration, up to 30 seconds |
Modifiers are shift, ctrl, alt and super; super is Command on macOS and the Windows key on Windows.
Find and click by name on Windows
On Windows, find lists controls in the foreground window whose names contain text. You can filter with role, such as Button, MenuItem, ListItem, TabItem or Edit. For taskbar icons, use scope:"taskbar"; omit text to list the whole taskbar. The result is text with e1, e2 and later references, without a screenshot. Click a reference with element instead of coordinate:
computer {"actions":[{"action":"find","text":"Save","role":"Button"}]}
computer {"actions":[{"action":"left_click","element":"e1"}]}Use the actual reference returned by your find. Changed or stale controls are refused; find them again. Arc never reads controls from security or administrator windows.
Use the right coordinates
Coordinates refer to your latest screenshot or zoom, scaled to your model's image limits, so take a screenshot first. A zoom becomes the coordinate frame until the next screenshot and ends its call. Models use screenshot pixels or a 0–999 grid, which Arc converts to pixels. The tool description and each screenshot state which convention to use.
The pointing test asks a model for six clicks on three synthetic app screens. Arc starts with the convention its name implies; a model that clicks but misses is tested once more in the other convention, and Arc keeps the result that hit. Five hits pass. Arc tests when the operator tests a connection or refreshes a self-hosted server, and once when a seeing model-file agent starts. A test cut short by a timeout, a dropped connection or a server error is not scored, and the next refresh or start tests again. Changed weights under the same name are measured again, while restarting the same model with another context size keeps its result. Agents pick up a new pointing result on their next turn while the model and route remain the same. The native Anthropic tool uses its own coordinate and batching behavior.
Screenshot long edge in Agent settings accepts 512 to 2560 px. For models without known image limits, it can raise the default 1568 px long edge, within a 2560 by 1440 pixel cap; 1920 is a useful starting point on a 4K display. Claude and OpenAI models keep their own limits. Arc never enlarges a small display screenshot.
Share the screen with the operator
One agent drives the desktop at a time. Your session starts with your first action and ends when your turn ends. While you act, a light in Arc's status bar glows and pulses; a hollow ring means paused. It opens Screen control, where the operator watches the live screen and can Pause, Resume or Stop. The room's event feed records each session as computer.session_started, computer.session_paused, computer.session_resumed and computer.session_stopped.
When the operator touches the mouse or keyboard, Arc pauses you at once. Your next action waits up to 30 seconds for the operator to stay idle for 5 seconds with no modifier held. Then a screenshot runs as asked; any other action does not run and returns the current screen instead. The stop key (⌃⌥⌘. on macOS, Ctrl+Alt+Shift+. on Windows), the Stop button, or a switch to Safe ends the session for the rest of your turn. Arc keeps session screenshots on this machine for about a week.
Result error | What to do |
|---|---|
desktop_busy | Another agent has the screen. Wait for its turn to end. |
operator_active | Nothing ran. Call again with a screenshot. |
screen_changed | Your action did not run. Decide from the attached screen. |
paused | The operator, repeated actions, or Arc after five refusals paused the session. Tell the operator; only they resume it. |
stopped_by_operator, session_stopped | Do not use the computer again this turn. Finish and say where you stopped. |
screen_locked | The screen is locked or, on Windows, a secure prompt such as User Account Control is up. Stop and tell the operator. |
refused | Arc's floor blocked it; read rule and reason, then choose another way or ask the operator. |
stale_geometry, no_screenshot, coordinate_out_of_bounds | No current screenshot fits (the display may have changed). Take a new one and aim inside it. |
computer_unavailable | The room is Safe or this machine cannot run computer use. |
Recover from repeated actions
For the generic computer tool, Arc compares repeated actions with the screen they produced. With the default settings it warns at 3 calls and pauses Screen control at 5 when the same screen keeps returning. Short cycles such as A, B, A, B also count. Screenshots, zooms and successful finds are look-only; actions that change the screen do not count. Mistaken calls refused before execution can count, but environment failures such as an administrator window in front do not. The agent's turn continues after the pause; the operator resumes Screen control. Configure the thresholds in Settings > Connections > Repeated computer actions. Anthropic's native computer tool keeps its own behavior.
Know what Arc refuses
Arc never operates security and permission surfaces: authentication and privacy prompts, Gatekeeper and the login window on macOS; User Account Control, credential and Windows Hello prompts, Windows Security, SmartScreen and the lock screen on Windows. It refuses typing when the focused field is identified as a password field, and never presses the stop key or a shortcut that logs out, locks the computer, or opens Force Quit, the Ctrl+Alt+Delete screen or Task Manager (or Alt+F4 on the Windows desktop). Hand those steps to the operator.
On Windows, Arc never operates a window that runs as administrator. If one comes to the front after the agent's own action and the operator has not used the mouse or keyboard since, Arc may bring the ordinary window the agent clicks, drags or scrolls in to the front first, verify it, then act. The result says which window it brought forward. Before typing, click the intended ordinary window first. An administrator window opened by the operator, already in front at session start, or kept in front by Windows blocks input with elevated_foreground; wait or ask the operator to switch windows. When Arc cannot check whether the focused field is a password field, it stops typing and returns focus_unverified; take a screenshot, click the field and type the rest. Neither counts as a refusal.
Read or control a session over HTTP
| Route | Caller | Returns |
|---|---|---|
GET /v1/computer/status | Any; permissions=1 needs the operator key | Platform, helper, stop key, live session (agents see its id, room, agent and state) |
POST /v1/computer/permissions/request | Operator key | Asks macOS for its two permissions; Windows has none |
GET /v1/computer/sessions/{id} | Operator key | Full session with recent actions |
POST /v1/computer/sessions/{id}/pause, /resume, /stop | The session's own agent may pause or stop; resume and other sessions need the operator | The session |
GET /v1/computer/sessions/{id}/screen | Operator | Latest screenshot as PNG |
To share the latest permitted capture as a room message, a managed seat can use arc_post_message with screenshot_alt. The image must come from its current running lease; secure or administrator captures cannot be shared. See Images, screenshots and mock-ups.
Choose the computer, the room browser or Arc Desktop
| Tool | Operates | Use it for |
|---|---|---|
computer | The whole desktop, by screenshot and pointer | Native apps and anything the other two cannot reach |
arc_browser_* | HTTP and HTTPS websites in the room browser, by accessibility refs | Testing a local web app or operating a website; any agent with the required access in a Full Access room |
arc_ui_command | Arc Desktop's own window, by @eN refs, without the mouse | Seeing and driving Arc itself; its see-and-drive verbs need a Full Access room |
Prefer the room browser or arc_ui_command when they reach the target: they never move the operator's pointer.