Reading Cua Driver 0.26.1 Computer Use Internals — Pinning the Window Target and Separating Action Facts from Postconditions

I recently wrote Reading Hermes Agent v2026.9.7 Computer Use Internals and Reading Open Interpreter rust-v0.0.42 Computer Use Internals in quick succession. Both were agent-side tours, and both delegated the actual OS operations to an external cua-driver. Without following the driver, the picture of Computer Use is incomplete. This post is that follow-up: a code-reading tour of Cua Driver 0.26.1 in the trycua/cua monorepo, pinned to commit b4e3caec.

I scoped this to libs/cua-driver. The same monorepo also contains agent-side code such as the Python ComputerAgent (agent.py L902–941). Saying “cua has no LLM” would be inaccurate, so this article stays on the model-agnostic native operation layer, which is Cua Driver. All code links point at the SHA above. I did not run GUI operations, install a driver, or run Rust builds or tests; everything here is a static source read.

The short version

The interesting part of this driver is not the operation API surface. It is the design that splits “which target did I actually talk to?”, “what did I observe?”, and “what may I call success?” into separate layers.

Separately from the section layout, the six highlights that stood out most in the code:

• Three entry points (MCP, CLI, and in-process SDK) all converge on the same native dispatch
• Window-scoped operations that only carry a PID are enumerated and refused as ambiguous_window_target before any input is sent
• Element references are bound to a snapshot as an element_token; a new snapshot on the same window immediately invalidates the previous one
• The driver picks between the macOS Accessibility API, Windows UI Automation, Linux AT-SPI, pixel input, and the browser DevTools Protocol path
• An input API accepting a request is not automatically confirmed. ActionResult and verify_state are kept explicitly separate
• The background contract is generic. Whether it holds is a function of OS, application, and input route, and has to be read on each backend

The rest of the post walks through those, with pointers into the source. I keep the subject as “this version” and “what a static read could confirm”, rather than generalizing across the other cua components (the VM lifecycle in Lume or the Python ComputerAgent).

What I pinned

At the time of investigation both my local HEAD and git ls-remote origin refs/heads/main pointed at b4e3caec. The working tree had no changes; I read the files as they stood.

The workspace version in libs/cua-driver/rust/Cargo.toml is 0.26.1. The public tag cua-driver-rs-v0.26.1 points at a different SHA (cc542544), and the GitHub API reports prerelease=true with published_at=2026-09-10T12:15:53Z (Cargo.toml L19–25).

The installer scripts add another twist. Even on the stable channel, an explicit version pin wins; otherwise the baked version is preferred, and only if neither is present does the installer resolve via the API (_install-rust.sh L606–619, install.ps1 L1025–1058). So the GitHub prerelease badge and “the version the official installer would actually pick on the stable channel” are different pieces of information. If you want to reproduce an experiment, record the source SHA you read, the driver binary version and hash, and, if the daemon runs out of process, its version, launch mode, and permission mode separately.

The overall shape

The path is easier to read when split in two: how the three entry points converge on a shared native dispatch, then how the shared core drops into an OS backend.

flowchart TD
    H[Hermes computer_use wrapper] --> M[cua-driver mcp]
    O[Open Interpreter cua-driver CLI] --> C[cua-driver call]
    P[Python / TypeScript SDK caller] --> S[CuaDriver SDK]
    M --> D{Launch mode and OS}
    D -->|Standard macOS or socket set| X[MCP proxy - daemon]
    D -->|Standard Windows or Linux| R[Same-process runtime]
    C --> X
    X --> A[SdkDaemonAdapter]
    A --> S
    S --> R
    R --> T[ToolRegistry.invoke_authorized]
    T --> N[Argument normalization, authorization, session namespace]
    N --> W[Target resolution, platform tool]
    W --> OS[AX / UIA / AT-SPI / native input / CDP]
    OS --> F[ActionExecutionRecord to public contract]

The Hermes and Open Interpreter edges reflect what those earlier articles confirmed on the versions they pinned; I did not re-audit the upstream heads for this post.

Responsibilities split like this:

mcp is not always a daemon proxy

mcp_uses_direct_runtime_for() branches by launch flags and OS (main.rs L389–425).

• --direct and --socket together is an error
• --direct selects the direct runtime
• An embedded host without a private socket is refused
• Standard macOS uses the proxy, so LaunchServices and TCC identity are preserved
• Standard non-macOS paths with no socket and no history-preview condition run the direct runtime

Drawing “MCP always forwards to a daemon” as a universal path would misrepresent the Windows and Linux behavior. On the proxy side, tools/list returns the inventory obtained from the daemon, tools/call is translated into a DaemonRequest, and tool errors are distinguished from transport errors (proxy.rs L697–829).

The CLI call is service-backed

run_call() does not build an independent native runtime per invocation. It confirms a connection to the existing daemon, checks version compatibility, and then sends the request. The comments explicitly justify this by saying policy, session state, cache, and platform identity should have one application point. This is a direct continuation of the “where does state live when you go through a shell?” thread from the Open Interpreter article (cli.rs L2255–2395).

One request: from get_window_state to click

An illustrative request. The PID, window ID, token, and label are placeholders to be replaced with values observed on a live machine; this is not a log of a successful run. Pick a target with something like list_windows, then read state:

{
  "pid": 1234,
  "window_id": 5678,
  "session": "article-research",
  "include_screenshot": true
}

The tool is get_window_state. Feed the element_token from an element in the response into the next click:

{
  "pid": 1234,
  "window_id": 5678,
  "session": "article-research",
  "element_token": "s00000001:12",
  "delivery_mode": "background"
}

Understanding the shape of element_token is not an invitation to hand-craft one. The value above is a dummy. The dispatch, in outline, goes:

1. The CLI / MCP / SDK hands the request to the native runtime
2. ToolRegistry::invoke_authorized() strips reserved internal args and normalizes aliases and typed targets
3. Authorization context, policy, manifest, and expiry state are checked
4. The public session name is translated into an internal namespace
5. The wrapper / platform tool resolves the target window and element reference
6. On macOS this means an action against a cached AX element, or a window-local pixel event
7. The native result becomes an execution record, and is then projected to the public ActionResult
8. The caller reads the result and, if needed, explicitly calls verify_state or a fresh capture
9. The harness decides whether the task is done, whether to re-observe, or whether to escalate

The important thing is not to describe click as automatically triggering verify_state on success (tool.rs L1035–1158, tool.rs L1538–1630).

Refuse an ambiguous target before any input

The shared WindowTargetGuard enumerates candidates when a window-scoped tool receives only a PID (window_target.rs L104–159).

Explicit window IDs, element tokens, or desktop-scope requests take a different resolution path from the PID-only case. This is not “pick the front-most window and hope.” When there are multiple candidates the harness can re-select. Tests also verify that the internal tool is not invoked at all in the ambiguous case, so the point is not the error text but the refusal before any side effect (window_target.rs L201–265). I only confirmed the tests exist; I did not run them.

On macOS, the WindowServer’s owner is checked too

get_window_state inspects the owner PID of the requested window ID both before and after the AX tree walk. When another process owns the window in the background (an Open / Save panel, for example), the PID the caller thinks it is targeting and the window’s actual owner can differ. The check is there so a tree from a different surface does not leak into an otherwise-normal result. The motivation is spelled out in the source comments (get_window_state.rs L175–374).

element_token: a snapshot reference, not a number

An element index of 12 alone cannot promise it will still point at the same button after a re-fetch. This driver uses element_token (or snapshot_id + element_index).

• The token contains a snapshot ID and an element index, but public callers treat it as opaque
• Registry keys are runtime scope plus PID
• Registering a new snapshot for the same window invalidates the previous snapshot immediately
• Snapshots for other windows are kept, capped at 8 per runtime / PID
• Stale tokens, tokens from a different runtime generation, and clashes between token and explicit window / index are errors
• The resolver refuses a bare element_index with snapshot_id_required

The evidence is in register_snapshot_entry()’s lane.retain(...) and the branches in resolve_element_args_wide(). The top-of-file comments carry historical explanation, so I confirmed current behavior by reading the function bodies and tests (element_token.rs L142–234, element_token.rs L445–532, element_token.rs L625–870).

What a token actually guarantees is the consistency of the driver’s snapshot references. It does not automatically detect every on-screen change and invalidate itself, and the runtime / PID namespace is not per-session GUI isolation. Another call within the same runtime can invalidate an earlier token by re-fetching. A safe summary is “the driver does not silently re-map an old index onto a different element in a new tree.”

Capture: tree and image together, but not an atomic snapshot

By default get_window_state on macOS returns both a tree and a screenshot. capture_mode is deprecated; include_accessibility_tree and include_screenshot are the current toggles. A request with both false is treated as meaningless, and screenshot_out_file has its own path that forces a capture (get_window_state.rs L175–374).

The AX walk has a 20000 ms wait cap plus limits on element count and depth. The comments do call out one caveat: dropping the spawn_blocking handle does not cancel the underlying native AX call. So writing “20000 ms is the hard wall clock” would be wrong. Tree fetch, cache update, and image fetch run sequentially, so the tree and image bind to the same window but do not form an atomic snapshot at one instant.

The primary macOS window-capture path is ScreenCaptureKit. Window filters and configurations are reused briefly, and SCScreenshotManager fetches a fresh image; on failure there is a screencapture -l ... fallback. The cache is a TTL 2000 ms, capacity 32 capture plan cache, not a frame cache that hands back a stale image for 2000 ms. That distinction is easy to get wrong in performance write-ups. I did not measure latency improvements here (capture.rs L349–443, capture.rs L625–749).

Retina, resize, and input coordinates

When the returned image is downscaled, the resize ratio is recorded against the PID and window ID. Pixel clicks undo that ratio and then use the window origin and backing scale to reach screen coordinates. Conceptually:

pixel in the returned image
  -> pixel in the pre-resize window screenshot
  -> window-local point without backing scale
  -> screen point after adding the window origin

If the frame cannot be resolved, the code does not silently reinterpret coordinates as screen-absolute. The width and height of the capture are cross-checked against the window bounds for a plausible scale, and background input refuses points outside the window. If you overlay this on the Hermes article’s bounds_scale, be careful to distinguish element bounds from image pixels, so you do not describe a double-multiply where the same ratio is applied at both ends (get_window_state.rs L376–464, click.rs L800–857, px_frame.rs L1–142).

What “background” actually is, per OS

macOS: AX action and targeted event are different paths

An element-targeted click has an AX action route. perform_ax_action() handles the API’s return value, but a successful API return does not guarantee an observable UI change (ax_actions.rs L215–230).

ClickTool splits the outcome into three. When a selection readback confirms it, the result is confirmed; when the element does not advertise the action, it becomes suspected_noop; when there is no independent way to read the effect, unverifiable. Because the selection case exists, “clicks are always unverifiable” is not quite right either (click.rs L686–746).

The pixel / keyboard side pulls in a SkyLight SPI bridge that resolves symbols at runtime, along with event handling tailored for Chromium and Catalyst, and a fallback via the public post_to_pid when the SPI is unavailable. Describing the macOS backend as “just Accessibility” misses this. Whether the SPI is available, and whether the target actually accepts the event, has to be verified per OS and per app (skylight.rs L1–85, mouse.rs L438–499).

Windows: UIA, PostMessage, and injected input are separate

The Windows click path branches between semantic actions, targeted injection, and PostMessageW. The internal record for a background element click carries the transport actually chosen and any fallback used. If the effect cannot be read independently, the result is unverifiable (impl_.rs L2951–3053).

The PostMessageW mouse route and the SendInput keyboard route sit in separate functions, and each handles high-integrity targets, foreground acquisition, and the number of events actually delivered. Neither “Windows does all background input via PostMessage” nor “UIA makes foreground unnecessary” is a safe generalization (mouse.rs L84–150, keyboard.rs L408–481).

The branch worth pointing at is finish_pixel_uia_attempt(). Only when a UIA point invocation returns Miss does the code proceed to the next route; on Busy, Timeout, or Unavailable it does not send a fallback input. That is a clean example of not conflating “no response yet” with “operation did not happen”, and refusing to retry in the second case. It is not, however, an exactly-once guarantee across every tool and transport.

Windows capture is not uniform PrintWindow either. The current function refuses minimized windows first. For known XAML, WinUI, and UWP targets it tries Windows.Graphics.Capture first, falling through to screen-region BitBlt and then PrintWindow on failure. Elsewhere, when PrintWindow returns black, WGC / BitBlt fallbacks kick in. Screen-region capture can pick up overlapping windows, so occlusion information is tracked. Treating WGC and screen-region capture as one “background screenshot” bucket loses that distinction (capture.rs L404–480, capture.rs L612–674).

Linux: X11 and Wayland do not share a story

X11 window capture is MIT-SHM, persistent XGetImage, and an ImageMagick fallback (capture.rs L1–102). X11 pixel clicks use an MPX / uinput virtual pointer when available, and drop to XSendEvent under specific conditions. But some uinput_unavailable errors are surfaced as-is, so “any failure gets rescued by XSendEvent” is not a safe statement (impl_.rs L2033–2083).

Generic Wayland input does not let you send events to an unfocused specific window the same way. This version does carry a dedicated Hyprland path, though. So the right stance is not “background operation is impossible on Wayland” but “pick a compositor, plug-in, and surface, and verify.” Linux here is a supplementary check on the main branches, not an audit across every compositor (impl_.rs L1694–1716).

The split between ActionResult and verify_state

This is the piece of Cua Driver I find most interesting. “Success” is layered.

Even the proxy separates tool failure from transport failure, and a tool that did not error can still emit effect=unverifiable (action-result-contract.md L1–132).

The public action contract requires effect and route, and optionally carries delivery, evidence, and escalation.

This is a closed contract. Legacy fields such as a verified boolean, coordinates, selectors, and internal transport diagnostics are deliberately not exposed on public action results.

Do not read platform JSON as if it were public

macOS and Windows tools still build fields like verified, path, and the older escalation.recommended internally. That does not make them the public result. The shared registry runs the platform result through ActionExecutionRecord::from_legacy() and similar, then publish_action_result() projects it to the public contract and validates the typed output. Even if a tool wrote "confirmed" internally, the projection can downgrade to unverifiable if there is no trustworthy public evidence (tool.rs L1538–1630, action_record.rs L501–514).

flowchart LR
    A[OS API returns and internal diagnostics] --> B[ActionExecutionRecord]
    B --> C[Public ActionResult]
    C --> H[Harness decides]
    H -->|Specifies conditions| V[verify_state]
    V --> S[satisfied / unsatisfied / unknown]
    S --> H
    H -->|If needed| I[Fresh screenshot read by the model]

Do not conclude that “the source still mentions verified, so it isn’t retired.” The producer’s internal representation and the public contract have to be followed separately.

verify_state is a small predicate evaluator

The contract covers window existence and bounds, element selector role / label, and value / enabled / selected. It evaluates one to eight predicates with AND. The default timeout is 5000 ms and the configured upper bound is 10000 ms; stability defaults to 2 samples with an allowed range of 1 to 5. An illustrative request:

{
  "pid": 1234,
  "window_id": 5678,
  "session": "article-research",
  "expect": [
    {
      "element": {
        "selector": {"role": "AXTextField", "label_contains": "Result"},
        "value_equals": "42"
      }
    }
  ],
  "timeout_ms": 5000,
  "stable_samples": 2,
  "include_screenshot": true
}

The loop observes, evaluates the predicates, checks the run of consecutive matches, and, if needed, waits with a per-iteration cap of 100 ms. If the conditions are met once but not enough consecutive samples, the result is unknown / stability_unproven. One subtlety: the function does not wrap the entire provider.observe(...).await in a single deadline timeout. The configured “up to 10000 ms” is a polling budget, not a strict wall-clock bound that includes native observation (expectation.rs L224–334, verification.rs L115–187).

An unreliable source, insufficient observations, or multiple matches when a value is being checked all become unknown. Because the tree may not be exhaustive, element.exists=false is refused as an input; the driver refuses to prove non-existence casually. And multiple matches during a pure existence check can still hold, so “multiple matches always means unknown” is not the rule (expectation.rs L493–610).

An image requested with include_screenshot=true is attached as an additional observation after the predicate result is produced. The driver does not interpret it. A failed image fetch does not automatically turn an existing predicate result into an error either. “Image attached, therefore visually verified” is not what happens.

Escalation is advice

The public contract’s escalation.target is one of pixel, foreground, page, or session. It is the harness that re-observes and decides whether to step up. Backend-internal low-level fallbacks and harness-level next actions are different things. Turning unverifiable directly into “resend the same click” invites double-input; you need to re-observe first, and the LLM loop that enforces that is not part of Cua Driver.

Session and authorization: three layers, not one

run_call() uses a cli-explicit transport ownership namespace when the caller passes a non-default session name explicitly. Without one, it mints cli-<UUID> and sends session_end after the response. For sequential CLI use, passing the same public session label consistently is what preserves state. The shared registry does not use the public session label directly as a key for all internal state; it translates it into a runtime namespace after authorization, and refuses operations against an ended session with session_ended.

That is not “different session names give you fully independent input on the same desktop.” The screen, OS input devices, and target apps are still shared. The shared registry also refuses conflicting text mutations against the same PID; the extent to which concurrent operation actually holds depends on the tool and the platform.

Authorization is best read as three layers:

authorize_tool_call_with_context() checks hard invariants, expiry, policy, risk classification, and manifest. Placing this on the shared native registry avoids the failure mode where an MCP check is bypassed by an in-process SDK. Modes come as standard, bounded, and unrestricted. standard is the promptless default for typical automation, bounded is scoped to a reviewed manifest, and unrestricted requires an explicit bypass configured at trusted startup. None of those erase OS constraints (authorization.rs L1121–1205, authorization.rs L1207–1312).

One reason the standard macOS MCP path leans on the daemon is to preserve TCC identity. Do not conflate app-bundle, embedded-host, and direct-MCP launch modes; that matters as an implementation constraint.

Deltas from the earlier two articles and what I left unverified

The Hermes and Open Interpreter columns below reflect what those articles described on the versions they pinned; I did not re-verify upstream heads for this post.

Things worth double-checking before you carry old assumptions into this version:

• Do not map “the Hermes effect + verified” onto today’s public contract. It is now split into ActionResult and VerifyStateOutput
• Do not conflate the old escalation.recommended="px" with today’s escalation.target="pixel"
• The current resolver refuses a bare element_index. Check how the upstream adapter passes element token or snapshot ID
• Separate the historical capture_mode narrative from today’s include_* toggles
• Reflect the difference between the standard macOS MCP proxy and the Windows / Linux direct runtime

Those are visible as API deltas. The conclusion is not “Hermes v2026.9.7 is guaranteed to break against this driver”: upstream adapters may follow driver capability and fall back on their own, so any concrete combination needs its own verification.

What I want to check on a live machine next

To keep the source-level conclusions separate from runtime behavior, I plan to follow up from a host terminal rather than through an agent. Sharing the results in daily standup and team meetings so the next person does not step on the same rakes.

• With multiple windows under one PID, send a click that carries only the PID and confirm ambiguous_window_target fires before any input
• Switch snapshots from A to B, then submit A’s token, and confirm the stale refusal without any mis-resolution
• Reproduce owner-PID mismatch on an Open / Save panel and see the refusal path
• Operate a background calculator while another window is foreground, and record the foreground window, the cursor, and the target separately
• On a generic click, confirm that a successful tool call can coexist with unverifiable
• On readable UI (selection, set value), map out how confirmed evidence relates to verify_state
• Under different Retina and resize settings, confirm that the pixel picked from the image lands on the same on-screen point
• On Windows XAML with the target occluded, unoccluded, and minimized, confirm the WGC / fallback branches and the occlusion signal
• Compare sequential CLI runs with and without an explicit shared session name to see the lifecycle-state difference
• Contrive a safe test app that loses its response right after a side effect and confirm the driver does not retry
• Split Japanese IME type_text and physical key input and record how each behaves

That’s all from reading Cua Driver 0.26.1 pinned to a specific SHA and walking through the shared native dispatch, the window guard, element_token, the macOS / Windows / Linux background input paths, the split between ActionResult and verify_state, and the three-layer permission story, from the Gemba.

References

• trycua/cua GitHub repository
• trycua/cua commit b4e3caec
• Cua Driver README
• Cua Driver action result contract
• cua-driver-rs v0.26.1 release
• Best-effort background (cua docs)
• cua-driver permission modes and capability manifest
• cua-driver Platform Support
• Model Context Protocol
• macOS Accessibility API documentation
• ScreenCaptureKit documentation
• Windows UI Automation documentation
• Windows.Graphics.Capture documentation
• AT-SPI2 repository
• Wayland
• Hyprland
• uinput kernel documentation
• Chrome DevTools Protocol
• macOS TCC documentation
• Earlier Hermes post: Reading Hermes Agent v2026.9.7 Computer Use Internals
• Earlier Open Interpreter post: Reading Open Interpreter rust-v0.0.42 Computer Use Internals