Reading Hermes Agent v2026.9.7 Computer Use Internals — Capture as Target Binding and Background-First Verify → Escalate

I recently wrote notes on cross-platform Computer Use agents for macOS and Windows and notes on enabling Hermes Agent Computer Use on macOS. The first compared candidates; the second was an actual setup log. While watching the agent drive Calculator in the background, I kept wanting to know what one computer_use call actually does under the hood. This is that follow-up: a code-reading tour of Hermes Agent at Git tag v2026.9.7 (commit 2237be35, product tag v0.21.1).

I scoped this to Hermes only. The trycua/cua native runtime (cua-driver) shows up here only through its official documentation; I did not audit the Rust source in full. All code links below point at the v2026.9.7 tree.

The short version

The first thing that surprised me: Hermes itself doesn’t call macOS’s Accessibility API, Windows UI Automation, or Linux AT-SPI directly. Hermes’s job is to show the model a single computer_use tool, validate and shape the model’s abstract requests, then hand them off to the external cua-driver MCP server. OS-specific window enumeration, accessibility trees, screen capture, and input delivery all live inside cua-driver.

So Hermes’s Computer Use isn’t a VLM staring at pixels and computing coordinates. It’s a hybrid stack combining AX / UIA / AT-SPI element references, window-scoped target binding, image recognition, and coordinate operations, and Hermes owns the control loop plus the translation layer between the conversation and cua-driver.

The five design forks that stood out most in the code:

• A model-agnostic single-tool schema. Rather than one tool per click or capture, everything lives under one JSON Schema with an action discriminator
• capture doubles as a sticky-target binding step. Subsequent click, type, and key calls go to the (pid, window_id) that capture selected
• Background-first with verify → escalate. Try the background path first, then escalate exactly one rung based on effect and verified
• Never auto-retry mutations on MCP timeout or transport drop. Only reads are replayed safely
• Three routes for returning screenshots (native multimodal, auxiliary vision, text-only) to keep context cost bounded

The rest of this post walks through those, with pointers into the source.

The overall shape

The control path is easier to read when split in two: what happens inside the Hermes process when computer_use is called, and what happens past the backend through transport into the native runtime.

Inside the Hermes process:

flowchart TB
    U[User] --> L[LLM / Hermes turn loop]
    L -->|computer_use JSON| T[tool executor]
    T --> R[model_tools / tools.registry]
    R --> H[handle_computer_use]
    H --> S[hard block / approval / session lock]
    S --> B[CuaDriverBackend]
    B --> V[ActionResult / CaptureResult / verdict]
    V --> X[text or multimodal tool result]
    X --> L

Past CuaDriverBackend, transport and the native runtime:

flowchart TB
    B[CuaDriverBackend] <--> A[asyncio bridge thread]
    A <--> M[MCP ClientSession over stdio]
    M <--> P[cua-driver mcp proxy]
    P <--> D[cua-driver native runtime]
    D <--> O[AX / UIA / AT-SPI / capture / input]

Responsibilities split like this:

The important thing is that the model does not see cua-driver’s many MCP tools directly. Hermes folds the external surface into one computer_use call, then internally reads driver capabilities and live input schemas from tools/list to avoid sending options an older driver doesn’t accept. Keeping the model-facing schema and system prompt stable this way is what makes the prompt cache survive across turns. The relevant files are schema.py and cua_backend_session.py.

How the tool gets exposed to the model

tools/computer_use_tool.py is a thin discovery shim that registers five things:

• Model-facing name: computer_use
• Toolset: computer_use
• Schema: COMPUTER_USE_SCHEMA
• Handler: handle_computer_use
• Availability check: check_computer_use_requirements

computer_use is part of the core tool bundle but is dropped under the coding posture. The check_fn verifies the OS and the driver binary, so on unsupported OSes or without the driver installed the definition never reaches the model. The registry’s availability check has a 30-second TTL with a last-good grace window, and it stops at “can we resolve the executable”; macOS TCC, MCP handshake, and display enumeration health are not part of that check. That’s why the tool can show up in the schema and still fail at the first real backend.start(). It’s a deliberate lazy-startup design: keep launch light, and only pay for the native runtime in sessions that actually use it.

The schema collapses capture, the various clicks, drag, scroll, type, key, set_value, wait, list, and focus into one action enum. capture has three modes (som, vision, ax). Inputs prefer element over coordinates, default to background delivery, and only specify delivery_mode="foreground" when needed.

The short operational rules live in the schema; the longer capture → act → verify workflow lives in a built-in skill. There is no dedicated system-prompt block (prompt_builder.py has the comment). Keeping both the system prompt and the tool surface stable is exactly what preserves the prompt cache.

The turn loop’s durability was another thing worth noting. When the model returns a tool call, Hermes flushes the assistant’s tool-call message to the session DB before executing any side effect. If that persistence fails, the tool doesn’t run. After execution, the tool result is appended and flushed before the next API call. That’s a generic turn-loop invariant, but it matters more here: if Hermes restarts or dies mid-desktop-action, you avoid the “we did it but there’s no record” state. From agent/tool_executor.py, the current Hermes session_id reaches model_tools.handle_function_call(), the registry looks up the handler, and eventually handle_computer_use(args, session_id=...) runs synchronously.

Following one computer_use call

Say the model returns:

{
  "action": "click",
  "element": 12,
  "capture_after": true
}

The internal path is roughly:

1. handle_computer_use normalizes the action
2. Hard blocks fire before approval: dangerous key combos and shell strings coming through type are rejected here
3. If the action mutates state, evaluate Hermes-side approval scope (foreground promotion is a separate scope from background)
4. Look up the backend for this session_id and take that session’s lock
5. The click handler passes the current sticky target and the element index to CuaDriverBackend.click()
6. The backend adds (pid, window_id) and, on drivers that advertise it, the snapshot’s element_token
7. Call MCP click or double_click
8. Normalize the cua-driver structured response into an ActionResult
9. Build a Hermes verdict from effect, verified, and escalation, not just ok
10. If capture_after=true and the transport call succeeded, re-capture the same (pid, window_id) and bundle the action result plus evidence image into one tool result
11. Emit that either as a multimodal or text tool result depending on vision capability
12. The LLM reads the verdict and the new picture, and picks: stop, re-capture, or escalate one rung

An important detail: capture_after doesn’t run on a failed action. The comment in tool.py is explicit that a screenshot right after a failure looks like a normal screenshot and would be misread as evidence of success.

capture doubles as target binding

This was the part I found most interesting. capture isn’t a side-effect-free observation that returns an image; it’s also a binding step that fixes “who receives the next click or type.”

Three modes:

Full-screen sentinels like app="screen" are special: they bypass window enumeration and return a composited desktop image. But full-screen captures are pixels only and carry no elements, so if you want to act, you have to go back to the interactive lane and specify an app or window. app="desktop" is different again: it targets the desktop shell’s own windows and icon elements.

Regular captures call list_windows and pick a target from an explicit pid / window_id, an app name, or the frontmost window. When nothing matches, it does not silently fall back to the current foreground; it returns a failure. That’s the whole point: it stops localized app name mismatches from being papered over as a misdirected click. The selected target is stored in _active_pid and _active_window_id and becomes the sticky target.

The thing I hadn’t understood before this reading: on input actions, the app= parameter is not a targeting parameter. It’s a guard that checks whether the current sticky target is consistent with what the model asserted. If they disagree, the handler returns input_target_mismatch and forces the model to run capture(app=...) or focus_app first. So the behavior “the last capture was Notes, but the model said app="Calculator" on the next type so we switched apps” simply does not happen. Retargeting has to be an explicit observe-and-select step.

Elements carry an index, a role, a label, and bounds. Internally they also carry the driver’s element_token. On subsequent clicks, if the driver advertises the capability, Hermes sends the token too, so a stale snapshot index can’t resolve to a different element than the one you meant. Transport reconnect throws the target and token cache away. This is genuinely more robust than pure coordinate automation, and the built-in skill’s core rule matches: after state changes, re-capture.

Background-first with verify → escalate

In the earlier macOS setup post I described Calculator running “in the background” in a way that reads as “Hermes always runs in the background.” Reading the code, that’s not quite right: it’s background-first, not background-only. cua-driver’s own documentation calls this best-effort background / the no-foreground contract: keep cursor and focus intact wherever AX / UIA / AT-SPI semantic actions, targeted input, and window capture allow it, and either give a structured refusal or escalate to the foreground on surfaces where they don’t.

Hermes’s ladder as read from the source:

1. Try a background action against element
2. If effect=confirmed or verified=true, stop
3. If effect=unverifiable, don’t retry; take a fresh capture and re-check
4. If it’s suspected_noop or an error with recommended="px", escalate one rung to coordinate operations on the same window
5. If recommended="foreground" or the pixel path also failed, explicitly set foreground delivery, for this one operation only
6. Only when a persistent focus change is actually required, use the separate bring_to_front action

The separation between transport-level success and semantic success is the load-bearing idea here. ActionResult.ok only means “the RPC returned”; whether the UI actually changed lives in effect and verified. The reason Hermes attaches verdict.decision to the final response is exactly to stop the model from treating RPC 200 as “the action worked.” The classification lives in tool.py around verdict construction with a comment explaining that intent.

If the model asked for foreground delivery but the connected driver’s schema doesn’t accept delivery_mode, Hermes refuses with foreground_unsupported rather than silently downgrading to background. Sending input to the wrong place is worse than not sending it at all.

MCP timeouts and at-most-once

This design was, in my view, the sharpest thing in the code, and the piece I most wanted to write down.

Hermes’s tool handler is a synchronous API, but the Python MCP SDK is async. _AsyncBridge runs an asyncio event loop on a dedicated daemon thread, and the synchronous side dispatches calls with asyncio.run_coroutine_threadsafe(). The MCP stdio context and ClientSession are opened and closed by one long-lived coroutine, so anyio’s cancel-scope invariants stay intact.

The important case is what happens on an MCP timeout. The driver may have finished the operation and only the reply got lost. If Hermes auto-retried here, you’d end up with double clicks, double text inputs, or the same string typed twice, which is a much more expensive class of failure than a normal API retry.

Instead: Hermes marks the session as suspect and rebuilds it before the next call, but does not retry the current action. On transport failure, CLI fallback and post-reconnect replay are only allowed for idempotent read tools like list_windows and get_window_state. Mutations are returned as timeout_outcome_unknown or transport_outcome_unknown. It’s a conscious at-most-once bias.

The same split shows up in CLI fallback. When get_window_state (a heavy call) hits an EAGAIN-like condition on stdio MCP, reads can be retried through cua-driver call instead, and screenshots come back through a temporary file rather than a giant base64 blob stuffed into a daemon-socket JSON payload. Mutations don’t ride that path.

How the image gets back to the model

The return trip is worth reading too. Hermes wraps the capture in a CaptureResult and recomputes the actual pixel dimensions. If the image is valid, it copies it to $HERMES_HOME/cache/images/computer_use_<uuid>.png and leaves an attachable path in the summary. It keeps up to 20 such files. If there are many elements, only the first 100 are inlined; the full tree spills to $HERMES_HOME/cache/computer_use/elements_<uuid>.json. Labels get truncated to 120 characters when inlined.

On Retina / HiDPI, the image pixels and the native element bounds can be in different coordinate spaces. Hermes compares the maximum bound to the image size and, if they disagree by enough, returns bounds_scale plus a conversion note. That’s a small but real fix for the “I read coordinates off the image and clicked them raw” class of bug.

Three return routes to the model:

• When the main model has vision and the provider accepts image parts in tool results, Hermes builds an OpenAI-style _multimodal envelope with text plus an image_url data URL, and the tool executor converts that into a provider-safe message content for the next round
• When the user has explicitly set auxiliary.vision, or the main model is non-vision, or the provider doesn’t accept multimodal tool results, or capability is unknown enough that sending the image isn’t safe, Hermes routes the screenshot through the auxiliary vision model first and passes only text to the main model. The auxiliary image is scaled so its long edge is 1456 px, and the prompt carries the scale factor for translating coordinates back
• If the auxiliary route fails, Hermes does not force the image into the main model. It degrades to a text-only result with an element list, so the loop can continue

Context cost is bounded explicitly. For Anthropic conversion, Computer Use screenshots are kept newest-first, only the last three survive as image blocks, and older ones become placeholders. The comment in anthropic_message_convert.py puts each screenshot at roughly 1,465 tokens, which makes it visible how much a long capture loop can cost against prompt cache and context budget.

Reading gateway/media_repair.py surfaced one more careful bit. When Gateway sends images to a chat surface, the model sometimes rewrites Windows paths into POSIX form. media_repair.py only rewrites MEDIA: paths whose UUID basename exactly matches a computer_use result from the same turn, back to the canonical path. It repairs explicit directives; it does not go around attaching images on its own.

Three approval layers, not one

The other thing that felt sharper after reading the code was the approval story. There are three layers, and they behave differently.

First, Hermes hard blocks. Independent of approval state, some system shortcuts and dangerous shell patterns typed through type are rejected at the handler’s entry point. Examples: lock / logout, force-delete shortcuts, curl | bash, sudo rm -rf, fork bombs. This is not a full command classifier; it’s a minimal hard block on the most obvious destructive input paths through Computer Use.

Second, Hermes approval. Mutating actions fall into Hermes-side approval scopes, with foreground promotion tracked as a separate scope from background. The CLI wires up Computer Use-specific callbacks in _install_tool_callbacks(), and modal prompt choices convert into approve_once, approve_session, always_approve, or deny. But (and this is important), _request_approval() is documented to default to allow when no callback is set, and the test suite test_computer_use_approval_isolation.py pins that behavior. Static reading only confirmed the CLI wiring; I couldn’t find the Computer Use-specific wire-up on Gateway or ACP. So “every click always shows a Hermes confirmation on every surface” isn’t something I can claim from a static read alone. It’s on my list to verify on-device or end-to-end.

Third, the cua-driver permission mode. standard / bounded / unrestricted (the permission modes doc) live in the native runtime, not in Hermes UI approval. The official docs describe standard as allowing routine click / type / scroll / focus, with the higher-risk edges handled by grants and host decisions. So the reading of “standard = a Cua confirmation for every action” isn’t right. When Hermes’s approval bypass (--yolo and the like) is on for a session, the backend permission mode also lifts to unrestricted with a warning. Since cua-driver’s mode is immutable after startup, that requires spinning up a private daemon for the session. A v3 capability manifest can still bound the surface even while bypass is on.

And beyond that there’s the OS permission layer: macOS TCC, Windows integrity / session boundaries, Linux compositor policy. Even if both Hermes and cua-driver approve, an operation the OS refuses will fail. Conversely, even when the OS grants everything, a bounded manifest can still refuse. Keeping the three layers apart matters.

Corrections to the earlier macOS setup post

Reading the code shook loose a few things I had written less precisely in the macOS setup notes:

• HERMES_CUA_DRIVER_VERSION=0.26.0 as a pinning knob doesn’t exist in v2026.9.7. The installer is written to pull the latest release deliberately, and adding that env var would only look like it pins. For a reproducible driver, point HERMES_CUA_DRIVER_CMD at a specific binary
• The Python MCP dependency numbers I quoted (mcp==1.26.0, starlette==1.0.1) reflected the venv I had at the time, not the tag. v2026.9.7’s tools/lazy_deps.py and pyproject.toml pin mcp==2.0.0, httpx2==2.7.0, starlette==1.3.1
• The line about “destructive actions require approval by default” is broader than a static read supports. Schema-wise, mutations fall into approval scopes. But the handler defaults to allow with no callback, and cua-driver’s standard mode itself allows routine click / type. Only the CLI wiring is directly confirmed
• The upstream tag I wrote as 61afcde8 may have been a local checkout or a different snapshot. Tag v2026.9.7 resolves to 2237be35. I’ll write both the tag name and the full commit hash going forward
• The built-in skill has a phrasing that reads as if passing app= on an input action auto-targets. The implementation is the opposite: input goes to the sticky target and app= is a mismatch guard. When the skill wording and the implementation disagree, trust the implementation and phrase it as “run capture or focus_app before changing app”

I’ll roll these into either a doc-fix PR upstream or a short addendum article, probably the addendum article route.

What I want to verify on real hardware next

To stop confusing source-read conclusions with runtime behavior, my next round will run on a real machine from a host terminal, not through an agent shell. I’ll share results in a daily standup or team meeting so the next person doesn’t step on the same rakes.

• Record git rev-parse v2026.9.7^{commit} and cua-driver --version together
• Pin the driver binary under test through HERMES_CUA_DRIVER_CMD and record its hash
• Compare the approval UX for background click and foreground escalation across CLI, TUI / Desktop, and Gateway surfaces (Telegram, Slack) separately
• Compare standard and bounded for the actual process tree, socket path, and TCC attribution
• Force an MCP timeout on a safe test app and confirm from logs that no mutation is retried
• Save real examples of effect, verified, and escalation after an element-index action
• Confirm that the sticky target holds when there are multiple windows with the same title
• Separate set_value from type_text for a Japanese IME composition test
• Compare the tool result shape across the three vision routes (native, auxiliary, text-only)
• Trigger a MEDIA: attachment from Gateway and observe Windows-path repair behavior

That’s all from reading Hermes Agent v2026.9.7 Computer Use at the tag, walking through the single-tool schema and capture doubling as target binding, background-first with verify → escalate, at-most-once on MCP timeout, three routes for returning images, and the three-layer approval model, from the Gemba.

References

• Hermes Agent official site
• NousResearch/hermes-agent GitHub repository
• Hermes Agent v2026.9.7 source tree
• Hermes Agent commit 2237be35
• Hermes Agent Computer Use documentation
• trycua/cua GitHub repository
• cua-driver-rs v0.26.0 release
• Best-effort background (cua docs)
• cua-driver permission modes and capability manifest
• cua-driver Platform Support
• Model Context Protocol official site
• macOS Accessibility API documentation
• Windows UI Automation documentation
• AT-SPI2 repository
• macOS TCC documentation
• Anthropic recommended image sizes for vision
• Earlier selection notes: Cross-platform Computer Use agents
• Earlier macOS enablement log: Hermes Agent Computer Use on macOS