Reading Open Interpreter rust-v0.0.42 Computer Use Internals — From the QA Skill to cua-driver CLI

I recently wrote Reading Hermes Agent v2026.9.7 Computer Use Internals. While writing it, I kept thinking back to a candidate I had listed in notes on cross-platform Computer Use agents for macOS and Windows: Open Interpreter. Its README documents trycua/cua usage, so if the driver is shared, how close is the agent-side implementation? This post is that follow-up, a code-reading tour of openinterpreter/openinterpreter pinned to Git tag rust-v0.0.42 (commit 67299aeb).

I scoped this to Open Interpreter itself. The trycua side comes in only as a supplementary reference, pinned separately to commit b4e3caec. That Cua commit is not locked as an Open Interpreter dependency; it is a snapshot I read to explain what the QA skill actually delegates to. All code links below point at the rust-v0.0.42 tree and the Cua commit above. I did not run GUI operations, install a driver, or run Rust tests; everything here is a static source read.

The short version

Open Interpreter does use trycua. But the standard QA path in this tag is “the bundled qa-testing skill makes the model drive the cua-driver CLI.” Both the README’s Computer Use section (README.md L99–103) and the skill itself (SKILL.md L1–101) confirm this. Calling it “the same integration style” as the dedicated computer_use handler I followed in Hermes would be inaccurate, so I keep the two apart.

The six design forks that stood out most in the code:

• What ships is not the driver itself but a QA procedure. Markdown is embedded in the Rust binary and expanded into a skills cache
• The skill points models at agent-browser for the web and cua-driver for native apps, and tells them to read the specifications from those tools directly
• “Observe, act, re-observe” is a model-facing convention written in the skill, not a QA-specific Rust path that auto-verifies after every click
• Execution rides the general command-running infrastructure. Command approval and sandboxing, Cua’s authorization, and OS permissions have to be considered separately
• Images do not reach the model directly. You need to combine Cua’s screenshot output path with a generic image input path such as view_image
• Pinning the Open Interpreter tag does not pin the experiment. If the driver is on PATH it is reused; if not, the skill fetches an installer from the moving main of trycua/cua

The rest of the post walks through those, with pointers into the source. Throughout, I try to keep the subject as “the standard QA path,” “this tag,” and “what I could confirm from a static read,” rather than generalizing to configurations where a user has registered a separate MCP server, alternative plug-ins, or the older Python OS mode.

The overall shape

The path is clearer when split in two: the skill-discovery lane and the actual PC-driving lane.

flowchart TD
    A[QA skill embedded in the OI binary] --> B[Expanded into skills/.system]
    B --> C[Skill catalog and body shown to the model]
    U[User's app-validation request] --> L[Chosen model and harness]
    C --> L
    L --> E[Generic command execution and approval]
    E --> W[agent-browser CLI]
    E --> D[cua-driver CLI]
    W --> WB[Web browser]
    D --> S[Cua daemon / native runtime]
    S --> O[Native apps]
    D --> R[Text, structured result, image files]
    R --> L
    R -.follow-up call to read the image.-> I[view_image, etc.]
    I --> L

The cua-driver CLI → daemon edge is what I confirmed on the supplementary Cua commit (Cua cli.rs L2255–2326). view_image is one example of a generic image-input path that exists in Open Interpreter (view_image.rs L92–211); the QA skill does not always dispatch it. Tool names and available capabilities also change with the chosen harness and runtime, so I did not draw the diagram as “every model always calls exec_command.”

Responsibilities split like this:

Present in samples, but actually shipped in the binary

The entry point is codex-rs/skills/src/assets/samples/qa-testing/SKILL.md. The path makes it look like a sample that would not be part of the distribution, but SYSTEM_SKILLS_DIR in skills/src/lib.rs uses include_dir! to embed all of src/assets/samples (lib.rs L55–100).

install_system_skills() does the following:

1. Prepare the skills directory under the user’s home
2. Compare the fingerprint recorded at skills/.system against the embedded content
3. Do nothing on a match
4. On a mismatch, delete .system and re-expand the embedded content
5. Write the new fingerprint into the marker

Note the word “install” here means expanding the skill files, not installing cua-driver. Cua adoption is a separate step the model performs later by following the skill’s body (SKILL.md L61–78).

HostSkillsService::ensure_system_skills_installed() triggers the above, and when the skill root is constructed the system cache is registered as SkillScope::System (host_service.rs L459–463 and host_roots.rs L93–111).

The path names deserve care too. Internal variables and comments still say codex_home / CODEX_HOME, but the Open Interpreter product’s home defaults to ~/.openinterpreter, its override is INTERPRETER_HOME, and it deliberately ignores CODEX_HOME (lib.rs L8–35). So the expansion target lands here:

~/.openinterpreter/skills/.system/qa-testing/SKILL.md

The test refreshes_only_the_oix_owned_system_skills_namespace confirms that a cache with a stale marker is refreshed, that the QA skill is restored to the embedded content, that removed system skills are cleaned up, and that user-owned skills outside the system namespace are preserved. That is evidence that the skill does ship, but it is not an end-to-end test for actual PC operation (lib.rs L254–305).

No bespoke action schema; read the external CLI’s spec instead

The QA skill itself is extremely short:

• Check network reachability first
• For the web, install agent-browser and read agent-browser skills get core
• For native apps, install cua-driver and read cua-driver list-tools
• Compare state before and after every action
• Confirm risky operations (purchase, message send, form submission, deletion) explicitly

The web side documents platform-specific binaries for macOS and Linux and an npm install path for Windows (SKILL.md L25–59). The native side documents a shell installer for macOS and Linux and a PowerShell installer for Windows (SKILL.md L61–78; verification steps at SKILL.md L80–101).

The presence check for Cua is command -v on macOS and Linux, Get-Command on Windows. If the binary exists, it is used as is; Open Interpreter’s skill does not verify a version pin or compatibility. When not present, the install target is trycua/cua/main/libs/cua-driver/scripts/install.*, which is not pinned to a specific Cua release corresponding to Open Interpreter’s tag.

The upside of this design is that Open Interpreter avoids growing a huge tool schema every time the external CLI surface changes. The downside is that the model has to read the CLI’s own help, build the arguments, and interpret status and errors. This is a design observation from source, not an empirical claim that Open Interpreter is better or worse than Hermes.

list-tools succeeding does not mean the GUI path is ready

On the supplementary Cua source, main.rs shows that ListTools renders its output from inspect_tools_without_runtime() (Cua main.rs L507–540). By contrast, Call enters run_call(), checks whether the daemon is responding, and exits if it is not (Cua cli.rs L2255–2326). So the skill’s list-tools step is a good spec-discovery step, not a health check for daemon startup, TCC grants, or window enumeration.

The OI skill also does not spell out Cua’s doctor or daemon-startup procedures. Whether an actual operation succeeds after installation depends on the driver’s install state and the runtime environment, and needs verification on real hardware.

How the harness passes the skill to the model

General skill discovery lives in ext/skills/src/catalog_prompt.rs. It shows the model each skill’s name, description, and location, and instructs a staged disclosure where the body is read only for matching tasks (catalog_prompt.rs L3–40). Showing the model where a file is is not the same as reading its body into the conversation.

A concrete implementation is easier to read in the Kimi Code harness’s kimi_code_skill.rs.

1. Parse the skill name and optional args from the call
2. If it is not a built-in special skill, resolve the name against the current skill snapshot
3. Read the body via read_skill_text()
4. Strip frontmatter and expand arguments
5. Wrap the text in <kimi-skill-loaded ...> and add it to the conversation as a user message
6. Return the tool result saying the body has been loaded

This is a Kimi Code example, not a harness-wide contract. Even here there is no Cua-specific handling; the model that has just read the QA skill decides what to run next (kimi_code_skill.rs L19–108).

Following one native QA turn

Suppose the request is “increment the counter in this test app by one and confirm the display changed.” The flow implied by the skill (from the source, not from a recorded run) is roughly this:

1. The model reads the qa-testing catalog entry and body
2. Following the skill, it verifies network and CLI availability
3. If needed, it installs the CLI through the normal command-approval path
4. It learns available operations and specs from cua-driver list-tools, and so on
5. It captures the initial state of the target app and window through the CLI
6. It picks an element or coordinate and constructs an action command
7. The generic command runtime handles the command
8. Cua’s runtime performs the operation and returns a result
9. The model re-reads the state and checks whether the counter changed
10. If needed, it reads a screenshot back and reports the confirmed outcome to the user

For the generic execution route in particular, the unified exec handler resolves approval and then calls manager.exec_command(). The process manager sets up the approval requirement for the shell command and pipes it through UnifiedExecRuntime and ToolOrchestrator::run(). This is a general command-execution path, not a GUI-specific click-arbiter (exec_command.rs L304–430 and process_manager.rs L1369–1439).

On the supplementary Cua source, run_call() sends a DaemonRequest with method="call", the tool name, and arguments to the daemon. Open Interpreter reaching Cua through a CLI, and Cua using a daemon internally, are compatible facts. The wrong reading is “since it is CLI, each process talks to the OS APIs directly” (Cua cli.rs L2255–2326).

Session persistence is decided at the CLI boundary too

run_call() on the Cua source branches on whether an explicit non-default session was passed. Anonymous one-shot calls get a temporary transport session that is closed with session_end after the response. Explicit session labels use a separate ownership namespace.

The OI QA skill does not prescribe how those sessions are attached. That means “the target or element reference from the previous capture is preserved into the next CLI call” is not guaranteed by the skill alone. The lifetimes of target, session, and element depend on the Cua version in use and the exact arguments the model produced. This is a good candidate for on-hardware verification when comparing against Hermes.

Verification is documented; where it is enforced differs

The QA skill does not treat a successful click return as completion. It requires re-observation and comparison of visible text, counters, statuses, selection state, input values, and images (SKILL.md L80–101).

Two things worth separating:

The QA path I found in Open Interpreter is the former. I did not find a Cua-specific adapter equivalent to the latter. Even if Cua itself returns some verification metadata, I cannot claim that Open Interpreter normalizes it into a bespoke verdict.

Similarly, the OI skill does not describe a background-first escalation ladder in detail, nor a “do not retry mutations on timeout” rule. Rather than sweeping “Open Interpreter does not retry,” the accurate wording is that this integration does not surface a Computer Use-specific retry guarantee that I could confirm.

Getting the CLI’s image back to the model needs a separate bridge

The OI QA skill says a screenshot can be used as evidence but does not define a fixed save location, a maximum retained count, or automatic fallback to an auxiliary vision model (SKILL.md L80–101).

On the supplementary Cua CLI, when --screenshot-out-file is present the image content is decoded and saved to that file. When it is not, the image can be inlined into the structured JSON as screenshot_png_b64, and similar fields (Cua cli.rs L2327–2388).

The subtle bit here is that base64 showing up in shell output is not the same as an image being delivered to the model. One usable path on the OI side is to open the saved file with view_image.

ViewImageHandler does the following:

• Check the model’s input modality for image support and error out on unsupported models
• Resolve the path in the chosen execution environment and read the file with the sandbox context
• Verify the bytes decode as an image
• Build an output containing an image data URL and return it as InputImage in the tool result

(view_image.rs L92–211 and view_image.rs L227–250)

The path exists in the source, but the QA skill does not guarantee view_image is always called. I also could not find a Cua-specific rescue path that automatically routes non-vision models through an auxiliary VLM within this scope. The README’s “any model” wording is best read as “your choice of model,” not “any model will reliably understand screenshots” (README.md L99–103).

Reading macOS / Windows and approval separately

The standard QA skill carries install examples for both macOS / Linux and Windows, and even documents that && is unavailable in Windows PowerShell 5.1 and that arguments starting with @ need quoting. This is evidence of intended cross-platform use, not proof that every OS and every GUI behaves as described (SKILL.md L1–101).

Permissions are easier to explain when separated into four boundaries. The first is a skill-level convention; the remaining three are the runtime permission layers I call “the three layers” in the excerpt and elsewhere.

(process_manager.rs L1369–1439 and Cua README.md L38–62)

On the Cua README, standard is the default profile for normal local CLI and MCP use, with its built-in boundaries preserved. bounded is deny-by-default and requires a capability manifest, and unrestricted requires an explicit bypass acknowledgment. The mode belongs to the process that owns the runtime and is fixed at startup. Reading that as “Open Interpreter prompts on every click” would be incorrect.

On macOS, the Cua README explains responsible app identity and TCC attribution, and the responsible party differs across standalone CuaDriver.app, direct MCP, and embedded startup (Cua README.md L158–169). The OI skill does not detail this branching, so an article should avoid claiming that granting the terminal permission is always sufficient.

On Windows, the supplementary read of the Cua installer showed a comment that reads as “autostart off by default,” while the actual parameter definition is $AutoStart = $true, with -NoAutoStart being the way to turn it off (Cua install.ps1 L79–126). Because the comment and the implementation can disagree, a real-hardware article should record the installed version and startup state. The OI skill fetches and runs this installer without extra arguments.

Do not assume computer_use.rs implies a Cua wiring

The repository does include ComputerUseConfigToml, a macOS bundle ID field, and Windows AUMID / exe configuration types (computer_use.rs L1–39). Their existence does not prove that this configuration is checked on every operation of the qa-testing → cua-driver CLI path. Any relationship to the Cua-specific connection route needs to be traced separately, so I did not describe this as Cua’s permission surface.

Reproducibility: pin OI’s version and Cua’s version independently

Open Interpreter itself can be pinned by tag, but the QA skill’s dependency retrieval is dynamic.

The Cua installer I looked at documents CUA_DRIVER_RS_VERSION, but that is a Cua-installer feature, not something OI sets. The shell installer further delegates to a public URL at _install-rust.sh, so pinning only the entry-point script does not necessarily pin what actually gets fetched (Cua install.sh L18–38 and Cua install.sh L93–122). The Windows side had a baked-in 0.26.1 at the time, but that does not mean OI rust-v0.0.42 always uses 0.26.1.

Experiment records should include the OI commit, harness name, model name, the Cua binary path, version, and hash, the OS version, and how the daemon was started, all together. What matters for reproducibility is less that both are written in Rust and more that runtime dependencies are distributed separately.

Comparison table against the Hermes article

The Hermes column is from Reading Hermes Agent v2026.9.7 Computer Use Internals. The OI column is from the pinned source read here.

The takeaway from the comparison is a difference in where responsibility sits: Open Interpreter uses a small procedure document to leverage an external CLI, while Hermes manages the meaning of the action request and its result in a dedicated layer. I did not measure success rate, cost, latency, or non-intrusiveness, so I am not ranking them.

Wordings I want to avoid, and their source-consistent alternatives

To leave the intent behind the phrasing visible, here is the pair list of formulations to avoid and the source-consistent alternatives.

What I want to verify on real hardware next

To stop confusing source-read conclusions with runtime behavior, I plan to try the next round from the host terminal directly, not through an agent shell. I will share findings in a daily standup or team meeting so the next person does not step on the same rakes.

• Record the OI version and commit, the harness, the model, and the Cua version and hash together, and confirm they align with the source read
• Inspect the skill listing and the loaded body, to see whether the QA skill has been disabled or overridden
• Compare PATH, installer version, and daemon state before deciding whether list-tools really means the operation path is ready
• Take a simple “increment the counter by one” task and save the before / after result and the CLI-argument history
• With multiple windows in play, verify that the correct pid / window / session / element is maintained across CLI calls
• Record focus and cursor states before and after operations to understand the driver / app non-intrusion boundary
• Compare Japanese input methods, confirmed output, and saved contents to see the difference between IME and paste paths
• Observe screenshot output paths alongside image-tool invocations to confirm images actually reach the model
• Record OI approval, the Cua mode, and OS grants as three independent traces, to see where a decision was made
• Provoke a timeout on a safe test app and verify from logs that the model does not resubmit the same mutation

Sample commands (when run in a terminal, no auto-install or GUI operation) would look like this:

# Inside an Open Interpreter checkout
git rev-parse 'rust-v0.0.42^{commit}'
git show -s --format=fuller rust-v0.0.42
 
# The actual binaries in use. If missing, record that too.
command -v interpreter
interpreter --version
command -v cua-driver
cua-driver --version
cua-driver list-tools

list-tools output is a spec-discovery record and should be kept separate from a real capture-success record. Hashes go through shasum -a 256 on macOS or Get-FileHash on Windows, and the wrapper and the actual binary should be distinguished.

Evidence and limitations

• The working checkout stayed on main. Investigation files were read with git show rust-v0.0.42:<path>, so the working tree at HEAD is not mixed with the tag content
• Tag commit was cross-checked with git ls-remote on the public remote
• I searched the entire tag for trycua / cua-driver / cua_driver. The prominent hits were the README translations, the QA skill, Kimi-related documents, and a test that handles skill bodies. I could not find evidence of a Cua-specific Rust adapter or a pinned Cargo dependency from that search and the call paths
• I also checked computer_use as a string and confirmed there are configuration types and guardian-related names. I keep “no Cua-specific path was observed” separate from “no Computer Use-related code exists”
• The skill-distribution test was read, not run in this investigation
• On the external Cua side I supplemented the README, the installer, the CLI entry point, and run_call() only. I did not audit the full per-OS native backend or the internal action-verification algorithms
• The Hermes comparison is drawn from the linked article, not from a fresh re-read of the current Hermes tree or all platforms
• Without GUI measurement, speed, success rate, Japanese input, image-recognition quality, and non-intrusion behavior are all unmeasured

That’s all from pinning Open Interpreter rust-v0.0.42 and reading through the binary-embedded QA skill, the delegation to external CLIs, generic command execution and approval, the view_image image path, the three permission layers across Open Interpreter, Cua, and the OS, and the version-pinning gap between OI and Cua, compared against the earlier Hermes read, from the Gemba.

References

• Open Interpreter official site
• openinterpreter/openinterpreter GitHub repository
• Open Interpreter rust-v0.0.42 source tree
• Open Interpreter commit 67299aeb
• trycua/cua GitHub repository
• trycua/cua commit b4e3caec (supplementary read)
• Best-effort background (cua docs)
• cua-driver permission modes and capability manifest
• cua-driver Platform Support
• agent-browser GitHub repository
• Model Context Protocol official site
• macOS Accessibility API documentation
• Windows UI Automation documentation
• AT-SPI2 repository
• macOS TCC documentation
• Earlier selection notes: Cross-platform Computer Use agents
• Earlier Hermes code-reading: Reading Hermes Agent v2026.9.7 Computer Use Internals