cua-driver

What it does

Orchestrates macOS app automation via cua-driver. When a user requests GUI operations in a real macOS application, it adheres to a snapshot-before-action loop, capturing the AX tree, then clicking, typing, or scrolling by element_index, and verifying by re-snapshot. The skill is used to drive native macOS apps (e.g., TextEdit, Finder, Numbers) and can operate via MCP server or CLI. It emphasizes not changing the user’s frontmost app and uses an invariant where snapshots and visible targets guide actions.

How it works

• Uses cua-driver to perform actions (click, type_text, scroll, etc.) by translating each reference like click({pid, window_id, x, y}) into the corresponding cua-driver command. All mentions of click(...), get_window_state(...), etc. map to cua-driver invocations.
• Maintains a frontmost-app rule: do not foreground or activate apps unless explicitly allowed; before any shell command that could activate or bring forward an app, checks and follows the policy outlined in the no-foreground contract.
• Performs a diagnostic self-check before each Bash call touching a macOS app, ensuring the command won’t foreground the target, move the real cursor, or bypass cua-driver.
• Uses a workflow that emphasizes taking an initial AX-tree snapshot, performing a target action by element_index or pixel-based coordinates when necessary, and immediately re-snapshot to verify observable changes.
• Provides prerequisites for running cua-driver: ensuring it is on PATH, permissions granted, and the daemon started (via serve mode). It recommends start commands and status checks to maintain a persistent per-pid element cache for window interactions.

When to use it

Use when the user asks you to operate, drive, automate, or perform a GUI task in a real macOS application on the host (e.g., "open a file in TextEdit", "navigate to /Applications in Finder", "click the Save button in Numbers"). Employs the snapshot-before-action invariant and frontmost-app constraints unless the user explicitly requests foregrounding.

What it can touch

• Commands and tools: cua-driver with subcommands like click, get_window_state, and other referenced actions translated to cua-driver <tool-name> '<JSON-args>'.
• It documents translating click(...), get_window_state(...), etc. into cua-driver CLI calls and uses per-pid window IDs and element indices for interaction.
• Prerequisites mention cua-driver being on PATH and daemon management commands like open -n -g -a "DeepChat Computer Use" --args serve, along with cua-driver status.

Caveats

• Enforces the no-foreground contract: many commands that would foreground or activate apps are forbidden unless explicitly asked for by the user.
• Requires a persistent cua-driver daemon; without it, per-invocation caches die and element_index workflows may fail.
• The approach relies on accurate AX trees and visible targets; if snapshots are sparse or invalid, it recommends re-snapshot and fallback to in-window AX or pixel actions.
• Open/activate behaviors and keyboard shortcuts that focus apps are discouraged; only use launch_app for startup, not open forms, and only use activation when the user explicitly requests frontmost state.