computer-use-agents
Build AI agents that interact with computers like humans do - viewing screens, moving cursors, clicking buttons, and typing text. Covers Anthropic's Computer Use, OpenAI's Operator/CUA, and open-source alternatives.
Profile →npx skills add sickn33/agentic-awesome-skills --skill computer-use-agents --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Provides a computer-use agent that observes the screen, reasons about actions, and executes mouse/keyboard operations in a sandboxed environment. It outlines a Perception-Reasoning-Action loop with components for PERCEPTION (screenshot), REASONING (vision-language model analysis), ACTION (mouse/keyboard operations), and FEEDBACK (result observation). It includes concrete action types (click, type, key, scroll, move) and a done signal to terminate tasks. It demonstrates an implementation based on Anthropic Computer Use patterns and includes an anti-patterns and sandboxing guidance, plus Dockerfile and docker-compose configurations for a sandboxed environment.
How it works
- Perception: capture_screenshot returns a base64-encoded image of the screen and resizes it for token efficiency.
- Reasoning: sends a structured message containing the task and the current screenshot to a vision model, receiving a JSON action.
- Action: execute_action interprets the action dict and performs corresponding UI operations using pyautogui (click, type, key, scroll, move).
- Loop: run runs the sequence up to max_steps (50) with a small action_delay to allow UI updates, and stops when a done action is received.
- System prompt: instructs the agent to respond only with a JSON action and to use exact coordinates.
- Sandboxing: provides a Docker-based sandbox with network, filesystem, and resource constraints, plus a Python wrapper (SandboxedAgent) to start, run tasks, and stop containers.
When to use it
- When deploying any computer-use agent that needs desktop control in a safe, sandboxed environment.
- When testing agent behavior safely and running untrusted automation tasks.
What it can touch
- Perception and control surface via a sandboxed environment: screen capture, mouse actions (click, move), keyboard input (type, key), and scrolling.
- The sandbox and tooling are described with Dockerfile, docker-compose.yml, and a Python wrapper (SandboxedAgent) to start/stop containers and execute tasks through an API endpoint.
Caveats
- Includes notes on anti-patterns (e.g., infinite loops, no action delay, full-resolution screenshots).
- Explicitly emphasizes sandboxed environments to minimize blast radius and restrict system access.
- Mentions Anthropic tools and potential UI element limitations (some dropdowns/scrollbars can be tricky) as a documented limitation.
- License: MIT
# Computer Use Agents Build AI agents that interact with computers like humans do - viewing screens, moving cursors, clicking buttons, and typing text. Covers Anthropic's Computer Use, OpenAI's Operator/CUA, and open-source alternatives. Critical focus on sandboxing, security, and handling the unique challenges of vision-based control. ## Patterns ### Perception-Reasoning-Action Loop The fundamental architecture of computer use agents: observe screen, reason about next action, execute action, repeat. This loop integrates vision models with action execution through an iterative pipeline. Key components: 1. PERCEPTION: Screenshot captures current screen state 2. REASONING: Vision-language model analyzes and plans 3. ACTION: Execute mouse/keyboard operations 4. FEEDBACK: Observe result, continue or correct Critical insight: Vision agents are completely still during "thinking" phase (1-5 seconds), creating a detectable pause pattern. **When to use**: Building any computer use agent from scratch,Integrating vision models with desktop control,Understanding agent behavior patterns from anthropic import Anthropic from PIL import Image import base64 import pyautogui import time class Comput
- Patterns
- Perception-Reasoning-Action Loop
- Antipatterns
- Sandboxed Environment Pattern
- Anthropic Computer Use Implementation
- Browser-Use Pattern (Playwright-based)
- User Confirmation Pattern
- Action Logging Pattern
- Sharp Edges
- Web Content Can Hijack Your Agent
- Defense in depth - no single solution works
- Vision Agents Click Exact Centers
- Add human-like variance to actions
- Rotate user agents and fingerprints
What does the computer-use-agents skill do?
Build AI agents that interact with computers like humans do - viewing screens, moving cursors, clicking buttons, and typing text. Covers Anthropic's Computer Use, OpenAI's Operator/CUA, and open-source alternatives.
How do I install it?
Run `npx skills add sickn33/agentic-awesome-skills --skill computer-use-agents --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From sickn33/agentic-awesome-skills, a repository with 44,414 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.