ClawBench is an open-source Python-based benchmark that evaluates browser AI agents on real online tasks across live websites, with V1 and V2 task sets and a 5-layer recording pipeline. It provides Docker/isolate harnesses and a Python package for one-command install via clawbench-eval.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (2 so far).
What it is
ClawBench is an open-source benchmark for evaluating AI browser agents on everyday online tasks across live websites. It includes V1 (153 tasks across 144 websites) and V2 (130 tasks) task sets, a five-layer recording pipeline, and an agentic evaluator that compares runs against human references.
How it works
The project describes a workflow where you pick a task from V1 or V2, ClawBench spins up an isolated Docker container with Chromium, and an AI agent drives the browser to perform actions while an Interceptor captures data across five layers for evaluation.
Getting started
Clone the repository and run the provided script:
git clone https://github.com/reacher-z/ClawBench.git && cd ClawBench && ./run.sh
Human quick start provides alternative installation methods for the clawbench-eval package:
uv tool install clawbench-eval
Or:
pipx install clawbench-eval
Or:
python -m pip install clawbench-eval
The README also notes that the installed commands are clawbench, clawbench-run, clawbench-batch, and clawbench-harbor-adapt, and mentions running the root uv package entrypoint for more granular control after cloning.
Getting started (alternative commands)
The one-line clone/install path is complemented by quick-start sections for human users and an LLM quick start referring to AGENTS.md for prompting an agent to run tasks.
Recent releases
Latest releases include:
- v0.7.0 (2026-06-22): Refactor/replace extension by cdp and docs update
- v0.6.0 (2026-06-05): add Harbor (Terminus 2) agent harness
- v0.5.0 (2026-05-24): ci delay before download from testpypi and display token usage in USD
- v0.4.1 (2026-05-23): Fix/201 task mismatch and build release
- v0.4.0 (2026-05-23): docs sync BibTeX key and templates
Traction
The repository has 510 stars, 28 forks, and 41 open issues. It gained 5 stars in the last 24 hours.
Behind the repo
Not applicable from the provided facts.
Caveats
License is Apache-2.0. Created 2026-04-10. Last push 2026-07-17. The repository is Python-based and ships datasets describing V1 (153 tasks) and V2 (130 tasks) across 144 websites. Open issues count is 41.