OSWorld

What it is

OSWorld is a benchmark environment and evaluation framework for multimodal agents operating in real computer environments. It supports multiple providers (VMware/VirtualBox, Docker, Modal, Daytona, AWS) and includes documentation for setup, quickstart, experiments, and evaluation tooling. The project is hosted under the Apache-2.0 license and has a large number of stars and forks.

How it works

The repository provides installation instructions, including a VMware/VirtualBox path and a Docker path, with commands to install dependencies and set up the environment. It also describes provider-specific workflows (Modal, Daytona, AWS) and a Quick Start workflow for running experiments. The tooling outputs results including logs, screenshots, actions, and videos of agent task completion.

Getting started

VMware/VirtualBox (Desktop)

1. Clone the repository and install dependencies from requirements.txt. The README provides the exact commands:

# Clone the OSWorld repository
git clone https://github.com/xlang-ai/OSWorld

# Change directory into the cloned repository
cd OSWorld

# Optional: Create a Conda environment for OSWorld
# conda create -n osworld python=3.10
# conda activate osworld

# Install required dependencies
pip install -r requirements.txt

pip install desktop-env

2. Install VMware Workstation Pro and verify with:

vmrun -T ws list

If issues, VirtualBox is available as an alternative. The setup script downloads VMs automatically.

Docker

Prerequisites include ensuring KVM support and installing Docker Desktop/Engine as appropriate. The docs show how to check KVM support:

egrep -c '(vmx|svm)' /proc/cpuinfo

Running experiments involves initializing DesktopEnv with provider_name = docker and appropriate OS VM settings, with a note about cleaning up residual containers.

Modal, Daytona, AWS

The README includes commands to install Modal SDK and run OSWorld with modal provider, and notes for Daytona and AWS deployment. Example Modal setup:

pip install 'modal>=1.5.0'
modal setup
# Stage OSWorld disk image into Modal
python -m desktop_env.providers.modal.setup --os Ubuntu
# Run OSWorld with Modal Sandboxes
python quickstart.py --provider_name modal --headless true

In AWS, the docs reference Setup Guideline and AWS Configuration Guide for parallel evaluation.

Quick Start

# Basic usage with default settings
python quickstart.py

# Customize provider and VM path
python quickstart.py --provider_name vmware --path_to_vm "path/to/your/vm.vmx"

Recent releases

• v0.1.16 (2024-06-26): VM + Cloud support, VirtualBox snapshots, AWS support, annotation bug fixes.
• v0.1.0 (2024-04-11): The first release of the OSWorld environment.

Traction

Stars: 3063 Forks: 510 Open issues: 173

Behind the repo

Website, Paper, Documentation, Data Viewer, Discord, and Cache links are included in the README.

Caveats

License: Apache-2.0. The README notes support for multiple providers (VMware, VirtualBox, Docker, Modal, Daytona, AWS) and provides environment setup details, including potential residual containers if tasks are interrupted when using Docker.