What it is

AgentBench is a benchmark designed to evaluate LLMs as agents across multiple environments. The repository includes a variant focused on function calling that integrates with AgentRL and supports containerized deployment for several tasks (alfworld, dbbench, knowledgegraph, os_interaction, webshop).

How it works

The FC version uses function-calling style prompts and provides a Docker Compose setup to start multiple task workers and services (AgentRL Controller, task workers for each environment, Redis, and a freebase server for KG).

Getting started

Quick Start (from README)

"""Shell examples provided in repository"""

docker compose setup for FC variant

docker compose -f extra/docker-compose.yml up

Docker images and services start as listed:

• AgentRL Controller
• alfworld task worker (x1, increase as needed)
• dbbench task worker (x1, increase as needed)
• knowledgegraph task worker (x1, increase as needed)
• os_interaction task worker (x1, increase as needed)
• webshop task worker (x1, increase as needed)
• freebase server (for knowledgegraph task)
• Redis server (for container allocation)

WARNING: webshop environment requires ~16GB RAM; alfworld may leak memory and disk space until task worker restart.

Quick docker commands (examples)

# dbbench
docker pull mysql

# os_interaction environments
docker build -t local-os/default -f ./data/os_interaction/res/dockerfiles/default data/os_interaction/res/dockerfiles
docker build -t local-os/packages -f ./data/os_interaction/res/dockerfiles/packages data/os_interaction/res/dockerfiles
docker build -t local-os/ubuntu -f ./data/os_interaction/res/dockerfiles/ubuntu data/os_interaction/res/dockerfiles

Start the stack

docker compose -f extra/docker-compose.yml up

Notes

• If Redis is already running (version 7+), you can omit the Redis service.
• The README provides additional guidance on starting task servers and the lite preset for reduced RAM usage.

Step-by-step usage (from README)

1. Prerequisites: clone, create a conda environment, install dependencies, ensure Docker installed.
2. Configure agent: edit configs/agents/openai-chat.yaml to set the OpenAI API key and agent model (default gpt-3.5-turbo-0613).
3. Start task server: python -m src.start_task -a (or with lite preset: python -m src.start_task -a --config configs/start_task_lite.yaml).
4. Start the assigner: python -m src.assigner (or with lite: python -m src.assigner --config configs/assignments/lite.yaml).

Recent releases

• The latest release line is listed as 0 with no releases; latest tag indicates 0 releases. The Releases section shows "latest 0: - none".

Traction

• Stars: 3641
• Forks: 272
• Open issues: 74

Behind the repo

• The project links to AgentRL for the FC variant and mentions integration with AgentRL in the FC deployment.

Caveats

• License: Apache-2.0
• Created: 2023-07-28
• Last push: 2026-02-08
• CPU/memory requirements noted: webshop ~16GB RAM; varying memory usage per task listed in a table.