OpenCompass is an extensive LLM evaluation platform in Python that supports many models and 100+ datasets. It provides CLI and Python scripts for running evaluations and supports multiple backends like LMDeploy and vLLM.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
OpenCompass is an LLM evaluation platform that supports a wide range of models (Llama3, Mistral, InternLM2, GPT-4, LLaMa2, Qwen, GLM, Claude, etc) over 100+ datasets.
How it works
The project offers configurations and tooling to run evaluations via CLI or Python scripts, with optional acceleration backends (LMDeploy, vLLM) for inference. It includes a data preparation workflow (offline and automatic dataset download) and a set of example scripts for evaluating various benchmarks.
Getting started
Installation sections show environment setup and installation commands:
-
Create a virtual environment with conda:
conda create --name opencompass python=3.12 -y conda activate opencompass -
Install OpenCompass via pip (full installation options available):
pip install -U opencompass ## Full installation (with support for more datasets) # pip install "opencompass[full]" ## Environment with model acceleration frameworks ## Manage different acceleration frameworks using virtual environments ## since they usually have dependency conflicts with each other. # pip install "opencompass[lmdeploy]" # pip install "opencompass[vllm]" ## API evaluation (i.e. Openai, Qwen) # pip install "opencompass[api]" -
Install OpenCompass from source:
git clone https://github.com/open-compass/opencompass opencompass cd opencompass pip install -e . # pip install -e ".[full]" # pip install -e ".[vllm]" -
Data preparation can be offline or automatic, with commands to download datasets and options to use ModelScope for on-demand loading.
-
Example usage for evaluation:
# CLI opencompass --models hf_internlm2_5_1_8b_chat --datasets demo_gsm8k_chat_gen # Python scripts opencompass examples/eval_chat_demo.py -
For accelerated evaluation with LMDeploy:
# CLI opencompass --models hf_internlm2_5_1_8b_chat --datasets demo_gsm8k_chat_gen -a lmdeploy # Python scripts opencompass examples/eval_lmdeploy_demo.py -
Supported models and datasets can be listed via tools, with examples:
# List all configurations python tools/list_configs.py # List configurations related to llama and mmlu python tools/list_configs.py llama mmlu
Recent releases
Latest releases show v0.5.3 (2026-06-29) adding extensive new benchmarks and dataset support (including AIME2026); v0.5.2 (2026-02-14) with comprehensive scientific and general benchmark support; prior notes indicate ongoing feature additions.
Traction
OpenCompass repository has 7273 stars and 823 forks, with 367 open issues.
Behind the repo
Not included in the provided facts.
Caveats
License: Apache-2.0. Created 2023-06-15; last push 2026-08-04. Installation notes include Python version requirements (Python 3.12 for regular/full installation; Python 3.10 for pyext-backed datasets). Data preparation options include offline dataset download from a specific release URL and ModelScope integration.