VLMEvalKit is an open-source Python toolkit to evaluate large vision-language models on multiple benchmarks with one command. It supports 200+ LMMs and 70+ benchmarks, and is actively updated through 2026.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
VLMEvalKit is an open-source evaluation toolkit for large vision-language models (LVLMs). It enables one-command evaluation of LVLMs on various benchmarks, with generation-based evaluation and results produced using exact matching and LLM-based answer extraction.
How it works
The project provides a framework to evaluate LVLMs across multiple benchmarks without extensive data preparation. It supports models through a config interface and offers utilities to run predictions and compute metrics, including options for thinking-mode handling and long-response formats via environment variables.
Getting started
Install and basic usage instructions are located in the repository's documentation. The README provides a sample demonstrating a model config lookup and forward pass:
# Demo
from vlmeval.config import supported_VLM
model = supported_VLM['idefics_9b_instruct']()
# Forward Single Image
ret = model.generate(['assets/apple.jpg', 'What is in this image?'])
print(ret) # The image features a red apple with a leaf on it.
# Forward Multiple Images
ret = model.generate(['assets/apple.jpg', 'assets/apple.jpg', 'How many apples are there in the provided images? '])
print(ret) # There are two apples in the provided images.
Getting started
The project includes QuickStart and Development guides in the docs, covering how to set up models, benchmarks, and configuration files. It emphasizes one-command evaluation and generation-based scoring with options for prompt templates and evaluation pipelines.
Recent releases
Latest releases include:
- v0.3rc1 VLMEvalKit v0.3rc1 (2025-06-21)
- v0.2 VLMEvalKit v0.2 (2025-03-24)
- v0.2rc1 (2024-06-29)
- v0.1 VLMEvalKit v0.1 (2024-01-22)
Note
The READMEs reference quickstart and development guides in the docs, and mention new features like multi_generate support and TSV output for long responses in specific PRs.
Traction
The repository has 4323 stars and 741 forks, with 272 open issues. It has activity in 2026 as of the last push on 2026-08-03.
Behind the repo
Open-compass is the organization hosting VLMEvalKit; the repository lists various benchmarks and model support details, including integration with HuggingFace spaces and model leaderboards.
Caveats
License: Apache-2.0. Language: Python. Created: 2023-12-01. Last push: 2026-08-03.