BEIR is a Python benchmark package for evaluating information retrieval models across 15+ datasets, with multiple release updates and examples. It supports preprocessing, various architectures, and evaluation metrics.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
BEIR is a heterogeneous benchmark containing diverse IR tasks. It also provides a common and easy framework for evaluation of NLP-based retrieval models within the benchmark.
How it works
The repository offers a framework to preprocess datasets, run retrieval models (lexical, dense, sparse, and reranking-based), and evaluate with standard metrics. It includes examples and tutorials for usage with different model types (Sentence-BERT, LoRA/vLLM, HuggingFace, and APIs).
Getting started
Install via pip:
pip install beir
If you want to build from source, use:
$ git clone https://github.com/beir-cellar/beir.git
$ cd beir
$ pip install -e .
Tested with python versions 3.9+
Recent releases
- v2.2.0: Crucial updates in the BEIR benchmark! Multi-gpu huggingface support, vLLM with LoRA evaluation, API evaluation support, encode_and_retrieve function, etc. (2025-06-04)
- v2.1.0: Let's drink BEIR! Now you can evaluate latest embedding models such as E5, Stella, NV-Embed-v2, LLM2Vec, Tevatron etc. (2025-02-25)
- v2.0.0: We are back with bugfixes and improving BEIR after a long break (2023-08-03)
- v1.0.1: Multi-GPU, HF dataloaders, MonoT5 rerankers and a brand new Wiki page (2022-06-30)
- v1.0.0: BEIR is back with a brand new organization of its own moving forward, New sparse model releases, ColBERT evaluation and fixing breaking changes (2022-03-21)
Traction
Stars: 2263 Forks: 247 Open issues: 82
Behind the repo
URL: https://github.com/beir-cellar/beir/
Caveats
License: Apache-2.0 Created: 2021-01-18 Last push: 2025-10-16 Language: Python