A Python-based leaderboard to compare LLM hallucinations when summarizing short documents, using Vectara's HHEM model. It includes a public interactive page and historical versions.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
Hallucination Leaderboard is a public LLM leaderboard computed using Vectara's Hallucination Evaluation Model (HHEM). It evaluates how often an LLM introduces hallucinations when summarizing a document. The README notes an interactive leaderboard on Hugging Face and references previous versions in GitHub trees: hhem-1.0-final and hhem-2.3-old-dataset. The page shows a Last updated timestamp and a table listing models with Hallucination Rate, Factual Consistency Rate, Answer Rate, and Average Summary Length (Words). The Model section states HHEM-2.3 is used and mentions an open-source variant HHEM-2.1-Open on Hugging Face and Kaggle. The Dataset section describes over 7700 articles from various sources and lengths up to 24K words.
How it works
The leaderboard feeds a full dataset to each LLM and asks for a summary containing only facts from the document. It computes the factual consistency rate and the hallucination rate, along with the answer rate and summary length per model. It uses a temperature of 0 when calling LLMs where possible. The Methodology section explains that evaluation is based on summarization truthfulness rather than general factual accuracy.
Getting started
The README provides a link to the interactive leaderboard on Hugging Face but does not include explicit installation commands or setup steps for local reproduction. It mentions an open-source variant of the evaluation model and references multiple model endpoints.
Recent releases
RELEASES (latest 0):
- none
Traction
Stars: 3298 Forks: 107 Open issues: 19
Behind the repo
Not applicable in provided facts.
Caveats
License: Apache-2.0 Created: 2023-10-31 Last push: 2026-05-11 Language: Python