sparse-retrieval-eval
Evaluate sparse retrieval models on standard IR benchmarks (BEIR, MIRACL, mMARCO). Covers all IR metrics (nDCG@k, Recall@k, MAP, MRR), dataset loading, sparse corpus encoding to CSR matrices, IDF-weighted retrieval, caching, and result interpretation. Triggers on: evaluate retrieval, BEIR benchmark, nDCG, recall@k, sparse retrieval evaluation, MIRACL evaluation, information retrieval metrics, IR evaluation, search quality metrics.
npx skills add majiayu000/claude-skill-registry --skill sparse-retrieval-eval --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Evaluates sparse retrieval models on standard IR benchmarks (BEIR, MIRACL, mMARCO). Covers all IR metrics (nDCG@k, Recall@k, MAP, MRR), dataset loading, sparse corpus encoding to CSR matrices, IDF-weighted retrieval, caching, and result interpretation. Triggers on: evaluate retrieval, BEIR benchmark, nDCG, recall@k, sparse retrieval evaluation, MIRACL evaluation, information retrieval metrics, IR evaluation, search quality metrics.
How it works
- Provides definitions and formulas for nDCG@k, Recall@k, MAP, and MRR, and shows how to compute them via a metrics module located at resources/metrics.py.
- Includes Python examples using evaluate_retrieval to produce a results dictionary.
- Details loading BEIR datasets (download_beir, load_beir_corpus, load_beir_queries, load_beir_qrels) and notes available BEIR datasets and standard evaluation.
- Describes MIRACL and mMARCO loading utilities, including language and split considerations.
- Explains sparse encoding of vectors to CSR via encode_to_csr and batched encoding via encode_corpus_batched.
- Provides IDF computation from a corpus CSR matrix using a BM25-inspired formula, and describes IDF-weighted retrieval where query vectors are weighted by idf at retrieval time.
- Defines retrieval pipelines: retrieve_top_k, retrieve_all_queries, and retrieve_batch for per-query and batched retrieval, including memory cautions.
- Covers caching encoded corpora with save_encoded_corpus and load_encoded_corpus, and cache naming conventions to avoid collisions.
- Finishes with the skeleton of a complete evaluation pipeline evaluate_sparse_model, indicating expected model interfaces for encoding documents and queries and handling qrels.
When to use it
Use when evaluating sparse retrieval models against BEIR, MIRACL, and mMARCO benchmarks, especially to obtain nDCG@k, Recall@k, MAP, and MRR metrics across specified k values, and to manage dataset loading, CSR encoding, IDF weighting, and caching.
What it can touch
- Tools: claude-code
- Files and modules: resources/metrics.py, data loading functions, CSR encoding utilities, IDF computation, retrieval functions, and caching utilities as described in the code blocks.
Caveats
- License: MIT
- Declared limitations include memory considerations for dense score matrices during batch retrieval and the need to batch queries for large corpora.
# Sparse Retrieval Evaluation Complete reference for evaluating sparse retrieval models on standard IR benchmarks. --- ## 1. IR Evaluation Metrics All metrics implementations are in `resources/metrics.py` (importable). ### nDCG@k (Normalized Discounted Cumulative Gain) The standard metric for graded relevance. Used as the primary metric in BEIR and MIRACL. ``` DCG@k = sum_{i=1}^{k} (2^{rel_i} - 1) / log2(i + 1) nDCG@k = DCG@k / IDCG@k ``` Where IDCG@k is DCG@k computed on the ideal (best possible) ranking. Range: [0, 1]. **Implementation note:** The denominator is `log2(i + 1)` where `i` is 1-indexed. In 0-indexed code this becomes `log2(i + 2)`. ### Recall@k Fraction of relevant documents found in the top-k results. ``` Recall@k = |retrieved@k ∩ relevant| / |relevant| ``` **Important:** Only documents with relevance > 0 in qrels count as relevant. Documents absent from qrels are non-relevant (score 0), not unjudged. ### MAP (Mean Average Precision) For binary relevance. Average of precision values at each relevant-document rank. ``` AP(q) = (1/|relevant|) * sum_{k where doc_k relevant} Precision@k MAP = mean over queries of AP(q) ``` ### MRR (Mean Reciprocal Rank) ``` RR(q) = 1 /
- 1. IR Evaluation Metrics
- nDCG@k (Normalized Discounted Cumulative Gain)
- Recall@k
- MAP (Mean Average Precision)
- MRR (Mean Reciprocal Rank)
- Using the metrics module
- 2. Loading Datasets
- 2.1 BEIR Datasets
- 2.2 MIRACL (Multilingual)
- 2.3 mMARCO (Cross-lingual)
- 3. Sparse Encoding & CSR Matrix
- 3.1 Encoding Sparse Vectors to CSR
- 3.2 Batched Encoding for Large Corpora
- 4. IDF Computation
What does the sparse-retrieval-eval skill do?
Evaluate sparse retrieval models on standard IR benchmarks (BEIR, MIRACL, mMARCO). Covers all IR metrics (nDCG@k, Recall@k, MAP, MRR), dataset loading, sparse corpus encoding to CSR matrices, IDF-weighted retrieval, caching, and result interpretation. Triggers on: evaluate retrieval, BEIR benchmark, nDCG, recall@k, sparse retrieval evaluation, MIRACL evaluation, information retrieval metrics, IR evaluation, search quality metrics.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill sparse-retrieval-eval --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.