ref-hallucination-arena
Benchmark LLM reference recommendation capabilities by verifying every cited paper against Crossref, PubMed, arXiv, and DBLP. Measures hallucination rate, per-field accuracy (title/author/year/DOI), discipline breakdown, and year constraint compliance. Supports tool-augmented (ReAct + web search) mode. Use when the user asks to evaluate, benchmark, or compare models on academic reference hallucination, literature recommendation quality, or citation accuracy.
Profile →npx skills add agentscope-ai/OpenJudge --skill ref-hallucination-arena --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Reference Hallucination Arena Skill Evaluate how accurately LLMs recommend real academic references using the OpenJudge `RefArenaPipeline`: 1. **Load queries** — from JSON/JSONL dataset 2. **Collect responses** — BibTeX-formatted references from target models 3. **Extract references** — parse BibTeX entries from model output 4. **Verify references** — cross-check against Crossref / PubMed / arXiv / DBLP 5. **Score & rank** — compute verification rate, per-field accuracy, discipline breakdown 6. **Generate report** — Markdown report + visualization charts ## Prerequisites ```bash # Install OpenJudge pip install py-openjudge # Extra dependency for ref_hallucination_arena (chart generation) pip install matplotlib ``` ## Gather from user before running | Info | Required? | Notes | |------|-----------|-------| | Config YAML path | Yes | Defines endpoints, dataset, verification settings | | Dataset path | Yes | JSON/JSONL file with queries (can be set in config) | | API keys | Yes | Env vars: `OPENAI_API_KEY`, `DASHSCOPE_API_KEY`, etc. | | CrossRef email | No | Improves API rate limits for verification | | PubMed API key | No | Improves PubMed rate limits | | Output directory | No | De
- Prerequisites
- Gather from user before running
- Quick start
- CLI
- Python API
- CLI options
- Minimal config file
- Full config reference
- task
- dataset
- targetendpoints.\<name\>
- verification
- evaluation
- output
Install OpenJudge pip install py-openjudge Extra dependency for ref_hallucination_arena (chart generation) pip install matplotlib Run evaluation with config file python -m cookbooks.ref_hallucination_arena --config config.yaml --save Resume from checkpoint (default behavior) Start fresh, ignore checkpoint python -m cookbooks.ref_hallucination_arena --config config.yaml --fresh --save Override output directory
What does the ref-hallucination-arena skill do?
Benchmark LLM reference recommendation capabilities by verifying every cited paper against Crossref, PubMed, arXiv, and DBLP. Measures hallucination rate, per-field accuracy (title/author/year/DOI), discipline breakdown, and year constraint compliance. Supports tool-augmented (ReAct + web search) mode. Use when the user asks to evaluate, benchmark, or compare models on academic reference hallucination, literature recommendation quality, or citation accuracy.
How do I install it?
Run `npx skills add agentscope-ai/OpenJudge --skill ref-hallucination-arena --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From agentscope-ai/OpenJudge, a repository with 764 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.