hugging-face-evaluation-manager
Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format.
npx skills add majiayu000/claude-skill-registry --skill hugging-face-evaluation-manager --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Coordinates adding evaluation results to Hugging Face model cards, enabling management of evaluation data from README tables, Artificial Analysis imports, and custom evaluations using vLLM/accelerate backends. It supports updating model-index metadata and supports PR creation and batch processing.
How it works
- Inspecting README content to locate and parse Markdown tables with evaluation data, using markdown-it-py for parsing and converting the selected table into model-index YAML; supports selecting a specific table with --table N and aligning models/columns via --model-column-index or --model-name-override.
- Importing benchmark scores directly from Artificial Analysis via API and formatting responses into model-index YAML, preserving source attribution and URLs; supports creating PRs for updates.
- Managing model-index entries: generating properly formatted YAML, merging new evaluations into existing model cards without overwriting, and validating against Papers with Code specifications.
- Running evaluations: submitting jobs on Hugging Face infrastructure (HF Jobs) or locally, using vLLM with lighteval or accelerate backends, or using inspect-ai workflows; includes guidance for hardware selection and secret/token handling.
- Providing CLI scripts in the scripts/ directory to perform these operations (inspect-tables, extract-readme, import-aa, show, validate, get-prs, run_eval_job, lighteval_vllm_uv, inspect_vllm_uv).
When to use it
Use when you need to add or update evaluation results in a model’s Hugging Face model card, either by extracting existing data from a README, importing external benchmark scores, or running new evaluations with vLLM/inspect-ai on local or HF Jobs infrastructure.
What it can touch
- Tools and scripts: uv run scripts/evaluation_manager.py, lf hf jobs uv run, and various helper scripts like scripts/lighteval_vllm_uv.py, scripts/inspect_vllm_uv.py, scripts/run_eval_job.py, scripts/evaluation_manager.py extract-readme, import-aa, inspect-tables, show, validate, get-prs.
- Dependencies include huggingface_hub, markdown-it-py, python-dotenv, pyyaml, requests, and evaluation-specific packages like lighteval, vllm, accelerate, inspect-ai, openai.
Caveats
- Requires environment variables HF_TOKEN and, for Artificial Analysis, AA_API_KEY; .env may be loaded if python-dotenv is installed.
- vLLM evaluation is GPU-dependent and requires uv and sufficient GPU memory; documented prerequisites include checking script paths, GPU availability, and hardware compatibility.
- The workflow emphasizes checking for existing PRs before creating new ones to avoid duplicates; requires manual confirmation to proceed with PR creation when applicable.
- Output formats rely on model-index YAML compatibility and exact table/column mappings; mismatches may affect YAML generation.
# Overview This skill provides tools to add structured evaluation results to Hugging Face model cards. It supports multiple methods for adding evaluation data: - Extracting existing evaluation tables from README content - Importing benchmark scores from Artificial Analysis - Running custom model evaluations with vLLM or accelerate backends (lighteval/inspect-ai) ## Integration with HF Ecosystem - **Model Cards**: Updates model-index metadata for leaderboard integration - **Artificial Analysis**: Direct API integration for benchmark imports - **Papers with Code**: Compatible with their model-index specification - **Jobs**: Run evaluations directly on Hugging Face Jobs with `uv` integration - **vLLM**: Efficient GPU inference for custom model evaluation - **lighteval**: HuggingFace's evaluation library with vLLM/accelerate backends - **inspect-ai**: UK AI Safety Institute's evaluation framework # Version 1.3.0 # Dependencies ## Core Dependencies - huggingface_hub>=0.26.0 - markdown-it-py>=3.0.0 - python-dotenv>=1.2.1 - pyyaml>=6.0.3 - requests>=2.32.5 - re (built-in) ## Inference Provider Evaluation - inspect-ai>=0.3.0 - inspect-evals - openai ## vLLM Custom Model Evaluation (GPU req
- Integration with HF Ecosystem
- Core Dependencies
- Inference Provider Evaluation
- vLLM Custom Model Evaluation (GPU required)
- ⚠️ CRITICAL: Check for Existing PRs Before Creating New Ones
- 1. Inspect and Extract Evaluation Tables from README
- 2. Import from Artificial Analysis
- 3. Model-Index Management
- 4. Run Evaluations on HF Jobs (Inference Providers)
- 5. Run Custom Model Evaluations with vLLM (NEW)
- Before running the script
- Running the script
- Features
- Prerequisites
uv run scripts/evaluation_manager.py get-prs --repo-id "username/model-name" uv run scripts/evaluation_manager.py --help uv run scripts/evaluation_manager.py inspect-tables --help uv run scripts/evaluation_manager.py extract-readme --help uv run scripts/train_sft_example.py uv run scripts/evaluation_manager.py inspect-tables --repo-id "username/model" uv run scripts/evaluation_manager.py extract-readme \ or Create .env file echo "AA_API_KEY=your-api-key" >> .env
What does the hugging-face-evaluation-manager skill do?
Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill hugging-face-evaluation-manager --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.