hugging-face-evaluation
Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format.
npx skills add majiayu000/claude-skill-registry --skill hugging-face-evaluation-sickn33-antigravity-awesome --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
A toolset for adding structured evaluation results to Hugging Face model cards. It supports three methods: extracting evaluation tables from README content, importing benchmark scores from Artificial Analysis, and running custom model evaluations with vLLM or accelerate backends (lighteval/inspect-ai). It updates model-index metadata to reflect leaderboard data and can prepare changes for PRs.
How it works
- Inspect and extract: Use the provided workflow to inspect README tables with inspect-tables, identify the target table with --table N, and extract it into model-index YAML via extract-readme, optionally using --model-column-index or --model-name-override. The tool formats the table as model-index YAML and applies a task type with --task-type.
- Import from Artificial Analysis: Use import-aa with AA_API_KEY to fetch benchmarks and convert responses into model-index entries, preserving source attribution and URLs. Can create pull requests with --create-pr.
- Run evaluations: Run HF Jobs or local scripts to perform evaluations via vLLM or inspect-ai paths. Includes guidance for using uv run scripts/evaluation_manager.py with various subcommands (inspect-tables, extract-readme, import-aa, get-prs, show, validate).
- Model-index management: Generates properly formatted YAML for model-index, supports merging with existing evaluations, and validates against Papers with Code specifications.
- Prerequisites and workflow: Requires environment setup (HF_TOKEN, AA_API_KEY), and suggests using uv run for automatic dependencies. Provides explicit command examples for each step.
When to use it
- You need to add structured evaluation results to a Hugging Face model card.
- You want to import benchmark data or run custom evaluations with vLLM, lighteval, or inspect-ai.
- You are preparing leaderboard-compatible model-index metadata for a model release.
What it can touch
- Commands and scripts: the workflow revolves around scripts/evaluation_manager.py and multiple uv run invocations, with dependencies such as huggingface_hub, markdown-it-py, python-dotenv, pyyaml, requests, and optional AA toolkit.
- Tools mentioned: uv run, hf jobs, and scripts within the skill directory (e.g., extract-readme, inspect-tables, import-aa).
Caveats
- Risk is declared as unknown.
- Requires environment variables HF_TOKEN and AA_API_KEY as applicable.
- Certain capabilities depend on external services (Artificial Analysis, HF Jobs) and GPU resources for vLLM-based evaluations.
# Overview This skill provides tools to add structured evaluation results to Hugging Face model cards. It supports multiple methods for adding evaluation data: - Extracting existing evaluation tables from README content - Importing benchmark scores from Artificial Analysis - Running custom model evaluations with vLLM or accelerate backends (lighteval/inspect-ai) ## When to Use - You need to add structured evaluation results to a Hugging Face model card. - You want to import benchmark data or run custom evaluations with vLLM, lighteval, or inspect-ai. - You are preparing leaderboard-compatible `model-index` metadata for a model release. ## Integration with HF Ecosystem - **Model Cards**: Updates model-index metadata for leaderboard integration - **Artificial Analysis**: Direct API integration for benchmark imports - **Papers with Code**: Compatible with their model-index specification - **Jobs**: Run evaluations directly on Hugging Face Jobs with `uv` integration - **vLLM**: Efficient GPU inference for custom model evaluation - **lighteval**: HuggingFace's evaluation library with vLLM/accelerate backends - **inspect-ai**: UK AI Safety Institute's evaluation framework # Version 1.3.0
- When to Use
- Integration with HF Ecosystem
- Core Dependencies
- Inference Provider Evaluation
- vLLM Custom Model Evaluation (GPU required)
- ⚠️ CRITICAL: Check for Existing PRs Before Creating New Ones
- 1. Inspect and Extract Evaluation Tables from README
- 2. Import from Artificial Analysis
- 3. Model-Index Management
- 4. Run Evaluations on HF Jobs (Inference Providers)
- 5. Run Custom Model Evaluations with vLLM (NEW)
- Before running the script
- Running the script
- Features
uv run scripts/evaluation_manager.py get-prs --repo-id "username/model-name" uv run scripts/evaluation_manager.py --help uv run scripts/evaluation_manager.py inspect-tables --help uv run scripts/evaluation_manager.py extract-readme --help uv run scripts/train_sft_example.py uv run scripts/evaluation_manager.py inspect-tables --repo-id "username/model" uv run scripts/evaluation_manager.py extract-readme \ or Create .env file echo "AA_API_KEY=your-api-key" >> .env
What does the hugging-face-evaluation skill do?
Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill hugging-face-evaluation-sickn33-antigravity-awesome --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.