Agent skill · Backend & API

hugging-face-evaluation

Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format.

majiayu000github.com/majiayu000GitHub ↗
claude-codeMIT
Install
npx skills add majiayu000/claude-skill-registry --skill hugging-face-evaluation-ihatesea69-hieunghi-ai-skills --agent claude-code

Same command for any agent — swap --agent for codex, cursor, copilot.

Facts
Files in the skill folder: 2
SKILL.md size: 22 KB
Bundled scripts: none
Path: skills/ai-ml/hugging-face-evaluation-ihatesea69-hieunghi-ai-skills/SKILL.md
Open the folder on GitHub →
Where it comes from
Stars: 534
Language: HTML

Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.

Review
written from the skill's own SKILL.md · Aug 5, 2026

What it does

Provides tools to add structured evaluation results to Hugging Face model cards, including extracting existing evaluation tables from README content, importing benchmark scores from Artificial Analysis API, and running custom model evaluations with vLLM or accelerate backends (lighteval/inspect-ai). It updates model-index metadata for leaderboard integration and supports preserving source attribution and URLs when importing scores.

How it works

  • Inspects README to locate and parse evaluation tables using markdown-it-py, then converts a selected table into model-index YAML.
  • Imports benchmark scores from Artificial Analysis via API and formats them into model-index entries, preserving metadata and enabling PR creation.
  • Manages model-index entries: generates properly formatted YAML, merges updates into existing model cards without overwriting, and validates against Papers with Code specifications.
  • Runs evaluations on Hugging Face Jobs or locally using vLLM or lighteval backends, with options for CPU/GPU, backend selection, and secret management via CLI.
  • Offers a multi-step workflow: inspect-tables, extract-readme, then apply changes via push or create-pr, and supports importing AA data or running evaluation jobs with scripts mentioned in the CLI help.

When to use it

Use when you need to add or update evaluation results in a model's Hugging Face model card, either by extracting from existing README tables, importing external benchmark scores, or performing new evaluations with vLLM/lighteval backends. Trigger sequence typically: inspect-tables → extract-readme → apply (push or PR).

What it can touch

  • Tools and scripts referenced: uv run scripts/evaluation_manager.py and related subcommands such as inspect-tables, extract-readme, import-aa, get-prs, show, and validate.
  • Dependencies include huggingface_hub, markdown-it-py, python-dotenv, pyyaml, requests, and specialized backends like lighteval, vllm, and inspect-ai.
  • Requires environment configuration variables such as HF_TOKEN and AA_API_KEY as described in Usage Instructions.

Caveats

  • CRITICAL: Check for existing PRs before creating new ones; use uv run scripts/evaluation_manager.py get-prs --repo-id "username/model-name" and only proceed if user confirms.
  • Requires proper hardware and GPU availability for vLLM/backends and may rely on Hugging Face Jobs or local GPU resources.
  • Some features depend on external services (Artificial Analysis, HF Jobs) and may require API keys and tokens.
From the SKILL.md

# Overview This skill provides tools to add structured evaluation results to Hugging Face model cards. It supports multiple methods for adding evaluation data: - Extracting existing evaluation tables from README content - Importing benchmark scores from Artificial Analysis - Running custom model evaluations with vLLM or accelerate backends (lighteval/inspect-ai) ## Integration with HF Ecosystem - **Model Cards**: Updates model-index metadata for leaderboard integration - **Artificial Analysis**: Direct API integration for benchmark imports - **Papers with Code**: Compatible with their model-index specification - **Jobs**: Run evaluations directly on Hugging Face Jobs with `uv` integration - **vLLM**: Efficient GPU inference for custom model evaluation - **lighteval**: HuggingFace's evaluation library with vLLM/accelerate backends - **inspect-ai**: UK AI Safety Institute's evaluation framework # Version 1.3.0 # Dependencies ## Core Dependencies - huggingface_hub>=0.26.0 - markdown-it-py>=3.0.0 - python-dotenv>=1.2.1 - pyyaml>=6.0.3 - requests>=2.32.5 - re (built-in) ## Inference Provider Evaluation - inspect-ai>=0.3.0 - inspect-evals - openai ## vLLM Custom Model Evaluation (GPU req

What's inside
Steps it walks through
  1. Integration with HF Ecosystem
  2. Core Dependencies
  3. Inference Provider Evaluation
  4. vLLM Custom Model Evaluation (GPU required)
  5. ⚠️ CRITICAL: Check for Existing PRs Before Creating New Ones
  6. 1. Inspect and Extract Evaluation Tables from README
  7. 2. Import from Artificial Analysis
  8. 3. Model-Index Management
  9. 4. Run Evaluations on HF Jobs (Inference Providers)
  10. 5. Run Custom Model Evaluations with vLLM (NEW)
  11. Before running the script
  12. Running the script
  13. Features
  14. Prerequisites
Ships with 1 file
  • metadata.json
Commands it runs
uv run scripts/evaluation_manager.py get-prs --repo-id "username/model-name"
uv run scripts/evaluation_manager.py --help
uv run scripts/evaluation_manager.py inspect-tables --help
uv run scripts/evaluation_manager.py extract-readme --help
uv run scripts/train_sft_example.py
uv run scripts/evaluation_manager.py inspect-tables --repo-id "username/model"
uv run scripts/evaluation_manager.py extract-readme \
or
Create .env file
echo "AA_API_KEY=your-api-key" >> .env
More from claude-skill-registry
All skills →
descriptive-statisticsSummarizing and visualizing data through measures of center, spread, shape, and position. Covers…Data & Analyticsperplexity-researcher-reasoning-proHighest level of research and reasoning capabilities for complex decision-making with significant…OtherretroBrutally honest engineering retrospective. Analyzes git history for what actually happened — not…Data & AnalyticsanalyzeUniversal multi-perspective analyzer for any topic, file, idea, or decision. Extract key points…Otherarize-prompt-optimizationINVOKE THIS SKILL when optimizing, improving, or debugging LLM prompts using production trace data…Data & Analyticscellxgene-contextGuide for integrating CellxGene Census single-cell data with ENCODE bulk experiments. Use when…Data & Analytics
About this skill
What does the hugging-face-evaluation skill do?

Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format.

How do I install it?

Run `npx skills add majiayu000/claude-skill-registry --skill hugging-face-evaluation-ihatesea69-hieunghi-ai-skills --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.

Where does this skill come from?

From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.

Is a popular skill a good skill?

Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.

Keep going
All skills →
The full catalog, by category and by tool.
Open Source Radar →
The repositories these skills come from, watched daily.
The people behind them →
Makers and builders across the ecosystem.