hugging-face-evaluation
Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format.
Profile →npx skills add sickn33/agentic-awesome-skills --skill hugging-face-evaluation --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Adds and manage evaluation results in Hugging Face model cards by supporting three methods: extracting existing evaluation tables from README content, importing benchmark scores from Artificial Analysis, and running custom model evaluations with vLLM or lighteval backends. It updates model-index metadata to support leaderboard integration and compatibility with Papers with Code.
How it works
- Inspect and extract evaluation tables from a README using a workflow that parses Markdown tables with markdown-it-py, ignores code blocks, and converts a selected table into model-index YAML. It supports specifying a table number and model column hints, with task.type set via a flag (e.g., text-generation, summarization).
- Import from Artificial Analysis by fetching benchmark scores via API, converting responses to model-index format, preserving metadata like source attribution and URLs, and optionally creating a PR.
- Model-index management to generate properly formatted YAML entries, merge updates into existing model cards, and validate against Papers with Code specifications. Supports batch processing.
- Run evaluations on Hugging Face Jobs or locally using vLLM or accelerate backends. Includes options for hardware selection, secret handling, and safe token management.
- Provide methods to run evaluation jobs via HF infrastructure, using uv run or hf jobs uv run, and include a Python helper for submission.
When to use it
- You need to add structured evaluation results to a Hugging Face model card.
- You want to import benchmark data or run custom evaluations with vLLM, lighteval, or inspect-ai.
- You are preparing leaderboard-compatible model-index metadata for a model release.
What it can touch
- Tools referenced:
uv(via evaluation_manager scripts),hf jobs, and Python scripts underscripts/. - It requires environment variables like
HF_TOKENand, for Artificial Analysis,AA_API_KEY.
Caveats
- Risk declared as critical. Ensure PR checks and existing PRs are handled before creating new PRs using the provided workflow commands.
- vLLM evaluation requires GPU hardware and specific backend/dependency setups (e.g.,
lighteval,vllm, and related packages). - Validation relies on compatibility with Papers with Code specifications and proper formatting of model-index YAML.
# Overview This skill provides tools to add structured evaluation results to Hugging Face model cards. It supports multiple methods for adding evaluation data: - Extracting existing evaluation tables from README content - Importing benchmark scores from Artificial Analysis - Running custom model evaluations with vLLM or accelerate backends (lighteval/inspect-ai) ## When to Use - You need to add structured evaluation results to a Hugging Face model card. - You want to import benchmark data or run custom evaluations with vLLM, lighteval, or inspect-ai. - You are preparing leaderboard-compatible `model-index` metadata for a model release. ## Integration with HF Ecosystem - **Model Cards**: Updates model-index metadata for leaderboard integration - **Artificial Analysis**: Direct API integration for benchmark imports - **Papers with Code**: Compatible with their model-index specification - **Jobs**: Run evaluations directly on Hugging Face Jobs with `uv` integration - **vLLM**: Efficient GPU inference for custom model evaluation - **lighteval**: HuggingFace's evaluation library with vLLM/accelerate backends - **inspect-ai**: UK AI Safety Institute's evaluation framework # Version 1.3.0
- When to Use
- Integration with HF Ecosystem
- Core Dependencies
- Inference Provider Evaluation
- vLLM Custom Model Evaluation (GPU required)
- ⚠️ CRITICAL: Check for Existing PRs Before Creating New Ones
- 1. Inspect and Extract Evaluation Tables from README
- 2. Import from Artificial Analysis
- 3. Model-Index Management
- 4. Run Evaluations on HF Jobs (Inference Providers)
- 5. Run Custom Model Evaluations with vLLM (NEW)
- Before running the script
- Running the script
- Features
uv run scripts/evaluation_manager.py get-prs --repo-id "username/model-name"
uv run scripts/evaluation_manager.py --help
uv run scripts/evaluation_manager.py inspect-tables --help
uv run scripts/evaluation_manager.py extract-readme --help
uv run scripts/train_sft_example.py
uv run scripts/evaluation_manager.py inspect-tables --repo-id "username/model"
uv run scripts/evaluation_manager.py extract-readme \
or
env "AA_API_KEY=${AA_API_KEY:?set AA_API_KEY first}" uv run scripts/evaluation_manager.py import-aa \
Create .env fileWhat does the hugging-face-evaluation skill do?
Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format.
How do I install it?
Run `npx skills add sickn33/agentic-awesome-skills --skill hugging-face-evaluation --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From sickn33/agentic-awesome-skills, a repository with 44,414 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.