hugging-face-community-evals
Run local evaluations for Hugging Face Hub models with inspect-ai or lighteval.
npx skills add majiayu000/claude-skill-registry --skill hugging-face-community-evals --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Overview ## When to Use Use this skill for local model evaluation, backend selection, and GPU smoke tests outside the Hugging Face Jobs workflow. This skill is for **running evaluations against models on the Hugging Face Hub on local hardware**. It covers: - `inspect-ai` with local inference - `lighteval` with local inference - choosing between `vllm`, Hugging Face Transformers, and `accelerate` - smoke tests, task selection, and backend fallback strategy It does **not** cover: - Hugging Face Jobs orchestration - model-card or `model-index` edits - README table extraction - Artificial Analysis imports - `.eval_results` generation or publishing - PR creation or community-evals automation If the user wants to **run the same eval remotely on Hugging Face Jobs**, hand off to the `hugging-face-jobs` skill and pass it one of the local scripts in this skill. If the user wants to **publish results into the community evals workflow**, stop after generating the evaluation run and hand off that publishing step to `~/code/community-evals`. > All paths below are relative to the directory containing this `SKILL.md`. # When To Use Which Script | Use case | Script | |---|---| | Local `inspect-ai
- When to Use
- Option A: inspect-ai with local inference providers path
- Option B: inspect-ai on Local GPU
- Option C: lighteval on Local GPU
- Limitations
uv --version printenv HF_TOKEN >/dev/null nvidia-smi uv run scripts/inspect_eval_uv.py \ uv run scripts/inspect_vllm_uv.py \ uv run scripts/lighteval_vllm_uv.py \
What does the hugging-face-community-evals skill do?
Run local evaluations for Hugging Face Hub models with inspect-ai or lighteval.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill hugging-face-community-evals --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.