hugging-face-community-evals
Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals...
Profile →npx skills add sickn33/agentic-awesome-skills --skill hugging-face-community-evals --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Overview ## When to Use Use this skill when you need run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals... This skill is for **running evaluations against models on the Hugging Face Hub on local hardware**. It covers: - `inspect-ai` with local inference - `lighteval` with local inference - choosing between `vllm`, Hugging Face Transformers, and `accelerate` - smoke tests, task selection, and backend fallback strategy It does **not** cover: - Hugging Face Jobs orchestration - model-card or `model-index` edits - README table extraction - Artificial Analysis imports - `.eval_results` generation or publishing - PR creation or community-evals automation If the user wants to **run the same eval remotely on Hugging Face Jobs**, hand off to the `hugging-face-jobs` skill and pass it one of the local scripts in this skill. If the user wants to **publish results into the community evals workflow**, stop after generating the evaluation run and hand off that pub
- When to Use
- Option A: inspect-ai with local inference providers path
- Option B: inspect-ai on Local GPU
- Option C: lighteval on Local GPU
- Limitations
uv --version printenv HF_TOKEN >/dev/null nvidia-smi uv run scripts/inspect_eval_uv.py \ uv run scripts/inspect_vllm_uv.py \ uv run scripts/lighteval_vllm_uv.py \
What does the hugging-face-community-evals skill do?
Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM / Transformers / accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals...
How do I install it?
Run `npx skills add sickn33/agentic-awesome-skills --skill hugging-face-community-evals --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From sickn33/agentic-awesome-skills, a repository with 44,414 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.