create-skill-test
Scaffolds eval.yaml evaluation specs for agent skills in the dotnet/skills repository. Use when creating skill tests, writing evaluation stimuli, defining graders and rubrics, sizing an eval for statistical power, or setting up test fixture files. Handles the Vally eval.yaml schema, fixture organization, and overfitting avoidance. Do not use for running or debugging existing evals (use improve-skill-quality) nor for skills authoring (use create-skill).
npx skills add dotnet/skills --skill create-skill-test --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Create Skill Test Scaffold an evaluation spec (`eval.yaml`) for a skill or agent so it conforms to the Vally schema, passes `skill-validator check` and `check_eval_quality.py`, is powerful enough to return a verdict, and does not overfit to the skill's own wording. ## When to Use - Creating a new `eval.yaml` for a skill or agent - Adding stimuli to an existing eval - Sizing an eval so the pass gate can actually be reached - Setting up or repairing fixture files alongside an eval - Reviewing whether rubric items and graders risk overfitting ## When Not to Use - Diagnosing a failing or regressed eval — use `improve-skill-quality` - Modifying the skill-validator or the evaluation workflows - Creating or editing `SKILL.md` files — use `create-skill` ## Inputs | Input | Required | Description | |-------|----------|-------------| | Skill or agent name | Yes | Must exist under `plugins/<plugin>/skills/` or `plugins/<plugin>/agents/` | | Plugin name | Yes | e.g. `dotnet-msbuild` | | Skill content | Yes | Read it — you cannot write non-overfitted rubric items without it | | Failure modes to discriminate | Recommended | Each becomes one stimulus | ## Workflow ### Step 1: Locate the target
- When to Use
- When Not to Use
- Inputs
- Workflow
- Step 1: Locate the target and the test directory
- Step 2: Write the spec skeleton
- Step 3: Size the eval for power before writing content
- Step 4: Write stimuli
- Step 5: Configure the environment
- Step 6: Write graders
- Step 7: Write rubric items
- Step 8: Add constraints sparingly
- Step 9: Add dormancy guards
- Step 10: Validate
dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- check --plugin ./plugins/<plugin> python eng/eval-quality/check_eval_quality.py copy dotnet-skills.experiment.yaml, widen its evals: glob to tests/*/agent.*/eval.yaml
What does the create-skill-test skill do?
Scaffolds eval.yaml evaluation specs for agent skills in the dotnet/skills repository. Use when creating skill tests, writing evaluation stimuli, defining graders and rubrics, sizing an eval for statistical power, or setting up test fixture files. Handles the Vally eval.yaml schema, fixture organization, and overfitting avoidance. Do not use for running or debugging existing evals (use improve-skill-quality) nor for skills authoring (use create-skill).
How do I install it?
Run `npx skills add dotnet/skills --skill create-skill-test --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From dotnet/skills, a repository with 4,927 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.