evaluation-methodology
PluginEval quality methodology — dimensions, rubrics, statistical methods, and scoring formulas. Use this skill when understanding how plugin quality is measured, when interpreting a low score on a specific dimension, when deciding how to improve a skill's triggering accuracy or orchestration fitness, when calibrating scoring thresholds for your marketplace, or when explaining quality badges to external partners like Neon.
Profile →npx skills add wshobson/agents --skill evaluation-methodology --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Conveys how PluginEval measures quality across three evaluation layers (Static Analysis, LLM Judge, Monte Carlo Simulation), the ten scoring dimensions, and how a final composite score is computed with anti-pattern penalties and Elo ranking. It also defines badge thresholds, anti-pattern flags, and guidance for interpreting dimension scores and improving a skill's quality and triggering accuracy.
How it works
- The skill defines three evaluation layers, each producing a score 0.0 to 1.0 per applicable dimension; Layer 1 uses static analysis with six sub-checks (frontmatter_quality, orchestration_wiring, progressive_disclosure, structural_completeness, token_efficiency, ecosystem_coherence).
- Layer 2 uses an LLM judge to score four dimensions (Triggering accuracy, Orchestration fitness, Output quality, Scope calibration) via anchored rubrics; returns a structured JSON merged into the composite.
- Layer 3 runs a Monte Carlo simulation (N=50 by default) to compute activation_rate, output_consistency, failure_rate, and token_efficiency, then computes mc_score with a defined formula.
- The final composite score blends dimensions with weights across all three layers, applying an anti-pattern penalty in the range down to 0.5 and scaling to a percentage.
- Dimension weights and layer blend weights are laid out, along with interpretation (A–F grades) and badge thresholds (Platinum, Gold, Silver, Bronze) that depend on composite and Elo thresholds when available.
- There are commands and scripts described for scoring and certifying skills, as well as a scripting example reproducing the composite formula and an example of a JSON output format for CI integration.
When to use it
Use when understanding how plugin quality is measured, interpreting a low score on a specific dimension, deciding how to improve a skill's triggering accuracy or orchestration fitness, calibrating scoring thresholds for a marketplace, or explaining quality badges to external partners like Neon.
What it can touch
- References to the evaluation pipeline components and scoring formulas. Commands for scoring and certification are provided (e.g.,
plugin-eval score,plugin-eval certify,plugin-eval init). - The methodology includes file paths (e.g.,
references/rubrics.md) and example code snippets to reproduce scoring offline.
Caveats
- Describes a multi-layer evaluation system with potential variability depending on depth and the use of Monte Carlo simulations.
- Uses Elo ranking with a starting rating of 1500 and a K-factor of 32, including bootstrap CI reporting and cross-order comparisons.
- Includes anti-pattern definitions and prescribed fixes for them (e.g., OVER_CONSTRAINED, EMPTY_DESCRIPTION, MISSING_TRIGGER, BLOATED_SKILL, ORPHAN_REFERENCE, DEAD_CROSS_REF).
# Evaluation Methodology This document is the authoritative reference for how PluginEval measures plugin and skill quality. It covers the three evaluation layers, all ten scoring dimensions, the composite formula, badge thresholds, anti-pattern flags, Elo ranking, and actionable improvement tips. Related: [Full rubric anchors](references/rubrics.md) --- ## The Three Evaluation Layers PluginEval stacks three complementary layers. Each layer produces a score between 0.0 and 1.0 for each applicable dimension, and later layers override or blend with earlier ones according to per-dimension blend weights. ### Layer 1 — Static Analysis **Speed:** < 2 seconds. No LLM calls. Deterministic. The static analyzer (`layers/static.py`) runs six sub-checks directly against the parsed SKILL.md: | Sub-check | What it measures | |---|---| | `frontmatter_quality` | Name presence, description length, trigger-phrase quality | | `orchestration_wiring` | Output/input documentation, code block count, orchestrator anti-pattern | | `progressive_disclosure` | Line count vs. sweet-spot (200–600 lines), references/ and assets/ bonuses | | `structural_completeness` | Heading density, code blocks, examples sectio
- The Three Evaluation Layers
- Layer 1 — Static Analysis
- Layer 2 — LLM Judge
- Layer 3 — Monte Carlo Simulation
- Composite Scoring Formula
- Dimension Weights
- Layer Blend Weights
- Blended Score Calculation
- Interpreting Dimension Scores
- Quality Badges
- Anti-Pattern Flags
- OVERCONSTRAINED
- EMPTYDESCRIPTION
- MISSINGTRIGGER
plugin-eval init ./plugins --corpus-dir ~/.plugineval/corpus plugin-eval score ./path/to/skill --depth quick plugin-eval score ./path/to/skill plugin-eval score ./path/to/skill --output json plugin-eval score ./path/to/skill --depth quick --output json --threshold 70 exits with code 1 if score < 70 plugin-eval certify ./path/to/skill plugin-eval compare ./skill-a ./skill-b plugin-eval init ./plugins if (( $(echo "$score < 70" | bc -l) )); then
What does the evaluation-methodology skill do?
PluginEval quality methodology — dimensions, rubrics, statistical methods, and scoring formulas. Use this skill when understanding how plugin quality is measured, when interpreting a low score on a specific dimension, when deciding how to improve a skill's triggering accuracy or orchestration fitness, when calibrating scoring thresholds for your marketplace, or when explaining quality badges to external partners like Neon.
How do I install it?
Run `npx skills add wshobson/agents --skill evaluation-methodology --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From wshobson/agents, a repository with 38,479 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.