keyword-based-llm-eval
Use when evaluating LLM-generated structured output against expected results using keyword matching and F1 metrics.
npx skills add majiayu000/claude-skill-registry --skill keyword-based-llm-eval --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Keyword-Based LLM Output Evaluation **Extracted:** 2026-02-10 **Context:** Evaluating structured LLM output (cards, summaries, extractions) against expected results without exact match or expensive semantic similarity. ## Problem LLM generates structured output (e.g., Anki cards with front/back text, card types, tags). Need to measure prompt quality quantitatively — but: - Exact match is too strict (LLM wording varies) - Semantic similarity (embeddings) is expensive and adds dependency - Manual review doesn't scale ## Solution Keyword presence-based lightweight matching with greedy best-match: ### 1. Define Expected Output as Keywords ```python @dataclass(frozen=True, slots=True) class ExpectedCard: front_keywords: list[str] # must appear in generated front back_keywords: list[str] # must appear in generated back card_type: CardType | None = None # optional type constraint ``` ### 2. Keyword Similarity Score ```python def _keyword_similarity(keywords: list[str], text: str) -> float: if not keywords: return 0.0 found = sum(1 for kw in keywords if kw in text) return found / len(keywords) ``` ### 3. Weighted Pair Scoring ```python def _score_pair(expected, card) -> float: front_sim
- Problem
- Solution
- 1. Define Expected Output as Keywords
- 2. Keyword Similarity Score
- 3. Weighted Pair Scoring
- 4. Greedy Best-Match Algorithm
- 5. Aggregate Metrics
- 6. YAML Dataset Format
- Architecture
- When to Use
- Trade-offs
What does the keyword-based-llm-eval skill do?
Use when evaluating LLM-generated structured output against expected results using keyword matching and F1 metrics.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill keyword-based-llm-eval --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.