openjudge
Build custom LLM evaluation pipelines using the OpenJudge framework. Covers selecting and configuring graders (LLM-based, function-based, agentic), running batch evaluations with GradingRunner, combining scores with aggregators, applying evaluation strategies (voting, average), auto-generating graders from data, and analyzing results (pairwise win rates, statistics, validation metrics). Use when the user wants to evaluate LLM outputs, compare multiple models, design scoring criteria, or build an automated evaluation system.
Profile →npx skills add agentscope-ai/OpenJudge --skill openjudge --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# OpenJudge Skill Build evaluation pipelines for LLM applications using the `openjudge` library. ## When to Use This Skill - User wants to evaluate LLM output quality (correctness, relevance, hallucination, etc.) - User wants to compare two or more models and rank them - User wants to design a scoring rubric and automate evaluation - User wants to analyze evaluation results statistically - User wants to build a reward model or quality filter ## Sub-documents — Read When Relevant | Topic | File | Read when… | |-------|------|------------| | Grader selection & configuration | `graders.md` | User needs to pick or configure an evaluator | | Batch evaluation pipeline | `pipeline.md` | User needs to run evaluation over a dataset | | Auto-generate graders from data | `generator.md` | No rubric yet; generate from labeled examples | | Analyze & compare results | `analyzer.md` | User wants win rates, statistics, or metrics | Read the relevant sub-document **before** writing any code. ## Install ```bash pip install py-openjudge ``` ## Architecture Overview ``` Dataset (List[dict]) │ ▼ GradingRunner ← orchestrates everything │ ├─► Grader A ──► EvaluationStrategy ──► _aevaluate() ──► GraderScor
- When to Use This Skill
- Sub-documents — Read When Relevant
- Install
- Architecture Overview
- 5-Minute Quick Start
- Key Data Types
- Result Handling Pattern
- Model Configuration
pip install py-openjudge
What does the openjudge skill do?
Build custom LLM evaluation pipelines using the OpenJudge framework. Covers selecting and configuring graders (LLM-based, function-based, agentic), running batch evaluations with GradingRunner, combining scores with aggregators, applying evaluation strategies (voting, average), auto-generating graders from data, and analyzing results (pairwise win rates, statistics, validation metrics). Use when the user wants to evaluate LLM outputs, compare multiple models, design scoring criteria, or build an automated evaluation system.
How do I install it?
Run `npx skills add agentscope-ai/OpenJudge --skill openjudge --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From agentscope-ai/OpenJudge, a repository with 764 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.