align-human
Use when the user has a judge/grader and human-labeled data, and wants to measure how well the judge agrees with humans, detect systematic biases, determine whether automatic evaluation can replace human review, or build a human-reduction roadmap. Also use when the user mentions calibration, TPR/TNR, judge validation, inter-rater agreement, Cohen's kappa, bias detection, or "is my automatic evaluation trustworthy." Merges the calibrate and align functions into one skill.
Profile →npx skills add agentscope-ai/OpenJudge --skill 03-align-human --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
<HARD-GATE> NO calibrated:true WITHOUT TPR >= 0.8 AND TNR >= 0.8 AND boundary stratum TPR >= 0.6 AND TNR >= 0.6 AND n_dev >= 10 per class AND test-set drop < 10%. NO human_reduction_phase >= 2 WITHOUT kappa >= 0.6 AND boundary kappa >= 0.6. NO alignment conclusion WITHOUT all 5 bias checks completed. </HARD-GATE> # Align Human Measure whether your automatic judge agrees with human judgment, detect where and why they disagree, and build a roadmap to reduce human review over time. ## When to Activate - You have a working judge/grader and 50+ human-labeled examples - You want to know if the judge is trustworthy enough to replace human review - You've noticed the judge's decisions being overturned by humans - You're preparing to deploy an evaluation as a production gate ## Checklist You MUST create a task for each item and complete them in order: 1. **Load paired data** — match judge verdicts with human labels 2. **Measure TPR/TNR** — confusion matrix + per-stratum breakdown 3. **Calculate agreement** — Cohen's kappa, Gwet's AC1, systematic bias 4. **Run bias detection** — 5 systematic bias checks 5. **Analyze disagreements** — cluster patterns + diagnose root causes 6. **Build human-r
- When to Activate
- Checklist
- Fast path: run the bundled script
- Step 1: Load and Pair Data
- Step 2: Measure TPR/TNR
- Bootstrap 95% CI
- Step 3: Calculate Agreement
- Cohen's Kappa (chance-corrected agreement)
- Gwet's AC1 (robust to class imbalance)
- Systematic Bias
- Step 4: Five Bias Detection Checks
- Step 5: Disagreement Pattern Analysis
- Step 6: Human-Reduction Roadmap
- Step 7: Confirmation and Output
python scripts/calibration.py --pairs pairs.jsonl # one paired file, OR python scripts/calibration.py --verdicts verdicts.jsonl --labels labels.jsonl --stratum-key difficulty
What does the align-human skill do?
Use when the user has a judge/grader and human-labeled data, and wants to measure how well the judge agrees with humans, detect systematic biases, determine whether automatic evaluation can replace human review, or build a human-reduction roadmap. Also use when the user mentions calibration, TPR/TNR, judge validation, inter-rater agreement, Cohen's kappa, bias detection, or "is my automatic evaluation trustworthy." Merges the calibrate and align functions into one skill.
How do I install it?
Run `npx skills add agentscope-ai/OpenJudge --skill 03-align-human --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From agentscope-ai/OpenJudge, a repository with 764 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.