compare-attempts
This SOP compares evaluated brazil-bench attempts across multiple dimensions to produce a ranked leaderboard and detailed comparison summary. It supports up to 10 attempts in the "Top 10" format, automatically pruning lower-ranked entries when more are added.
npx skills add majiayu000/claude-skill-registry --skill compare-attempts --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
This skill compares evaluated brazil-bench attempts across multiple metrics to build a ranked leaderboard and a detailed summary. It handles up to 10 entries, pruning lower-ranked ones as more are added.
How it works
- Discover evaluation results by scanning the results_dir for md files (excluding LEADERBOARD.md) and ensuring each has a "# Evaluation:" header.
- Extract metrics from each report, including Pattern, Spec Compliance, Lines of Code, Files, Dependencies, Commits, Duration, Fix Commits, Test Scenarios, Token Usage, and Phase Durations.
- Check methodology consistency by comparing denominators (canonical 16/16) and flagging any differences; normalize metrics where necessary as described, using 12/12 → 12/16 for normalization.
- Calculate a composite Score using a weighted formula prioritizing Spec Compliance (50%), Effective Tests (30%), Code Quality (15%), and Efficiency (5%), with explicit definitions for each term.
- Rank attempts by score, tie-break with test counts and fix commits, and limit to max_entries.
- Generate comparison tables including a summary leaderboard, detailed metrics, pattern distribution, and coverage matrices, plus phase analyses from git data.
- Extract initial prompts from prompts.txt, determine orchestration pattern (Hive, Swarm, Solo), and summarize prompt similarities.
- Collect issue counts via GitHub for each attempt repository and categorize issues.
- Produce a final Analysis section with identified winners, patterns, and cross-attempt insights, followed by writing the report to output_file with a generation timestamp and source references.
When to use it
Use when you need a cross-attempt leaderboard and detailed, metric-driven insights for Brazil-Bench evaluations to decide which approach performed best under canonical methodology.
What it can touch
- Results are read from {results_dir} as Markdown reports with a Metrics table, Phase Durations, and Test Summaries.
- It touches GitHub repositories for issue counts and commit history for phase analysis, and prompts.txt for orchestration and token usage data.
Caveats
- Normalization rules require adjusting denominators to 16 when necessary (e.g., 12/12 → 12/16) and tagging with an asterisk in the leaderboard.
- Token and prompt data are taken from manually captured prompts.txt entries and may be incomplete.
- The scoring assumes the canonical 16-requirement checklist; discrepancies are flagged and normalized to that standard.
# Compare Benchmark Attempts ## Overview This SOP compares evaluated brazil-bench attempts across multiple dimensions to produce a ranked leaderboard and detailed comparison summary. It supports up to 10 attempts in the "Top 10" format, automatically pruning lower-ranked entries when more are added. ## Parameters - **results_dir** (optional, default: `./results`): Directory containing evaluation reports - **output_file** (optional, default: `./results/LEADERBOARD.md`): Output comparison file - **max_entries** (optional, default: 10): Maximum entries in leaderboard ## Steps ### 1. Discover Evaluation Results Find all completed evaluation reports in the results directory. **Constraints:** - You MUST scan for `*.md` files in the results directory (excluding LEADERBOARD.md) - You MUST verify each file is a valid evaluation report (has "# Evaluation:" header) - You MUST extract the attempt name from each report - You SHOULD skip any malformed or incomplete reports ```bash ls {results_dir}/*.md | grep -v LEADERBOARD.md ``` ### 2. Extract Metrics from Each Report Parse each evaluation report to extract comparable metrics. **Constraints:** - You MUST extract all metrics from the Metrics ta
- Overview
- Parameters
- Steps
- 1. Discover Evaluation Results
- 2. Extract Metrics from Each Report
- 2a. Check for Evaluation Methodology Differences
- 3. Calculate Ranking Score
- 4. Rank and Sort Attempts
- 5. Generate Comparison Tables
- 6. Analyze Development Phases
- 7. Extract Initial Prompts
- 8. Analyze Token Usage
- 9. Collect Issue Counts
- 10. Generate Analysis
ls {results_dir}/*.md | grep -v LEADERBOARD.md
Check spec compliance denominators across all evaluations
grep -h "Spec Compliance" {results_dir}/*.md | grep -v LEADERBOARD
Look for different requirement counts
grep -E "requirements|/16|/12|/15" {results_dir}/*.md
Get issue counts for an attempt
gh issue list -R brazil-bench/{attempt_repo} --state open --json number -q 'length'
gh issue list -R brazil-bench/{attempt_repo} --state closed --json number -q 'length'
Get issue details with titles
gh issue list -R brazil-bench/{attempt_repo} --state all --json number,title,stateWhat does the compare-attempts skill do?
This SOP compares evaluated brazil-bench attempts across multiple dimensions to produce a ranked leaderboard and detailed comparison summary. It supports up to 10 attempts in the "Top 10" format, automatically pruning lower-ranked entries when more are added.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill compare-attempts --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.