redteam
Use when the user wants to test their LLM/agent application for safety and security vulnerabilities — jailbreaks, prompt injection, PII extraction, harmful content generation, or evaluator gaming. Also use when the user mentions security testing, adversarial testing, red teaming, safety evaluation, ASR (Attack Success Rate), or "is my app safe to deploy." Outputs ASR paired with over-refusal rate and an audit document.
Profile →npx skills add agentscope-ai/OpenJudge --skill 07-redteam --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
<HARD-GATE> NO ASR report WITHOUT paired over-refusal rate measurement. NO attack vector distribution WITHOUT reading a policy document first. NO regulated-stakes redteam WITHOUT a sign-off block in the audit document. </HARD-GATE> # Redteam Test your application's safety boundaries systematically. This skill generates attack prompts from a policy document, measures what gets through, and pairs the Attack Success Rate (ASR) with the Over-Refusal Rate so you don't reward models that simply refuse everything. ## When to Activate - Pre-deployment safety audit - Regulatory compliance check - After major model or prompt changes that could affect safety - User reports a jailbreak or injection vulnerability - Periodic (quarterly) safety re-evaluation ## Checklist You MUST create a task for each item and complete them in order: 1. **Establish policy** — what's prohibited, what's allowed, what's gray 2. **Derive attack vectors from policy** — not a fixed template 3. **Generate attack set + over-refusal set** — paired measurement 4. **Run attacks and evaluate** — ASR per policy category 5. **Compute over-refusal rate** — paired with ASR 6. **Output safety scorecard + audit document** ## Fast
- When to Activate
- Checklist
- Fast path: run the bundled script
- Step 1: Establish Policy
- Step 2: Derive Attack Vectors from Policy
- Attack vector catalog (choose based on policy needs)
- Example: Deriving distribution from policy
- Step 3: Generate Attack and Over-Refusal Sets
- Attack generation
- Over-refusal set
- Step 4: Run Attacks
- Step 5: Compute Metrics
- ASR (Attack Success Rate) per policy category and attack vector
- Over-Refusal Rate
python scripts/asr_report.py --attacks attacks.jsonl --over-refusals over.jsonl
What does the redteam skill do?
Use when the user wants to test their LLM/agent application for safety and security vulnerabilities — jailbreaks, prompt injection, PII extraction, harmful content generation, or evaluator gaming. Also use when the user mentions security testing, adversarial testing, red teaming, safety evaluation, ASR (Attack Success Rate), or "is my app safe to deploy." Outputs ASR paired with over-refusal rate and an audit document.
How do I install it?
Run `npx skills add agentscope-ai/OpenJudge --skill 07-redteam --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From agentscope-ai/OpenJudge, a repository with 764 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.