evaluating-with-leakage-gates
Evaluate an OpenMed de-identification or clinical NER model against the leakage-first release gates G1a through G8, which gate releases on residual PHI leakage rather than on F1. Use when the user wants to run the OpenMed eval harness on a synthetic golden set, decide whether a de-id model is RELEASABLE or QUARANTINED, enforce direct-identifier recall floors, require zero critical leakage, fit calibration thresholds, or produce a signed gate report. Trigger on \"release gate\", \"leakage\", \"is this model safe to ship\", \"G1a\", \"G3\", \"quarantine\", \"recall floor\", or \"calibration thre
npx skills add maziyarpanahi/openmed --skill evaluating-with-leakage-gates --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Evaluating with Leakage Gates OpenMed's release gates answer one question: **did any PHI leak?** A de-id model with a beautiful F1 can still leak a single SSN — and that one leak is a HIPAA breach. So `openmed.eval` gates on *residual leakage* and *per-label recall floors*, not on aggregate F1. The candidate is either `RELEASABLE` or `QUARANTINED`; there is no partial credit. ## When to use this skill - You have a candidate de-id or PII model and need a ship / no-ship decision. - You want to run the benchmark harness over a **synthetic** golden suite. - You need to enforce direct-identifier recall floors and `critical_leakage == 0`. - You need calibration thresholds (`thresholds.json`) before the gate will pass. - You want a signed, reproducible gate report for governance. This is the flagship eval skill. For a pure NER scorecard see `benchmarking-clinical-ner`; for CI wiring see `gating-deid-leakage`. ## The gates (G1a–G8) | Gate | Checks | Floor / rule | | --- | --- | --- | | **G1a** | Direct & quasi identifiers (PERSON, EMAIL, PHONE, SSN, ID_NUM, DATE_OF_BIRTH, ...) | recall ≥ 0.990 (v1.6) / 0.995 (v2.0); strict-no-leak policies raise the floor | | **G1b** | Structured secrets
- When to use this skill
- The gates (G1a–G8)
- Quick start
- Workflow
- Hand-off to / from OpenMed
- Edge cases & gotchas
- Standards & references
python -m openmed.eval.release_gates \
What does the evaluating-with-leakage-gates skill do?
Evaluate an OpenMed de-identification or clinical NER model against the leakage-first release gates G1a through G8, which gate releases on residual PHI leakage rather than on F1. Use when the user wants to run the OpenMed eval harness on a synthetic golden set, decide whether a de-id model is RELEASABLE or QUARANTINED, enforce direct-identifier recall floors, require zero critical leakage, fit calibration thresholds, or produce a signed gate report. Trigger on \"release gate\", \"leakage\", \"is this model safe to ship\", \"G1a\", \"G3\", \"quarantine\", \"recall floor\", or \"calibration thre
How do I install it?
Run `npx skills add maziyarpanahi/openmed --skill evaluating-with-leakage-gates --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From maziyarpanahi/openmed, a repository with 4,851 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.