benchmark-pii-recall
Benchmark an OpenMed PII model with synthetic gold spans and report label-aware exact-span and grapheme recall without emitting identifier surfaces. Use when an agent must compare a model, threshold, backend, or quantized artifact and enforce a recall floor before release.
npx skills add maziyarpanahi/openmed --skill benchmark-pii-recall --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Benchmark PII recall Measure PII recall before optimizing F1, size, or latency. A missed direct identifier is a privacy failure even when aggregate F1 improves. ## Procedure 1. Build synthetic fixtures with exact offsets and canonical PII labels. 2. Include direct identifiers, boundary cases, languages/scripts, and the target device or quantization. 3. Run `extract_pii` at the candidate threshold. 4. Normalize prediction labels and score each document separately. 5. Aggregate counts only; do not persist raw text or identifier surfaces. 6. Fail the release when the recall floor or zero-critical-leak requirement is not met. ## Runnable synthetic benchmark Install the model runtime first with `python -m pip install "openmed[hf]"`. ```python from openmed import extract_pii from openmed.core.labels import normalize_label from openmed.eval import compute_character_recall, compute_exact_span_f1 MODEL = "OpenMed/OpenMed-PII-SuperClinical-Small-44M-v1" RECALL_FLOOR = 0.99 FIXTURES = [ { "text": ( "Call the synthetic clinic at 212-555-0198 or email " "demo.patient@example.test." ), "spans": [ ("PHONE", "212-555-0198"), ("EMAIL", "demo.patient@example.test"), ], }, { "text": ( "The syntheti
- Procedure
- Runnable synthetic benchmark
- Release gates
- Repository example
What does the benchmark-pii-recall skill do?
Benchmark an OpenMed PII model with synthetic gold spans and report label-aware exact-span and grapheme recall without emitting identifier surfaces. Use when an agent must compare a model, threshold, backend, or quantized artifact and enforce a recall floor before release.
How do I install it?
Run `npx skills add maziyarpanahi/openmed --skill benchmark-pii-recall --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From maziyarpanahi/openmed, a repository with 4,851 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.