ieeesp-experiments
Use when designing or auditing the evaluation of an IEEE S&P (Oakland) paper, including end-to-end attack demonstration, adaptive-adversary evaluation of defenses, measurement sampling and validity, baselines and ablations, statistical reporting of attack success, and the ethics constraints that shape what experiments are permissible.
Profile →npx skills add brycewang-stanford/Awesome-Journal-Skills --skill ieeesp-experiments --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# IEEE S&P Experiments Use this to build or audit the evidence an S&P paper stands on. The venue's reviewers are professional adversaries; an evaluation that would satisfy a systems or ML PC often leaves an Oakland reviewer's central question — "does this survive a real, adaptive attacker?" — unanswered. ## Match the evaluation to the contribution type | Contribution | Evaluation that closes the loop | Fatal gap | |---|---|---| | Attack | End-to-end demonstration against a realistic, current target | Toy target; unrealistic preconditions | | Defense | Adaptive adversary who knows the design; cost/overhead | Only non-adaptive or prior attacks | | Measurement | Representative sampling + validation + ethics | Convenience sample presented as population | | System | Security property demonstrated *and* performance | Property asserted, not tested | | SoK | Systematic re-analysis under one framework | Cherry-picked coverage | ## The adaptive-adversary rule dominates defense papers A defense evaluated only against existing or non-adaptive attacks is the most common S&P defense rejection. The standard: - Define the adaptive adversary explicitly: knows the mechanism, the parameters, and the
- Match the evaluation to the contribution type
- The adaptive-adversary rule dominates defense papers
- Measurement validity is an evidence question and an ethics question
- Statistics for attacks and fuzzing
- Baselines and ablations Oakland reviewers ask for
- Ethics as an experimental design constraint, not an afterthought
- Audit worksheet
- Output format
What does the ieeesp-experiments skill do?
Use when designing or auditing the evaluation of an IEEE S&P (Oakland) paper, including end-to-end attack demonstration, adaptive-adversary evaluation of defenses, measurement sampling and validity, baselines and ablations, statistical reporting of attack success, and the ethics constraints that shape what experiments are permissible.
How do I install it?
Run `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill ieeesp-experiments --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From brycewang-stanford/Awesome-Journal-Skills, a repository with 984 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.