paper-writing-bench
Reverse-engineer raw materials (Sparse idea, Dense idea, experimental log) from an existing AI research paper to build a benchmark case for evaluating paper-writing pipelines. Replicates the PaperWritingBench dataset construction procedure from arXiv:2604.05018 §3 / App. C. TRIGGER when the user asks to "build a benchmark case from this paper", "reverse-engineer raw materials", or "evaluate my pipeline against PaperWritingBench".
npx skills add Ar9av/PaperOrchestra --skill paper-writing-bench --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# PaperWritingBench (§3) Faithful implementation of the PaperWritingBench dataset construction procedure from PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §3 and App. C, F.2). The original benchmark contains 200 papers (100 CVPR 2025 + 100 ICLR 2025). For each paper, the authors reverse-engineer the (I, E) tuple by stripping narrative flow from the original PDF using the three prompts in App. F.2. You can use this skill to reverse-engineer your own benchmark cases from any paper PDF. ## What this skill does Given an existing AI research paper (PDF or markdown extract), produce: - `idea.md` (Sparse variant) — high-level concept note, no math, no experimental results - `idea.md` (Dense variant) — detailed technical proposal with LaTeX equations and variable definitions, but still no experimental results - `experimental_log.md` — exhaustive raw experimental setup, numeric data, and qualitative observations, with all narrative references stripped These three files form a complete (I, E) input pair for the paper-orchestra pipeline. You can then run the pipeline and compare its output to the original paper using `paper-autoraters`. ## Inputs - A paper PDF or extracted markdown te
- What this skill does
- Inputs
- Outputs
- Workflow
- 1. Sparse idea generation
- 2. Dense idea generation
- 3. Experimental log generation
- Critical rules from the prompts
- How the bench is used
- Resources
What does the paper-writing-bench skill do?
Reverse-engineer raw materials (Sparse idea, Dense idea, experimental log) from an existing AI research paper to build a benchmark case for evaluating paper-writing pipelines. Replicates the PaperWritingBench dataset construction procedure from arXiv:2604.05018 §3 / App. C. TRIGGER when the user asks to "build a benchmark case from this paper", "reverse-engineer raw materials", or "evaluate my pipeline against PaperWritingBench".
How do I install it?
Run `npx skills add Ar9av/PaperOrchestra --skill paper-writing-bench --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From Ar9av/PaperOrchestra, a repository with 627 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.