review-pap

What it does

The skill coordinates a six-agent, parallel review of a pre-analysis plan (PAP) for a given registration target or journal. It parses the input to determine TARGET_REGISTRY and the main PAP file, auto-detects PAP files when none is provided, identifies supporting documents, and collects study metadata (title, PI, abstract) if available. It then launches six agents in parallel, passing the full PAP and supporting file paths to each agent, ensuring agent 6 sees TARGET_REGISTRY substituted in its prompt. It emphasizes structured, signaled issue tagging and a clearly formatted output for consolidation.

How it works

• Phase 1: Parse Arguments and Discover the PAP
  • Recognizes registration targets (Trial registries, Journal standards, General standards) case-insensitively and splits ARGUMENTS into TARGET_REGISTRY and main PAP file path when possible.
  • If no match, treats ARGUMENTS as a file path with TARGET_REGISTRY = top-journal. If ARGUMENTS is empty, defaults to no file path and TARGET_REGISTRY = top-journal.
  • If a path is provided but unreadable or not main PAP, falls back to auto-detection and notes the fallback in the report.
  • Auto-detection scans the repo for likely PAP files (*.md, *.txt, *.tex, *.docx, *.pdf), prioritizes names hinting at PAP content, identifies the main PAP (favoring documents with hypotheses, outcomes, and analysis specifications), and reads for references to supporting documents (power calculations, survey instruments, randomization, code skeletons, data dictionaries, IRB protocols).
  • Searches recursively for supporting files by keywords (power/sample, instruments, randomization, code, ethics) and records their paths and roles.
  • Records: main PAP file path, supporting file paths, study title, PI(s)/team, abstract or research question, registry/ID, and any missing expected categories. Notes binary formats if unreadable.
• Phase 2: Launch 6 Review Agents in Parallel
  • Uses Agent tool with subagent_type: "general-purpose" to run 6 agents in parallel.
  • Each agent receives the complete list of PAP and supporting file paths. Agent 6 receives the prompt with TARGET_REGISTRY substituted for every occurrence in its prompt text.

AGENT 1 — Clarity, Writing Quality & Pre-specification Completeness

• Reviews prose for clarity, precision, and whether the PAP clearly states research questions, design, outcomes, analysis strategy, sample, timeline, and predefined subgroup/heterogeneity analyses.
• Flags potential vagueness, writing issues, and compliance signals. Tags issues with [CRITICAL], [MAJOR], or [MINOR].
• Outputs in a structured format listing Critical Vagueness, Minor Writing Issues, and Structural/Compliance Signals, referencing the listed PAP files.

AGENT 2 — Internal Consistency, Hypotheses & Outcomes

• Checks coherence between hypotheses and outcomes, primary vs secondary outcomes, data sources for outcomes, subgroup definitions, alignment of analysis with design, timeline consistency, terminology consistency, and cross-document consistency.
• Tags issues with appropriate severity and outputs a structured report section.

AGENT 3 — Identification Strategy, Causal Claims & Contribution

• Evaluates research question clarity, identification strategy (randomization, quasi-experiments, or observational assumptions), testability of hypotheses, external validity, contribution to literature, balance of overclaiming/underclaiming, and fit to TARGET_REGISTRY expectations.
• Provides structured findings with severity labels.

AGENT 4 — Statistical Analysis Plan, Power & Multiple Testing

• Assesses power calculations for primary and secondary outcomes, estimator specification, multiple testing corrections, missing data handling, robustness checks, outcome construction, and stopping/adaptation rules.
• Returns findings with severity labels and suggested improvements.

AGENT 5 — Data, Sample, Implementation & Operational Plan

• Examines sample definition, sampling frame, randomization procedures, data sources and accessibility, data collection timeline, baseline and endline timing, and feasibility aspects for operation.
• Focuses on whether the plan is executable given the data needs and timelines.

Output

• Each agent provides a structured section with its respective headings and issue lists, all referencing only the facts stated in the skill prompt. The consolidated report will be used for decision-making on installing the skill.