colm-experiments
Use when designing or auditing the empirical core of a COLM paper — contamination analysis for evaluation data, fair baselines under matched prompting and compute, pinned model versions and decoding parameters, uncertainty over runs and samples, scaling coverage, and honest reporting of API-model comparisons.
Profile →npx skills add brycewang-stanford/Awesome-Journal-Skills --skill colm-experiments --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# COLM Experiments COLM's reviewer pool works on language models daily, which changes what "rigorous" means: the failure modes they hunt are not missing error bars alone but the LM-specific ones — leaked test sets, un-tuned baselines, drifting API models, and decoding settings that quietly decide the result. Build the evaluation so those four attacks fail. ## The four LM-specific attacks on your results **1. Contamination.** Any public benchmark plausibly overlaps the pre-training data of any recent model. For each evaluation set, either (a) run and report an overlap analysis against known corpora, (b) use dated or freshly generated items and say so, or (c) state explicitly that contamination could not be assessed and bound what it could explain. Silence is the only unacceptable option — and a contamination paragraph placed *by the results*, not buried in an appendix, defuses the rebuttal question before it is asked. **2. Baseline fairness.** "Our method beats the baseline" means nothing if the baseline got one prompt attempt and your method got fifty. Matched budgets are the rule: same prompt-engineering effort, same number of few-shot exemplars, same decoding search, same token b
- The four LM-specific attacks on your results
- Uncertainty that matches the noise source
- One config per number
- Compute disclosure
- Scaling and coverage
- Human and model-judge evaluation
- Design review checklist
- Output format
What does the colm-experiments skill do?
Use when designing or auditing the empirical core of a COLM paper — contamination analysis for evaluation data, fair baselines under matched prompting and compute, pinned model versions and decoding parameters, uncertainty over runs and samples, scaling coverage, and honest reporting of API-model comparisons.
How do I install it?
Run `npx skills add brycewang-stanford/Awesome-Journal-Skills --skill colm-experiments --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From brycewang-stanford/Awesome-Journal-Skills, a repository with 909 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.