ds-roast

What it does

The skill directs the agent to act as a ruthless senior ML engineer performing a deep, exhaustive codebase review of data science, MLOps, and ML projects. It prescribes identifying anti-patterns, data leakage, data quality problems, modeling pitfalls, evaluation and baselines shortcomings, time-series pitfalls, and reproducibility gaps, then providing concrete, actionable fixes for each issue.

How it works

Step 1: Codebase Reconnaissance — Thoroughly explore the repository structure and relevant files, mapping layout, presence of notebooks, pyproject/requirements, configs, data files, tests, and any indication of testing or CI pipelines. Step 1 also analyzes the data pipeline by locating data loading, preprocessing, and feature engineering, tracing data flow from raw sources to model input, distinguishing train-time from inference-time transformations, and inspecting any aggregated features for potential leakage. It reviews lag depths and timing relative to the prediction point. Step 1 continues with Modeling — locating model definitions, training loops, hyperparameters, and evaluation code to confirm problem type and loss/metrics alignment. It assesses MLOps posture including experiment tracking, reproducibility, model versioning, artifacts, serving infrastructure, and CI/CD. Step 1 ends with Testing — checking for unit, data validation, integration, and property-based tests.

Step 2: The Roast — Structured Technical Review — delivers findings in mandatory sections with:

• Location: file and line number where issues are found.
• The Problem: exact technical description.
• Why It Matters: concrete downstream impact.
• Fix: specific, actionable code-level recommendations.

The roast categorizes issues into:

• 🔴 Critical Issues — Data Leakage and Data Quality, plus Other Critical Issues like wrong loss, NaN propagation, and serialization bugs.
• 🟠 Modeling & Methodology Issues — Evaluation, Baselines, Feature Engineering, Modeling Choices, Time-Series specifics.
• 🟡 MLOps & Reproducibility Gaps — Reproducibility practices and environment pinning.

It emphasizes preventing leakage (preprocessing, target, temporal, group, etc.), ensuring proper train/val/test separations, correct lagging, and strict sequencing of data processing relative to prediction windows. It also enforces explicit schema handling, proper handling of unknown categories, and robust evaluation protocols.

When to use it

Use this skill when you want an in-depth, production-focused technical critique of a DS/ML project, particularly to surface leakage, data quality issues, improper feature engineering, modeling flaws, evaluation pitfalls, and MLOps gaps. It expects explicit file/line locations and concrete fixes, and it avoids high-level praise without actionable content.

What it can touch

• Tools: claude-code
• It asks to reference file paths, code constructs, and feature engineering patterns exactly as they appear in the repository when pinpointing issues and suggesting fixes.
• It requires providing concrete, code-level suggestions (e.g., replace a calculation with a lagged version, wrap a feature inside a Pipeline, or move a split to TimeSeriesSplit).

Caveats

• Licensing: MIT (as declared).
• The analysis should be strictly based on the content and instructions within the skill’s own described process; no external assumptions beyond what is explicitly described in the guidance.
• The review will not promise outcomes beyond what is stated in the ROAST guidelines (e.g., it will not claim fixes will solve all issues without implementing them).