ml-experiment-lifecycle
Use when designing ML experiments, choosing evaluation metrics, tracking experiments, tuning hyperparameters, debugging training, ensuring reproducibility, or building ML pipelines. Covers W&B/MLflow integration, seed management, deterministic training, HPO strategy, and MLOps pipeline design.
npx skills add majiayu000/claude-skill-registry --skill ml-experiment-lifecycle --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# ML Experiment Lifecycle ## Metric Selection | Task | Primary | When to Prefer Alternative | |------|---------|---------------------------| | Classification | F1 (macro) | AUC-ROC for imbalanced; MCC for binary with skew | | Detection | mAP@0.5:0.95 | mAP@0.5 for coarse localization | | NLP Generation | BERTScore | ROUGE for extractive; perplexity for language modeling | | Regression | MAE | MSE when outlier penalty matters; MAPE for interpretability | | Ranking/RAG | NDCG | MRR when only first hit matters; Precision@K for top-heavy use | ## Statistical Comparison - **Paired bootstrap** over t-test: no normality assumption, more robust for ML - **McNemar's test** for classifier comparison on same test set - **Always report**: effect size (Cohen's d) alongside p-values - **Multiple comparisons**: Benjamini-Hochberg for exploratory ablations, Bonferroni for confirmatory claims - **Minimum seeds**: 5-10 for reliable multi-seed comparison; 3 is insufficient - **Paired tests** when models share test data; independent otherwise ## LLM Evaluation - LLM-as-judge: always validate with human agreement checks before trusting - Pairwise comparison (A vs B) more reliable than absolute scoring
- Metric Selection
- Statistical Comparison
- LLM Evaluation
- Experiment Tracking
- Platform Selection
- Metric Naming Convention
- W&B Run Configuration
- Artifact Versioning
- Sweep Configuration
- Reproducibility
- Seed Management
- Per-Component Seeds
- Deterministic PyTorch
- Known Non-Deterministic PyTorch Ops
What does the ml-experiment-lifecycle skill do?
Use when designing ML experiments, choosing evaluation metrics, tracking experiments, tuning hyperparameters, debugging training, ensuring reproducibility, or building ML pipelines. Covers W&B/MLflow integration, seed management, deterministic training, HPO strategy, and MLOps pipeline design.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill ml-experiment-lifecycle --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.