model-evaluation
This skill focuses on comprehensive evaluation of machine learning models across the entire ML lifecycle. It covers metric selection, validation strategies, fairness assessment, training debugging, hyperparameter tuning, LLM evaluation, A/B testing, and production monitoring for ensuring model quality and reliability.
npx skills add majiayu000/claude-skill-registry --skill model-evaluation --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Evaluates machine learning models for performance, fairness, and reliability using appropriate metrics and validation techniques. Covers training debugging, hyperparameter tuning, and production monitoring.
How it works
The skill instructs the agent to:
- Define evaluation criteria by translating business goals into ML metrics, selecting suitable metrics for classification, regression, and LLM tasks, and establishing fairness constraints.
- Design an evaluation strategy including data splits, cross-validation, handling class imbalance, and appropriate validation schemes.
- Conduct evaluation by computing primary and secondary performance metrics, performing error analysis, and assessing fairness with group-wise metrics and parity/odds definitions.
- Debug training issues such as overfitting, underfitting, convergence problems, learning rate issues, and batch size effects, with suggested mitigations.
- Perform hyperparameter tuning using search strategies (grid, random, Bayesian optimization, Hyperband) and validate on a dedicated validation set.
- Evaluate LLMs and generative models using automatic metrics (perplexity, BLEU, ROUGE, BERTScore) and human evaluation, plus prompt engineering evaluation.
- Conduct A/B testing design and analysis, including random traffic splits, significance testing, and decision criteria based on primary metrics and guardrails.
- Implement production model monitoring plans for performance, data drift, concept drift, and automated retraining triggers, with dashboards and alerting.
- Compile an evaluation report covering methodology, metrics, error analysis, fairness, and recommendations.
When to use it
Use this skill when you need to:
- Select and compute appropriate evaluation metrics for ML models
- Design cross-validation strategies and train/test splits
- Debug training issues (overfitting, underfitting, convergence problems)
- Tune hyperparameters and validate model improvements
- Evaluate LLMs and generative models
- Conduct A/B tests for model comparison in production
- Monitor deployed models for drift and degradation
- Assess model fairness across demographic groups
- Analyze error patterns and residuals
- Create evaluation reports and dashboards
What it can touch
- Tools: claude-code
- References to evaluation concepts include metric calculations, cross-validation schemes, A/B testing analyses, model monitoring approaches, and error analysis techniques as described in the skill instructions.
Caveats
- License: MIT
- Declared scope is comprehensive evaluation across ML lifecycle; no guarantees of specific outcomes beyond described metrics and workflows.
- No external promises about production readiness outside the monitoring and evaluation activities outlined.
--- name: model-evaluation description: Evaluates machine learning models for performance, fairness, and reliability using appropriate metrics and validation techniques. Covers training debugging, hyperparameter tuning, and production monitoring. Trigger keywords: model evaluation, metrics, accuracy, precision, recall, F1, F1-score, ROC, AUC, ROC-AUC, confusion matrix, cross-validation, k-fold, stratified, overfitting, underfitting, bias, variance, bias-variance tradeoff, hyperparameter, hyperparameter tuning, loss, loss function, metric, benchmark, benchmarking, model performance, classification metrics, regression metrics, RMSE, MSE, MAE, MAPE, R2, R-squared, train-test split, validation set, test set, hold-out, learning curve, validation curve, model selection, error analysis, residual analysis, ML testing, training issues, convergence, gradient, vanishing gradient, exploding gradient, training instability, LLM evaluation, language model evaluation, prompt engineering evaluation, A/B testing, champion-challenger, model monitoring, model drift, data drift, concept drift, model decay. allowed-tools: Read, Grep, Glob, Edit, Write, Bash --- # Model Evaluation ## Overview This skill
- Overview
- When to Use This Skill
- Related Skills and Agents
- Instructions
- 1. Define Evaluation Criteria
- 2. Design Evaluation Strategy
- 3. Conduct Evaluation
- 4. Debug Training Issues
- 5. Hyperparameter Tuning
- 6. LLM and Generative Model Evaluation
- 7. A/B Testing for Model Comparison
- 8. Production Model Monitoring
- 9. Report and Document
- Best Practices
What does the model-evaluation skill do?
This skill focuses on comprehensive evaluation of machine learning models across the entire ML lifecycle. It covers metric selection, validation strategies, fairness assessment, training debugging, hyperparameter tuning, LLM evaluation, A/B testing, and production monitoring for ensuring model quality and reliability.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill model-evaluation --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.