ml-monitoring
Production-grade ML model monitoring, drift detection, and observability
npx skills add majiayu000/claude-skill-registry --skill ml-monitoring --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Provides an ML monitoring workflow focusing on drift detection, model performance monitoring, and alerting. It supports input options to select monitoring_type (drift_detection, performance_monitoring, alerting, ab_testing, root_cause_analysis) and model_type, with drift_config controlling reference window, detection method, and threshold. The skill outputs a status, drift_results, alerts, recommendations, and next_steps, aligning with pre/post conditions that require a deployed model, accessible metrics endpoint, and baseline data. It includes concrete drift detection implementation using Evidently AI, a Prometheus-based metrics exporter for model predictions and performance, and an alerting design with severity rules and a sample implementation scaffold.
How it works
- Drift detection module: uses Evidently AI to compare reference and current data, producing a drift_result object that includes timestamp, dataset_drift, drift_share, drifted_columns, severity, and recommendations. It analyzes drift_share to set severity (critical if >0.5, degraded if >0.2) and appends corresponding recommendations. It stores results in drift_history.
- HTML report: can generate an interactive drift HTML report given current data and a path, using DataDriftPreset metrics.
- Model performance monitoring: provides a Prometheus metrics exporter MLMetricsExporter that initializes various metrics (ml_predictions_total, ml_prediction_latency_seconds, ml_prediction_confidence, ml_feature_value, ml_model_accuracy, ml_model_f1_score, ml_drift_score) and a decorator track_prediction to wrap prediction calls, recording latency, status, and confidence when available. It also exposes update_performance_metrics and update_drift_score for live updates and a /metrics endpoint to serve Prometheus data. A /predict endpoint is shown to demonstrate usage.
- Alerting design: outlines a severity matrix with critical, warning, and info alerts based on model accuracy, latency, error rate, and drift score, with defined response times and channels. It includes an Intelligent Alerting scaffold with data classes for Alert and AlertRule, and an AlertManager skeleton for noise-reduced alerting.
When to use it
Use when you need production-grade monitoring for ML models, including drift detection, observable performance metrics, and an alerting framework. Trigger conditions include a deployed and serving model, accessible metrics endpoint, and available baseline data.
What it can touch
- Tools: claude-code
- Components touched: drift detection via Evidently AI, Prometheus metrics via MLMetricsExporter, alerting components with Alert and AlertManager scaffolds. The skill specifies using Evidently-based drift detection, Prometheus exporters, and alerting logic with thresholds.
Caveats
- License: MIT
- The drift detection logic relies on Evidently metrics structure; exact drift interpretation may vary with data and configuration. The HTML report generation depends on the proper current_data input and the data formatting compatible with Evidently preset metrics. The alerting section provides a scaffold; concrete integration with real alerting services (pagerduty, slack, email) is described in the matrix but not implemented here.
# ML Monitoring Production-grade ML model monitoring, drift detection, and observability skill. ## Learning Objectives By mastering this skill, you will be able to: - Implement comprehensive data and model drift detection - Build production alerting systems with actionable notifications - Design and analyze A/B tests for model comparison - Create observability dashboards for ML systems - Perform root cause analysis on model degradation --- ## Module 1: Data Drift Detection ### Concept Overview Data drift occurs when the statistical properties of model inputs change over time, potentially degrading model performance. ### Drift Detection Methods | Method | Best For | Sensitivity | Compute Cost | |--------|----------|-------------|--------------| | KS Test | Continuous features | High | Low | | PSI | Categorical features | Medium | Low | | Wasserstein | Distribution shape | High | Medium | | Chi-Square | Categorical | Medium | Low | | JS Divergence | Probability distributions | High | Medium | ### Implementation: Evidently AI Drift Detection ```python """ Production-ready drift detection with Evidently AI. """ import pandas as pd from evidently import ColumnMapping from evidently.repo
- Learning Objectives
- Module 1: Data Drift Detection
- Concept Overview
- Drift Detection Methods
- Implementation: Evidently AI Drift Detection
- Exercise 1.1: Custom Drift Detection
- Module 2: Model Performance Monitoring
- Key Performance Indicators
- Implementation: Prometheus Metrics Exporter
- Module 3: Alerting System Design
- Alert Severity Matrix
- Implementation: Intelligent Alerting
- Module 4: A/B Testing for ML Models
- Statistical Framework
Check drift threshold settings grep -r "drift_threshold" config/ Review historical drift scores SELECT date, drift_score, model_accuracy FROM ml_metrics WHERE date > NOW() - INTERVAL '7 days' ORDER BY date; Check Evidently drift report python -m evidently.ui.app --project-path ./evidently_workspace Prometheus metrics query
What does the ml-monitoring skill do?
Production-grade ML model monitoring, drift detection, and observability
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill ml-monitoring --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.