agent-mlops
Production deployment and operationalization of AI agents on Databricks. Use when deploying agents to Model Serving, setting up MLflow logging and tracing for agents, implementing Agent Evaluation frameworks, monitoring agent performance in production, managing agent versions and rollbacks, optimizing agent costs and latency, or establishing CI/CD pipelines for agents. Covers MLflow integration patterns, evaluation best practices, Model Serving configuration, and production monitoring strategies.
npx skills add majiayu000/claude-skill-registry --skill agent-mlops --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Depicts a framework for deploying, logging, tracing, evaluating, and monitoring AI agents in production using Databricks MLflow and Model Serving.
How it works
- Establishes an agent MLOps lifecycle: Development → Logging → Evaluation → Deployment → Monitoring → Iteration, with a feedback loop to improve decisions.
- Provides concrete code examples to enable MLflow tracing for debugging agent decisions, including using mlflow.langchain.autolog() and return_intermediate_steps in the agent executor to capture full execution traces in MLflow UI.
- Defines an Agent Evaluation pattern with steps to create an evaluation dataset, implement metrics (e.g., tool_selection_accuracy via comparison of predicted and expected tools), and run evaluation inside an MLflow run to collect results like answer_similarity and tool_selection_accuracy.
- Details a Model Serving deployment workflow, including creating a robust MLflow model wrapper (AgentModel) with a defined signature and example inputs, logging dependencies, and testing a locally loaded model before deployment.
- Addresses production latency through strategies: proper model serving configuration (endpoint and workload settings), request batching in the model’s predict method, and caching strategies (in-memory cache) to reduce repeated work.
- Introduces cost tracking by instrumenting predictions with latency, tool call counts, token usage, and a cost calculation example to log per-query cost and total cost via MLflow spans.
- Outlines MLflow integration patterns for development workflows, model registration, and A/B testing, including how to register, promote, and alias models and how to compare versions via serving configurations.
- Presents Agent Evaluation Best Practices for building evaluation datasets and custom metrics for tool selection accuracy, latency, and cost.
When to use it
- Use when deploying agents to Model Serving and enabling MLflow logging and tracing.
- Use when implementing Agent Evaluation frameworks to measure decision quality and compare versions.
- Use for production monitoring, version management, and CI/CD pipelines for agents.
- Use to address high latency, debugging challenges, and cost tracking in production deployments.
What it can touch
- Tools: claude-code is listed as a declared tool.
- MLflow components: mlflow.langchain.autolog(), mlflow.start_run, mlflow.pyfunc.log_model, mlflow.evaluate, MLflow experiments, Runs, and signatures.
- Databricks Model Serving and WorkspaceClient for endpoint configuration.
- Python code examples that instantiate AgentModel, CostTrackingAgent, and caching/batching wrappers.
Caveats
- License: MIT.
- The content provides sample code and configurations; outcomes are not guaranteed and depend on the environment setup and agent implementations.
- Some sections rely on external services (MLflow, Databricks) and specific model/tool integrations which may require adaptations for your stack.
# Agent MLOps: Production Deployment & Monitoring Deploy, evaluate, and monitor AI agents in production using Databricks MLflow and Model Serving. ## Core Concepts ### Agent MLOps Lifecycle ``` Development → Logging → Evaluation → Deployment → Monitoring → Iteration ↑ ↓ └────────────────────────────────────────────────────────────┘ ``` **Key difference from traditional MLOps:** Agents make dynamic decisions, requiring evaluation of decision-making quality, not just prediction accuracy. ## Problem-Solution Patterns ### Problem 1: Can't Debug Agent Decisions **Symptoms:** - Agent makes unexpected tool choices - No visibility into reasoning process - Can't reproduce issues - Debugging requires re-running entire workflow **Root causes:** - No tracing enabled - Tool calls not logged - Intermediate steps discarded **Solution: Enable MLflow Tracing** ```python import mlflow # Enable automatic tracing for LangChain mlflow.langchain.autolog() # All agent executions now automatically traced agent_executor = AgentExecutor( agent=agent, tools=tools, return_intermediate_steps=True # Critical for tracing ) # Execute with tracing with mlflow.start_run(run_name="agent_execution"): result = agent_e
- Core Concepts
- Agent MLOps Lifecycle
- Problem-Solution Patterns
- Problem 1: Can't Debug Agent Decisions
- Problem 2: No Systematic Evaluation
- Problem 3: Model Serving Deployment Failures
- Problem 4: High Latency in Production
- Problem 5: Can't Track Agent Costs
- MLflow Integration Patterns
- Pattern 1: Development Workflow
- Pattern 2: Model Registration
- Pattern 3: A/B Testing
- Agent Evaluation Best Practices
- Building Evaluation Datasets
What does the agent-mlops skill do?
Production deployment and operationalization of AI agents on Databricks. Use when deploying agents to Model Serving, setting up MLflow logging and tracing for agents, implementing Agent Evaluation frameworks, monitoring agent performance in production, managing agent versions and rollbacks, optimizing agent costs and latency, or establishing CI/CD pipelines for agents. Covers MLflow integration patterns, evaluation best practices, Model Serving configuration, and production monitoring strategies.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill agent-mlops --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.