ml-pipeline-orchestration
ML pipeline design with Metaflow, Kubeflow, and ZenML including GPU steps, artifact tracking, and production patterns.
npx skills add majiayu000/claude-skill-registry --skill ml-pipeline-orchestration --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# ML Pipeline Orchestration ## Orchestrator Selection | Criteria | Metaflow | Kubeflow | ZenML | |----------|----------|----------|-------| | Setup complexity | Low (pip install) | High (K8s cluster) | Medium (pip + stack) | | GPU support | Via `@batch`/`@kubernetes` | Native K8s GPU scheduling | Via orchestrator backend | | Experiment tracking | Built-in (cards, artifacts) | External (MLflow/W&B) | Built-in + integrations | | Production readiness | High (Netflix-proven) | High (K8s-native) | Medium (maturing) | | Local dev experience | Excellent (`--local`) | Poor (needs cluster) | Good (local stack) | | Team size sweet spot | 2-20 | 20-100+ | 2-15 | | Best for | Data science teams | Platform engineering | MLOps standardization | **Recommendation**: Start with Metaflow for most teams. Move to Kubeflow only if you already run Kubernetes and need multi-tenant scheduling. ZenML fits teams wanting vendor-neutral MLOps with plugin architecture. ## Pipeline Design Pattern Standard ML pipeline stages: ``` data_ingest -> preprocess -> validate -> train -> evaluate -> register -> deploy ``` Keep each step single-responsibility. Pass artifacts (not raw data) between steps. ## Metaflow ### F
- Orchestrator Selection
- Pipeline Design Pattern
- Metaflow
- Flow Definition
- Metaflow Parallel Training (Fan-Out)
- Kubeflow Pipeline with GPU Steps
- Gotchas and Anti-Patterns
- Pipeline Serialization Issues
- GPU Resource Allocation
- Artifact Storage Costs
- Pipeline Versioning
kfp dsl compile --py pipeline.py --output pipeline.yaml kfp run submit --experiment training --pipeline pipeline.yaml
What does the ml-pipeline-orchestration skill do?
ML pipeline design with Metaflow, Kubeflow, and ZenML including GPU steps, artifact tracking, and production patterns.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill ml-pipeline-orchestration --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.