data-pipeline
Comprehensive guide to data pipeline design, ETL/ELT patterns, data quality, monitoring, orchestration, and cost optimization for production-grade data engineering.
Profile →npx skills add cosmicstack-labs/mercury-agent-skills --skill data-pipeline --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Outlines design principles for data pipelines, emphasizing idempotency, observability, graceful failure, incremental processing, and cost awareness. Provides a maturity model, architecture patterns (batch vs streaming, Lambda vs Kappa, Medallion), ETL vs ELT guidance, data quality practices, monitoring/observability, and orchestration patterns (Airflow, Dagster, Prefect) along with practical code examples and backfill guidance.
How it works
Describes a step-by-step approach to building pipelines: enforce data contracts and schemas, choose ELT with dbt where suitable, implement freshness and anomaly checks, and track lineage. Includes concrete patterns and sample code for schema validation, OpenLineage integration, DAG design, idempotent task patterns, and backfilling strategies. It also defines operational thresholds (SLA guidance) and alerting concepts.
When to use it
Use when designing or auditing production-grade data pipelines that require observability, data quality controls, and reliable orchestration across batch and streaming patterns. Applicable for teams aiming for Level 3 or higher in the pipeline maturity model and preferring ELT with modern orchestration and lineage tracking.
What it can touch
Mentions tools and platforms such as dbt, Spark, Beam, Flink, OpenLineage, Monte Carlo, Sifflet, Datadog, and various orchestration frameworks (Airflow, Dagster, Prefect). Includes code examples referencing these tools for validation, lineage emission, and task orchestration.
Caveats
License and author information indicate MIT license and a production-oriented scope; no explicit limitations or risk statements are provided beyond the content itself. All numbers and references come from the skill content and are presented as-is.
# Data Pipeline Design ## Core Principles Data pipelines are the arteries of modern data platforms. A well-designed pipeline is **reliable**, **observable**, **idempotent**, and **cost-efficient**. The following principles guide every decision: 1. **Idempotency First** — Running a pipeline twice should produce the same result. This enables safe retries and backfills without data duplication. 2. **Observability by Default** — Every stage must emit metrics, logs, and lineage metadata. If you can't see it, you can't fix it. 3. **Fail Gracefully** — Assume failures will happen. Design dead letter queues, retry logic with exponential backoff, and alerting on anomalies. 4. **Incremental Processing** — Process only what's changed. Full refreshes are for schema migrations and backfills only. 5. **Data Contracts** — Define and enforce schemas at every boundary. Catch drift before it reaches downstream consumers. 6. **Separation of Concerns** — Extract, transform, load are distinct phases. Each should be independently testable and debuggable. 7. **Cost Awareness** — Every byte processed costs money. Partition, compress, and prune aggressively. --- ## Pipeline Maturity Model | Level | Name |
- Core Principles
- Pipeline Maturity Model
- Pipeline Architecture Patterns
- Batch vs Streaming
- Lambda vs Kappa Architecture
- Medallion Architecture (Bronze/Silver/Gold)
- ETL vs ELT
- ETL (Extract, Transform, Load)
- ELT (Extract, Load, Transform)
- Transformation Strategies
- Data Quality
- Schema Validation
- Data Contracts
- Freshness Checks
What does the data-pipeline skill do?
Comprehensive guide to data pipeline design, ETL/ELT patterns, data quality, monitoring, orchestration, and cost optimization for production-grade data engineering.
How do I install it?
Run `npx skills add cosmicstack-labs/mercury-agent-skills --skill data-pipeline --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From cosmicstack-labs/mercury-agent-skills, a repository with 364 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.