data-pipeline-patterns
ETL/ELT patterns, batch vs streaming, idempotency, data quality framework, and pipeline orchestration
npx skills add vibeeval/vibecosystem --skill data-pipeline-patterns --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Data Pipeline Patterns ## ETL vs ELT Decision | Kriter | ETL | ELT | |--------|-----|-----| | Transform location | Pipeline'da | Data warehouse'da | | Data volume | Küçük-orta | Büyük | | Flexibility | Düşük | Yüksek | | Cost | Compute-heavy | Storage-heavy | | Use case | Legacy, compliance | Modern analytics | ## Batch vs Streaming | Kriter | Batch | Streaming | |--------|-------|-----------| | Latency | Dakika-saat | Saniye-milisaniye | | Complexity | Düşük | Yüksek | | Cost | Düşük | Yüksek | | Use case | Reporting, ETL | Real-time alerts, dashboards | | Tool | Airflow, dbt | Kafka Streams, Flink | ## Idempotency Patterns ```python # Pattern 1: Upsert INSERT INTO target (id, name, updated_at) VALUES (%(id)s, %(name)s, %(ts)s) ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name, updated_at = EXCLUDED.updated_at # Pattern 2: Partition overwrite DELETE FROM target WHERE partition_date = '2026-03-14'; INSERT INTO target SELECT * FROM staging WHERE partition_date = '2026-03-14'; # Pattern 3: Checkpoint last_checkpoint = get_checkpoint('pipeline_x') new_data = source.query(f"WHERE updated_at > '{last_checkpoint}'") process(new_data) save_checkpoint('pipeline_x', max(new_data.updated
- ETL vs ELT Decision
- Batch vs Streaming
- Idempotency Patterns
- Data Quality Framework
- Quality Dimensions
- Pipeline Orchestration
- Checklist
- Anti-Patterns
What does the data-pipeline-patterns skill do?
ETL/ELT patterns, batch vs streaming, idempotency, data quality framework, and pipeline orchestration
How do I install it?
Run `npx skills add vibeeval/vibecosystem --skill data-pipeline-patterns --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From vibeeval/vibecosystem, a repository with 521 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.