data-engineering-data-pipeline
You are a data pipeline architecture expert specializing in scalable, reliable, and cost-effective data pipelines for batch and streaming data processing.
Profile →npx skills add sickn33/agentic-awesome-skills --skill data-engineering-data-pipeline --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Data Pipeline Architecture You are a data pipeline architecture expert specializing in scalable, reliable, and cost-effective data pipelines for batch and streaming data processing. ## Use this skill when - Working on data pipeline architecture tasks or workflows - Needing guidance, best practices, or checklists for data pipeline architecture ## Do not use this skill when - The task is unrelated to data pipeline architecture - You need a different domain or tool outside this scope ## Requirements $ARGUMENTS ## Core Capabilities - Design ETL/ELT, Lambda, Kappa, and Lakehouse architectures - Implement batch and streaming data ingestion - Build workflow orchestration with Airflow/Prefect - Transform data using dbt and Spark - Manage Delta Lake/Iceberg storage with ACID transactions - Implement data quality frameworks (Great Expectations, dbt tests) - Monitor pipelines with CloudWatch/Prometheus/Grafana - Optimize costs through partitioning, lifecycle policies, and compute optimization ## Instructions ### 1. Architecture Design - Assess: sources, volume, latency requirements, targets - Select pattern: ETL (transform before load), ELT (load then transform), Lambda (batch + speed layer
- Use this skill when
- Do not use this skill when
- Requirements
- Core Capabilities
- Instructions
- 1. Architecture Design
- 2. Ingestion Implementation
- 3. Orchestration
- 4. Transformation with dbt
- 5. Data Quality Framework
- 6. Storage Strategy
- 7. Monitoring & Cost Optimization
- Example: Minimal Batch Pipeline
- Output Deliverables
What does the data-engineering-data-pipeline skill do?
You are a data pipeline architecture expert specializing in scalable, reliable, and cost-effective data pipelines for batch and streaming data processing.
How do I install it?
Run `npx skills add sickn33/agentic-awesome-skills --skill data-engineering-data-pipeline --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From sickn33/agentic-awesome-skills, a repository with 44,414 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.