ai-data-engineering

npx skills add majiayu000/claude-skill-registry --skill ai-data-engineering --agent claude-code

Same command for any agent — swap --agent for codex, cursor, copilot.

# AI Data Engineering ## Purpose Build data infrastructure for AI/ML systems including RAG pipelines, feature stores, and embedding generation. Provides architecture patterns, orchestration workflows, and evaluation metrics for production AI applications. ## When to Use **Use this skill when:** - Building RAG (Retrieval-Augmented Generation) pipelines - Implementing semantic search or vector databases - Setting up ML feature stores for real-time serving - Creating embedding generation pipelines - Evaluating RAG quality with RAGAS metrics - Orchestrating data workflows for AI systems - Integrating with frontend skills (ai-chat, search-filter) **Skip this skill if:** - Building traditional CRUD applications (use databases-relational) - Simple key-value storage (use databases-nosql) - No AI/ML components in the application ## RAG Pipeline Architecture RAG pipelines have 5 distinct stages. Understanding this architecture is critical for production implementations. ``` ┌─────────────────────────────────────────────────────────────┐ │ RAG Pipeline (5 Stages) │ ├─────────────────────────────────────────────────────────────┤ │ │ │ 1. INGESTION → Load documents (PDF, DOCX, Markdown) │ │ 2.

Run RAGAS evaluation (TOKEN-FREE script execution)
python scripts/evaluate_rag.py --dataset eval_data.json --output results.json
Run Qdrant setup script (TOKEN-FREE execution)
python scripts/setup_qdrant.py --collection docs --dimension 1024
Chunk documents (TOKEN-FREE execution)
python scripts/chunk_documents.py \
Run evaluation (TOKEN-FREE execution)
python scripts/evaluate_rag.py \
Core RAG
pip install langchain langchain-core langchain-openai langchain-voyageai langchain-qdrant