embedding-strategies

What it does

Guide to selecting and optimizing embedding models for vector search applications.

How it works

The skill provides structured guidance for when to use embedding strategies (e.g., choosing embedding models for RAG, optimizing chunking, domain-specific embeddings, comparing performance, reducing dimensions, multilingual handling). It includes core concepts such as Embedding Model Comparison (a table of models with their dimensions, max tokens, and best use), and an Embedding Pipeline diagram showing the flow from Document to Vector with optional steps like overlap, size, and normalization. It also offers multiple Templates (1-4) that demonstrate concrete code patterns for using Voyage AI embeddings, OpenAI embeddings (with optional dimensionality reduction), local embeddings with Sentence Transformers, and chunking strategies (token-based, sentence-based, semantic sections, and recursive splitting). Additionally, it defines a Domain-Specific Embedding Pipeline (DomainEmbeddingPipeline) and a CodeEmbeddingPipeline, illustrating how to preprocess, chunk, and embed documents or code, and how to handle code-specific chunking via tree-sitter. Finally, there is Embedding Quality Evaluation logic to measure retrieval metrics.

When to use it

Use when selecting embedding models for RAG, optimizing chunking strategies, fine-tuning embeddings for specific domains, comparing embedding model performance, reducing embedding dimensions, or handling multilingual content.

What it can touch

Templates show concrete code interactions:

• Voyage AI Embeddings via VoyageAIEmbeddings, including model variations (voyage-3-large, voyage-code-3, voyage-finance-2, voyage-law-2).
• OpenAI Embeddings via OpenAI client, with optional dimensions parameter for Matryoshka reduction.
• Local Embeddings via SentenceTransformer (e.g., BAAI/bge-large-en-v1.5, multilingual-e5-large).
• Chunking utilities: chunk_by_tokens, chunk_by_sentences, chunk_by_semantic_sections, recursive_character_splitter.
• DomainEmbeddingPipeline and CodeEmbeddingPipeline demonstrating document processing, chunking, and code-specific embeddings.

Caveats

License is MIT. The skill assumes availability of specific libraries (VoyageAIEmbeddings, OpenAI client, sentence_transformers, tree-sitter for code parsing). No guarantees on performance or outcomes are stated beyond the provided templates and guidance. No explicit risk disclosures beyond typical dependencies and environment requirements are listed in the text.