agentsop-hybrid-retrieval

What it does

Sets up a hybrid retrieval workflow that combines dense embedding-based recall with sparse BM25 retrieval to capture exact-token matches (identifiers, codes, names, etc.) when lexical traffic justifies it. It defines a decision rule that hybrid should be activated only when the query-share of lexical signals is non-trivial, and specifies how to fuse the two retrievers and tune alpha per query type.

How it works

• Activation is triggered by conditions including presence of exact-match tokens, bug reports about exact queries failing with dense-only, and evidence from PR reviews or traffic composition.
• Build two retrievers over the same node set: a dense retriever (index.as_retriever(...)) and a sparse BM25 retriever (BM25Retriever.from_defaults(nodes=...), with similarity_top_k widened to 10–20).
• Fuse them using a deliberate fusion method, defaulting to Reciprocal Rank Fusion (QueryFusionRetriever(mode="reciprocal_rerank")) or using relative_score with retriever_weights when a labeled set exists.
• Tune alpha per query type (semantic vs lexical) by evaluating a set of alpha values {0.0, 0.25, 0.5, 0.75, 1.0} and selecting per-type settings that lift lexical recall without regressing semantic recall; if needed, route by query type to apply per-type alpha.
• Consider native hybrid support in the vector store (Qdrant/Weaviate/Pinecone) to avoid a separate BM25 index when available.
• Gate changes with an eval loop to ensure lexical gains do not regress semantic performance.

When to use it

Use when the corpus contains exact-match tokens and users report missing results for exact queries, or when tuning recall on a corpus where both meaning and exact strings matter, and you are deciding between pure dense, hybrid, or sparse-first configurations.

What it can touch

• The two retrievers: dense index (via index.as_retriever) and sparse BM25 retriever (BM25Retriever.from_defaults).
• Fusion logic via QueryFusionRetriever with mode values like "reciprocal_rerank" or "relative_score" and retriever_weights.
• Alpha tuning workflows per query type and routing decisions for per-type blends.

Caveats

• Activation is traffic-driven; — not theoretical — and the lexical share must meet thresholds (5–50% for hybrid; >50% inverts to BM25-first). If the lexical share is below 5%, dense-only is recommended.
• Global alpha tuning can hurt semantic performance; per-type tuning is advised.
• If the vector store provides native hybrid, it may be preferable to use that instead of a separate BM25 index.
• The approach assumes a common node set for both dense and sparse retrievers; misalignment can reduce recall.