agentsop-map-reduce-fanout
Decision protocol for the map-reduce / dynamic fan-out pattern in LM pipelines — "given list L, run f(item) for each item in parallel, then combine". Activates when the coder agent is about to process N items with N LM calls (per-doc summarize, per-query retrieve, per-candidate rank, parallel tool fan-out). Encodes the *when*, *how many at once*, *what to do when one fails*, and *how to reduce* — not the API of any single framework. Cross-framework: LangGraph `Send`, CrewAI parallel tasks / Flow, `asyncio.gather`, `ThreadPoolExecutor`, LlamaIndex batch retrieval.
npx skills add agentsope/SkillAlchemy --skill agentsop-map-reduce-fanout --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
What it does
Describes a protocol for running per-item LM calls in parallel and then merging results with a reducer. It activates when the coder is about to process N items with N LM calls and covers when to fan out, how many concurrent executions to allow, how to handle failures, and how to reduce to a final result. It references cross-framework patterns such as LangGraph Send, CrewAI parallel tasks / Flow, asyncio.gather, ThreadPoolExecutor, and LlamaIndex batch retrieval. It does not specify the API of any single framework.
How it works
- Activation is triggered by scenarios where per-item LM calls are made in parallel and items are independent, or when existing code uses asyncio.gather, ThreadPoolExecutor, Send, or CrewAI kickoff_for_each.
- The Core Mental Model explains fan-out for latency and fan-in for coherence, with the unit of fan-out being the item and the reducer shaping the final output.
- Standard workflow steps include confirming item independence, estimating cost and budgets, choosing N_concurrent, deciding failure policy, selecting a reducer, and setting per-call and total timeouts. It also covers verifying the sanity of the reduction after it runs.
- Operation models provide concrete patterns: OP-1 via asyncio.gather with bounded concurrency; OP-2 for first-K results via asyncio.as_completed; OP-3 through LangGraph Send for dynamic fan-out; OP-4 using a reducer to merge parallel writes; OP-5 through OP-7 for parallel invocations with CrewAI, ThreadPoolExecutor, and LlamaIndex batch retrieval; OP-8 describes tree-reduce for large N; OP-9 adds a per-item retry wrapper; OP-10 adds item-level and reduce-step observability.
When to use it
- Activate when any activation rule is met, such as a task description describing per-item LM calls, an independent for item in items: lm(item) loop, or evidence of existing parallel patterns and potential fan-out concerns.
- Use when you need to decide how many concurrent calls to run and how to reduce results, especially if a LangGraph Send-based fan-out or CrewAI parallelization is involved.
What it can touch
- Tools and concepts named: LangGraph Send, CrewAI, asyncio.gather, ThreadPoolExecutor, LlamaIndex batch retrieval, and per-call timeouts. Specific code patterns demonstrate how to implement bounded concurrency, early completion (first-K), and parallel writes with reducers.
Caveats
- Mentions policies for failure handling (Abort-all, Best-effort, Retry-then-skip, Quorum) and various reducers (concatenate, summarize, vote, rank-top-K, merge-dedupe, tree-reduce), with safety and budgeting considerations. It emphasizes choosing semantics before fan-out and setting timeouts.
- Describes that a reducer is mandatory when multiple parallel writes occur to the same state key in LangGraph, and that parallel patterns require explicit reduction to produce a coherent output.
# Map-Reduce / Dynamic Fan-Out · SOP > Pattern: `results = reduce(combine, parallel_map(f, L))` where `f` is one > or more LM calls. The *only* reason to fan out is that latency or > throughput matters more than the cost of doing it. The *only* reason to > fan in is that the consumer wants one answer, not N. > Source posture: claims grounded in primary docs and 2026 production > write-ups, cited inline with short tags resolved in the citation index. --- ## 1. 何时激活 (Activation Rules) Activate this skill when **any** of these is true: - The task description contains "for each X, do Y" where Y involves an LM call, a retriever hit, or any I/O-bound step costing >100ms. - The coder is about to write a `for item in items: result = llm(item)` loop and the items are independent (no item depends on the previous result). - The codebase already has `asyncio.gather(...)`, `ThreadPoolExecutor(...)`, `Send(...)`, `Process.hierarchical` parallel branches, or `crew.kickoff_for_each(...)` and the question is *how* to use them safely. - The user mentions any of: "summarize N docs", "rank top-K candidates", "vote across M models", "ensemble", "parallel agents", "multi-query retrieval", "scatter-gathe
- 1. 何时激活 (Activation Rules)
- 2. 核心心智模型 (Core Mental Model)
- 3. SOP 工作流 (Standard Operating Protocol)
- Step 1 · Confirm items are independent
- Step 2 · Estimate cost honestly
- Step 3 · Pick Nconcurrent (the most important number in the file)
- Step 4 · Decide the failure policy before writing the fan-out
- Step 5 · Pick the reducer
- Step 6 · Set per-call timeout and total wall-clock budget
- Step 7 · Reduce, then verify the reduction is sane
- 4. 操作模型 (Operation Models)
- OP-1 · asyncio.gather with bounded concurrency (Python baseline)
- OP-2 · asyncio.ascompleted for quorum / first-K
- OP-3 · LangGraph Send for dynamic fan-out
What does the agentsop-map-reduce-fanout skill do?
Decision protocol for the map-reduce / dynamic fan-out pattern in LM pipelines — "given list L, run f(item) for each item in parallel, then combine". Activates when the coder agent is about to process N items with N LM calls (per-doc summarize, per-query retrieve, per-candidate rank, parallel tool fan-out). Encodes the *when*, *how many at once*, *what to do when one fails*, and *how to reduce* — not the API of any single framework. Cross-framework: LangGraph `Send`, CrewAI parallel tasks / Flow, `asyncio.gather`, `ThreadPoolExecutor`, LlamaIndex batch retrieval.
How do I install it?
Run `npx skills add agentsope/SkillAlchemy --skill agentsop-map-reduce-fanout --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From agentsope/SkillAlchemy, a repository with 255 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.