long-document-llm-pipeline
Use when processing documents over 50K characters through LLM APIs with section splitting and batch cost reduction.
npx skills add majiayu000/claude-skill-registry --skill long-document-llm-pipeline --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Long Document LLM Processing Pipeline **Extracted:** 2026-02-08 (updated 2026-02-09) **Context:** When processing documents over ~50K characters through LLM APIs for extraction, generation, or analysis tasks. ## Problem Sending large documents (>50K chars) as a single LLM prompt causes: 1. **Lost in the Middle** - LLMs lose attention on content in the middle of long inputs (30%+ accuracy drop, per Liu et al. 2023) 2. **High cost** - Entire document becomes input tokens even if only portions are relevant 3. **No partial retry** - If generation fails, must re-process the entire document 4. **No parallelism** - Single sequential API call ## Solution: 6-Step Pipeline ``` Document | v [1] Text Extraction (pymupdf4llm, page_chunks=True) | v [2] Structure Detection (Markdown headers, TOC, Japanese patterns) | v [3] Section Splitting (5K-30K chars per section) | v [4] Breadcrumb Context (prepend section path to each chunk) | v [5] Batch API / Async Parallel (50% cost reduction with Batch) | v [6] Merge + Deduplicate Results ``` --- ## Step 1: Structured Extraction (pymupdf4llm) Use `page_chunks=True` to get structured per-page data with metadata: ```python import pymupdf4llm # BAD: Flat
- Problem
- Solution: 6-Step Pipeline
- Step 1: Structured Extraction (pymupdf4llm)
- Key Parameters
- Heading Detection
- Steps 2-3: Heading-Stack Sectioning with Breadcrumb
- Behavior
- Fallback Chain
- Oversized Section Sub-splitting
- Data Model
- Step 4: Breadcrumb Context in Prompts
- Step 5: API Call Strategy
- Decision Matrix: When to Chunk
- Anthropic Batch API (50% Cost Reduction)
What does the long-document-llm-pipeline skill do?
Use when processing documents over 50K characters through LLM APIs with section splitting and batch cost reduction.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill long-document-llm-pipeline --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.