harvest-deep-crawl
Multi-page deep crawling - documentation sites, wikis, knowledge bases
npx skills add vibeeval/vibecosystem --skill harvest-deep-crawl --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Harvest Deep Crawl Crawl multi-page websites following internal links to a specified depth. Ideal for building complete knowledge bases from documentation sites, wikis, and reference materials. ## Usage ``` /crawl <url> --depth <N> ``` ## Examples ```bash # Crawl docs site 3 levels deep /crawl https://docs.example.com --depth 3 # Crawl a specific section /crawl https://docs.example.com/api --depth 2 # Crawl with page limit /crawl https://wiki.example.com --depth 5 --max-pages 50 ``` ## Parameters | Param | Default | Description | |-------|---------|-------------| | `--depth` | 2 | Max link-following depth | | `--max-pages` | 100 | Max pages to crawl | | `--same-domain` | true | Stay on same domain | | `--include` | * | URL pattern to include | | `--exclude` | - | URL pattern to exclude | ## How It Works 1. Start at root URL, extract all internal links 2. Follow links up to specified depth (BFS order) 3. Extract content from each page 4. Deduplicate pages with > 90% content overlap 5. Build table of contents from page hierarchy 6. Merge into coherent knowledge base 7. Save to `.claude/cache/agents/harvest/crawl-{domain}/` ## Output Structure ``` crawl-{domain}-{timestamp}/ index.m
- Usage
- Examples
- Parameters
- How It Works
- Output Structure
- Crawl Engine
- Primary: crawl4ai (Docker port 11235)
- Fallback: Manual Link Following
- Use Cases
- Rules
Crawl docs site 3 levels deep Crawl a specific section Crawl with page limit curl -s http://localhost:11235/crawl \
What does the harvest-deep-crawl skill do?
Multi-page deep crawling - documentation sites, wikis, knowledge bases
How do I install it?
Run `npx skills add vibeeval/vibecosystem --skill harvest-deep-crawl --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From vibeeval/vibecosystem, a repository with 521 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.