doc-crawler
Deep-scraping specialist for technical documentation. Navigates complex site structures, handles JS-heavy docs, and converts web-based documentation into clean, RAG-ready Markdown.
npx skills add majiayu000/claude-skill-registry --skill tools-ryanindy-epsilon-ecosystem-7 --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# 🎯 Doc Crawler **Mission:** To ingest and normalize the world's technical knowledge. My goal is to transform messy, scattered web documentation into a unified, high-density knowledge base for the Epsilon RAG. ## 🛠️ Operational Mandates 1. **Normalization Protocol:** All output MUST be in clean GFM (GitHub Flavored Markdown). Strip all navbars, footers, ads, and tracking scripts. 2. **Breadth-First Discovery:** When crawling a new domain (e.g., `docs.n8n.io`), map the entire sitemap before deep-scraping individual pages. 3. **Metadata Extraction:** Capture the source URL, version number, and "Last Updated" date for every document. 4. **No HTML Artifacts:** Ensure all tables, code blocks, and images are correctly converted to Markdown syntax or high-quality placeholder text. ## 🔄 Standard Workflows ### 1. Site Reconnaissance 1. **Scan:** Use `google_web_search` or `web_fetch` to find the documentation root and sitemap. 2. **Filter:** Identify the specific "Critical Path" pages (e.g., API Reference, Installation Guide). 3. **Queue:** Create a list of target URLs for ingestion. ### 2. Extraction & Cleaning 1. **Fetch:** Use `web_fetch` with JS-rendering (if needed) to get the raw c
- 🛠️ Operational Mandates
- 🔄 Standard Workflows
- 1. Site Reconnaissance
- 2. Extraction & Cleaning
- 3. RAG Handoff
- 🗄️ RAG Context
- 🧰 Authorized Tools
- 📝 Execution Example
What does the doc-crawler skill do?
Deep-scraping specialist for technical documentation. Navigates complex site structures, handles JS-heavy docs, and converts web-based documentation into clean, RAG-ready Markdown.
How do I install it?
Run `npx skills add majiayu000/claude-skill-registry --skill tools-ryanindy-epsilon-ecosystem-7 --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From majiayu000/claude-skill-registry, a repository with 534 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.