webcrawler-deep-crawl
Deep-crawl any website from start URLs, return per-page LLM-ready text/markdown/HTML plus metadata (title, description, author, language, canonical URL, OG) and in-scope outbound links. Use when user mentions deep crawl website, recursive crawl, crawl a whole site, scrape entire website, scrape docs site, scrape documentation, scrape knowledge base, scrape blog, build RAG corpus, build vector database from website, knowledge base for chatbot, GPT knowledge files, llms.txt, sitemap crawl, BFS crawl, scrape with depth or page limit, include exclude URL globs, remove boilerplate, strip navigation
npx skills add browser-act/skills --skill webcrawler-deep-crawl --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Website Deep Crawl > Input: one or more start URLs (+ optional scope, depth, page-count, globs, removal selectors). Output: per-page records `{url, crawl, metadata, text, markdown, html, outboundLinks}` for every page reached within scope. ## Language All process output to user (progress updates, process notifications) follows the user's language. ## Objective From a small set of start URLs, breadth-first crawl every reachable in-scope page, strip boilerplate (navigation, header, footer, cookie banners, etc.), and emit per-page LLM-ready content (text / markdown / HTML) plus structured metadata — suitable for feeding RAG pipelines, vector databases, or chatbot knowledge bases. ## Prerequisites - One or more start URLs are provided by the caller. - Target pages are publicly reachable, OR the running browser is already logged in for any pages behind authentication. - A working directory is available for writing per-page JSON records and the crawl state file. ## Pre-execution Checks ### 1. Tool Readiness If browser-act has been confirmed available in the current session → skip this step. Invoke `browser-act` via Skill tool to load usage. If installation or configuration issues arise
- Language
- Objective
- Prerequisites
- Pre-execution Checks
- 1. Tool Readiness
- 2. Login Verification (when prerequisites include login requirement)
- Capability Components
- API: discover URLs from /llms.txt
- API: discover URLs from /sitemap.xml
- DOM: discover URLs from the current page
- DOM: extract clean content + metadata + outbound links from the current page
- Composite: full deep crawl from start URL(s)
- Pagination
- Success Criteria
What does the webcrawler-deep-crawl skill do?
Deep-crawl any website from start URLs, return per-page LLM-ready text/markdown/HTML plus metadata (title, description, author, language, canonical URL, OG) and in-scope outbound links. Use when user mentions deep crawl website, recursive crawl, crawl a whole site, scrape entire website, scrape docs site, scrape documentation, scrape knowledge base, scrape blog, build RAG corpus, build vector database from website, knowledge base for chatbot, GPT knowledge files, llms.txt, sitemap crawl, BFS crawl, scrape with depth or page limit, include exclude URL globs, remove boilerplate, strip navigation
How do I install it?
Run `npx skills add browser-act/skills --skill webcrawler-deep-crawl --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From browser-act/skills, a repository with 5,160 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.