web-scraper
Scrape, crawl, and extract data from websites. Use when users ask to scrape web pages, extract content, crawl websites, or collect data from the internet.
Profile →npx skills add shobcoder/shob --skill web-scraper --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Web Scraper ## Overview Extract content and data from websites using various techniques including crawling, scraping, and structured data extraction. ## When to Use - Extract text content from web pages - Crawl entire websites - Collect structured data - Research and gather information - Monitor website changes - Extract tables and lists ## Tools Available ### Content Extraction ```javascript // Use extract_content_from_websites for structured extraction // Supports batch processing of multiple URLs // Returns JSON format with extracted content ``` ### Task Format ```javascript { tasks: [ { url: "https://example.com", prompt: "Extract specific information", task_name: "optional_name" } ] } ``` ## Usage Patterns ### Simple Content Extraction ```javascript // Extract main content from a page const result = await extract_content_from_websites({ tasks: [{ url: "https://news.example.com/article", prompt: "Extract the title, author, date, and main content" }] }); ``` ### Batch URL Processing ```javascript // Process multiple URLs in parallel const urls = [ "https://site.com/page1", "https://site.com/page2", "https://site.com/page3" ]; const results = await extract_content_from_websites
- Overview
- When to Use
- Tools Available
- Content Extraction
- Task Format
- Usage Patterns
- Simple Content Extraction
- Batch URL Processing
- Data Mining
- Extraction Modes
- Auto Mode (Default)
- Curl Only Mode
- Browser Only Mode
- Best Practices
What does the web-scraper skill do?
Scrape, crawl, and extract data from websites. Use when users ask to scrape web pages, extract content, crawl websites, or collect data from the internet.
How do I install it?
Run `npx skills add shobcoder/shob --skill web-scraper --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From shobcoder/shob, a repository with 583 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.