web-scraping
Scrape websites, extract structured data, and automate browsers. Use when asked to scrape, extract, crawl, parse, or pull data from web pages or any URL.
npx skills add gooseworks-ai/goose-skills --skill web-scraping --agent claude-code
Same command for any agent — swap --agent for codex, cursor, copilot.
Weekly change comes from our own snapshots, not the repository page — it measures attention, not adoption.
# Scrape — General-Purpose Web Scraping & Data Extraction ## Setup Read your credentials from ~/.gooseworks/credentials.json: ```bash export GOOSEWORKS_API_KEY=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json'))['api_key'])") export GOOSEWORKS_API_BASE=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json')).get('api_base','https://api.gooseworks.ai'))") ``` If ~/.gooseworks/credentials.json does not exist, tell the user to run: `npx gooseworks login` All endpoints use Bearer auth: `-H "Authorization: Bearer $GOOSEWORKS_API_KEY"` Scrape websites, extract structured data, and automate browser interactions. Pick the best API for the task — or combine several for comprehensive extraction. ## 1. Scrapegraph — AI-Powered Scraping with Natural Language Best for: Extracting data using plain English prompts, converting pages to markdown, crawling with AI extraction, and search-based scraping. **AI-powered extraction** (describe what you want in natural language): ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api
- Setup
- 1. Scrapegraph — AI-Powered Scraping with Natural Language
- 2. Olostep — Scalable Scraping & Batch Jobs
- 3. Riveter — Structured Extraction with Defined Schemas
- 4. Brand.dev — Brand Assets, Logos & Company Data
- 5. Notte — Browser Automation & Page Interaction
- Tips
- Discover More
export GOOSEWORKS_API_KEY=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json'))['api_key'])")
export GOOSEWORKS_API_BASE=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json')).get('api_base','https://api.gooseworks.ai'))")
curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \
Poll for results:
Step 1: Start crawl
Step 2: Check status
Step 3: Get pages
Step 4: Retrieve content
Check status:
Get items:What does the web-scraping skill do?
Scrape websites, extract structured data, and automate browsers. Use when asked to scrape, extract, crawl, parse, or pull data from web pages or any URL.
How do I install it?
Run `npx skills add gooseworks-ai/goose-skills --skill web-scraping --agent claude-code` — it drops the skill into your project so the agent can pick it up. Swap the --agent value for codex, cursor or copilot if you use one of those.
Where does this skill come from?
From gooseworks-ai/goose-skills, a repository with 1,091 stars. We read it straight from the repository tree rather than a submitted listing, so what you see here is what is actually published.
Is a popular skill a good skill?
Not necessarily. Stars measure attention, not adoption — a repository can trend for a week and be abandoned. That is why we show the weekly change from our own snapshots next to the total, instead of a single flattering number.