AnyCrawl is a Node.js/TypeScript crawler that converts web content into structured data for LLMs, with multi-threading support and SERP extraction across engines. It offers web scraping, site crawling, and batch tasks, plus LLM extraction integrations and an API for self-hosting.
Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).
What it is
AnyCrawl is a high-performance crawling and scraping toolkit that supports SERP crawling across multiple engines, web scraping of single pages, site-wide crawling, and AI-driven extraction producing JSON data. It emphasizes multi-threading/multi-process workflows and batch task handling, with LLM-ready outputs for integration with AI tools.
How it works
The project provides modules for:
- Web scraping via various engines (cheerio, playwright, puppeteer)
- Site crawling with depth/limit controls and path-based filters
- SERP data retrieval from multiple engines
- LLM extraction using JSON schemas and configurable models/providers (e.g., Atlas Cloud)
- Batch task processing and cache support
It exposes REST-like endpoints (scrape, crawl, search) and examples for API usage, including request parameter schemas for engine, proxy, max_age, and caching behavior. It also offers a File API integration for storage backends and a CloakBrowser-enabled browser runtime for rendered content.
Getting started
Key setup aspects from the README:
- API authentication (optional): enable ANYCRAWL_API_AUTH_ENABLED and generate an API key
- Generate API key commands:
pnpm --filter api key:generate
# optionally name the key
pnpm --filter api key:generate -- default
- Docker usage for self-hosted setups:
docker compose exec api pnpm --filter api key:generate
docker compose exec api pnpm --filter api key:generate -- default
docker exec -it <container_name_or_id> pnpm --filter api key:generate
docker exec -it <container_name_or_id> pnpm --filter api key:generate -- default
- Example usage for scraping:
curl -X POST https://api.anycrawl.dev/v1/scrape \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' \
-d '{
"url": "https://example.com",
"engine": "cheerio"
}'
- Example usage for crawling:
curl -X POST https://api.anycrawl.dev/v1/crawl \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' \
-d '{
"url": "https://example.com",
"engine": "playwright",
"max_depth": 2,
"limit": 10,
"strategy": "same-domain"
}'
- Example usage for SERP:
curl -X POST https://api.anycrawl.dev/v1/search \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_ANYCRAWL_API_KEY' \
-d '{
"query": "AnyCrawl",
"limit": 10,
"engine": "google",
"lang": "all"
}'
Recent releases
Latest 5 releases:
- v1.0.0-beta.33 (2026-07-13): merge latest updates; Implement the File API for checking out files from S3 or local storage
- v1.0.0-beta.32 (2026-06-30): merge latest updates; Implement the File API for checking out files from S3 or local storage
- v0.0.1-beta.20 (2026-03-06): Full Changelog at GitHub compare link
- v1.0.0-beta.13 (2026-02-08): Full Changelog at GitHub compare link
- v1.0.0-beta.10 (2026-01-28): Full Changelog at GitHub compare link
Traction
Stars: 3382. Forks: 359. Open issues: 6.
Behind the repo
Not provided in the given data.
Caveats
License: MIT. Created 2025-03-31; last push 2026-07-31. Language: TypeScript. Self-host options include Docker with API key generation and environment variable controls for authentication and browser runtime settings. The README references additional docs and environment configurations (e.g., CLOAKBROWSER settings) but exact file-level caveats are not enumerated here.