cxcscmu/

Craw4LLM

Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"

660stars

60forks

4issues

MITlicense

2025since

Star historydaily snapshots by VibeCrowd

Collecting history — the radar snapshots this repo daily. The trend line appears after 3 days of data (1 so far).

Alternatives & relatedmatched by topic overlap

firecrawl/

The context API to search, scrape, and interact with the web at scale. 🔥

161k9.1kTypeScript

rasbt/

Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

101k15kJupyter Notebook

D4Vinci/

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

73k7.2kPython

hiyouga/

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

74k9.0kPython

ray-project/

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

43k7.9kPython

HKUDS/

[EMNLP2025] "LightRAG: Simple and Fast Retrieval-Augmented Generation"

39k5.4kPython