scraping / library
WaterCrawl
Capability: WaterCrawl
Use it when
- self-host asynchronous website crawling with a queue and API
- crawl many pages into LLM-ready content with job state
What it solves
Not the fit when
- bot-detection-challenge guaranteed
Install
git clone https://github.com/watercrawl/WaterCrawl && cd WaterCrawl && docker compose up -d
Invoke
Create a crawl request through the API or UI, poll the job, and consume the returned page records as Markdown or structured data.