MerchantryTidbits

scraping / library

Crawl4AI

Capability: Crawl4AI

Use it when

  • LLM-friendly web crawling and extraction
  • crawl sites into clean text for RAG ingestion

What it solves

js-render-not-blocking

Not the fit when

  • bot-detection-challenge guaranteed

Install

pip install crawl4ai && crawl4ai-setup

Invoke

Create `AsyncWebCrawler`, call `arun(url=...)`, and consume `result.markdown`; configure extraction or crawl strategy when traversing multiple pages.

Alternatives

watercrawl, firecrawl, scrapy, crawlee