Crawlee for Python

Use it when

• Your Python scraper keeps dying on transient errors and blocks and you need built-in retries, proxy rotation, and session management
• You need one framework that crawls both plain HTML pages and JavaScript-rendered sites, with a persistent URL queue that survives interruptions

What it solves

Not the fit when

• scraping sites in violation of their terms of service
• solving captchas
• desktop app automation
• captcha solving
• PDF or document text extraction
• one-off article content extraction without a crawl

Install

python -m pip install 'crawlee[all]' && playwright install

Invoke

Instantiate BeautifulSoupCrawler or PlaywrightCrawler from crawlee.crawlers, register a @crawler.router.default_handler, then await crawler.run([urls]); or scaffold a project with: uvx 'crawlee[cli]' create my-crawler

Alternatives

No reviewed alternatives recorded yet.