Scrapy

Use it when

• large scale spider crawls with pipelines
• async scrapers with middleware and item pipelines

What it solves

Not the fit when

• bot-detection-challenge
• js-render-not-blocking without middleware

Install

pip install scrapy

Invoke

Run `scrapy startproject`, define a Spider that yields items and requests, enable AutoThrottle or pipelines, then execute it with `scrapy crawl NAME`.

Alternatives

crawlee