Crawlab

Use it when

• You run dozens of spiders in different languages and frameworks (Scrapy, Puppeteer, Selenium, plain scripts) and need one dashboard to deploy, schedule, and monitor them instead of cron plus ssh on each server.
• Your crawl workload outgrew one machine and you need distributed worker nodes with centralized task queues, logs, and result storage rather than a Scrapyd-only stack.

What it solves

Not the fit when

• Writing or generating the crawler code itself; it manages and runs spiders you provide
• Anti-bot evasion, proxy rotation, or content extraction; those live in the spiders
• Lightweight single-spider projects where a distributed control plane is unnecessary
• Using the README quick-start compose unchanged in production; it uses unpinned Crawlab images and MongoDB 4.2
• Spider versioning, which the README notes is not yet supported
• writing the scraper itself
• bypassing bot detection
• parsing html content
• single one-off page fetch

Install

For a local evaluation, clone the documented examples repository, enter `examples/docker/basic`, and run `docker-compose up -d`; alternatively create the README's master/worker/MongoDB compose file and start it the same way. Treat that compose file as a quick-start example, not a production manifest: it uses unpinned `:latest` Crawlab images and MongoDB 4.2.

Invoke

Open http://localhost:8080 for the web UI to deploy spiders, schedule cron jobs, and view task logs and results; integrate Scrapy via crawlab.scrapy.pipelines.CrawlabPipeline in ITEM_PIPELINES, or call save_item() from the crawlab Python SDK in any spider

Alternatives

No reviewed alternatives recorded yet.