What are the best web crawlers/spiders?

Options
Considered

User
Recs.

Dec 30, 2023

Last
Updated

Scrapy vs Scraperjs vs Advanced Web Scraper

vs node-crawler vs Portia vs Web Scraper (Chrome App) vs JetOctopus vs Norconex HTTP Collector vs simplecrawler vs pholcus vs scrape-it vs Netpeak Spider vs JetOtopus vs ProxyCrawl vs Oxylabs

Specs

Language

Python

Respects robots.txt

Optionally (default=yes)

Rate Limits

Yes, global or per domain, etc.

Entry package

Corporate package

Enterprise package

Advanced package

Pro package

Language

Node.js

Respects robots.txt

Rate Limits

Entry package

Corporate package

Enterprise package

Advanced package

Pro package

Language

Respects robots.txt

Rate Limits

Entry package

Corporate package

Enterprise package

Advanced package

Pro package

Language

Respects robots.txt

Rate Limits

Entry package

Corporate package

Enterprise package

Advanced package

Pro package

Language

Respects robots.txt

Rate Limits

Entry package

Corporate package

Enterprise package

Advanced package

Pro package

Language

JavaScript*

Respects robots.txt

Rate Limits

Entry package

Corporate package

Enterprise package

Advanced package

Pro package

Language

Respects robots.txt

Rate Limits

Entry package

Corporate package

Enterprise package

Advanced package

Pro package

Language

Java

Respects robots.txt

Yes

Rate Limits

Entry package

Corporate package

Enterprise package

Advanced package

Pro package

Language

Node.js

Respects robots.txt

Rate Limits

Entry package

Corporate package

Enterprise package

Advanced package

Pro package

Language

Respects robots.txt

Rate Limits

Entry package

Corporate package

Enterprise package

Advanced package

Pro package

Language

Node.js

Respects robots.txt

Rate Limits

Entry package

Corporate package

Enterprise package

Advanced package

Pro package

Language

Respects robots.txt

Rate Limits

Entry package

Corporate package

Enterprise package

Advanced package

Pro package

Language

Respects robots.txt

Rate Limits

Entry package

Corporate package

Enterprise package

Advanced package

Pro package

Language

Respects robots.txt

Rate Limits

Entry package

Corporate package

Enterprise package

Advanced package

Pro package

Language

Respects robots.txt

Rate Limits

Entry package

$ 400/month

Corporate package

Custom pricing

Enterprise package

$ 150,000/month

Advanced package

$ 2,000/month

Pro package

$ 50,000/month

Pros + Cons

Pro
Interactive shell to debug and set up extraction

Scrapy shell

Comment

Like: scrapy-redis: Stores scrapped info in a Redis DB. distribute_crawler: Distributed Scrapy cluster using Redis/MongoDB and graphite. scrapy-cluster: Distributed Scrapy cluster using Redis and Kafka. django-dynamic-scraper: Create scrapers via a Django admin web UI

Comment

Pro
Extensible

You can extend various parts of the framework via middlewares (like Django).

Comment

Pro
Popular

GitHub with over 20k stars.

Comment

Pro
Can scrape dynamic websites

[add description]

Comment

Con
No rate limiting

Doesn't seem to have a built-in requests/second limit.

Comment

Con
Not updated

Last released on 2015.

Comment

Pro
Allows rate limiting

[add description]

Comment

Pro
Doesn't require programming knowledge

[add description]

Comment

Pro
Supports Scrapy

https://scrapinghub.com/scrapy-cloud

Comment

Pro
IP rotation (rental)

Crawlera allows to use many different IPs and avoid IP-ban when crawling.

Comment

Con
Free version limited 5 pages/min

Pricing, also no IP rotation on their free plan. However it's open source so you can run it on your machine for free without those limitations.

Comment

Pro
Fully support dynamic content

Even one page websites can be scraped as it run in a real fully fetched browser.

Comment

Pro
Easy to install and use

It's a Chrome App so if you've Chrome, it's one click to start using it on any platform.

Comment

Con
Not so easy to use

The interface isn't that intuitive especially if you want to follow links.

Comment

Con
Cannot perform complex operations

Has a basic UI but you cannot script it really to customize certain aspects.

Comment

Pro
Easy to run

[add description]

Comment

Pro
Powerful

Can crawl sites with millions of document

Comment

Pro
Flexible

There is a ton of option that can be configured through the XML file.

Comment

Pro
Distributed and high concurrency

Is capable crawling lots of web pages very rapidly.

Comment

Con
Documentation only in Chinese

[add description]

Comment

Con
Mostly single page scrapping

Doesn't seem to have built-in support for recursive scrapping of many pages.

Comment

Pro
Crawling tool and proxies combined

Real-Time SERP API does it all for you – it routes traffic through geo-located IPs, sets correct headers, reuses cookies, and more.

Comment

Pro
HTML or JSON data

It will return you a well-formatted data that can be easily handled by your backend or even frontend application framework.

Comment

Pro
2 data delivery options

Callback data delivery method, there is no need to check your task status as Real-Time SERPs API will notify you once the data is ready. Real-time SERP delivery ensures that data is retrieved on the same connection.

Comment

Pro
E-commerce

Allows you to scrape Amazon data by ASIN

Comment

Pro
Easy to use

Just give URL, and it will return you a well-formatted data that is ready to use.

Comment