Introducing
The Slant team built an AI & it’s awesome
Find the best product instantly
Add to Chrome
Add to Edge
Add to Firefox
Add to Opera
Add to Brave
Add to Safari
Try it now
4.7 star rating
0
Development
Backend Development
Web
What are the best web crawlers/spiders?
15
Options
Considered
45
User
Recs.
Dec 30, 2023
Last
Updated
Related Questions
Activity
Have feedback or ideas?
Join our community
on Discord
Ad
11
Options
Considered
Best web crawlers/spiders
Price
Language
Respects robots.txt
--
Scrapy
-
Python
Optionally (default=yes)
--
Scraperjs
-
Node.js
-
--
Advanced Web Scraper
-
-
-
--
node-crawler
-
-
-
--
Portia
-
-
-
See Full List
--
Scrapy
My Rec
ommendation
for
Scrapy
My Recommendation for
Scrapy
All
6
Pros
5
Specs
Top
Pro
•••
Interactive shell to debug and set up extraction
Scrapy shell
See More
Specs
Language:
Python
Respects robots.txt:
Optionally (default=yes)
Rate Limits:
Yes, global or per domain, etc.
Top
Pro
•••
Allows rate limiting
See More
Top
Pro
•••
Has a bunch of extensions
Like: scrapy-redis: Stores scrapped info in a Redis DB. distribute_crawler: Distributed Scrapy cluster using Redis/MongoDB and graphite. scrapy-cluster: Distributed Scrapy cluster using Redis and Kafka. django-dynamic-scraper: Create scrapers via a Django admin web UI
See More
Top
Pro
•••
Extensible
You can extend various parts of the framework via middlewares (like Django).
See More
Top
Pro
•••
Popular
GitHub with over 20k stars.
See More
Hide
See All
Get it
here
Recommend
18
--
Scraperjs
My Rec
ommendation
for
Scraperjs
My Recommendation for
Scraperjs
All
4
Pros
1
Cons
2
Specs
Top
Pro
•••
Can scrape dynamic websites
See More
Top
Con
•••
No rate limiting
Doesn't seem to have a built-in requests/second limit.
See More
Specs
Language:
Node.js
Top
Con
•••
Not updated
Last released on 2015.
See More
Hide
See All
Get it
here
Recommend
3
--
Advanced Web Scraper
My Rec
ommendation
for
Advanced Web Scraper
My Recommendation for
Advanced Web Scraper
Hide
Get it
here
Recommend
6
--
node-crawler
My Rec
ommendation
for
node-crawler
My Recommendation for
node-crawler
All
1
Pros
1
Top
Pro
•••
Allows rate limiting
See More
Hide
Get it
here
Recommend
1
--
Portia
My Rec
ommendation
for
Portia
My Recommendation for
Portia
All
4
Pros
3
Cons
1
Top
Pro
•••
Doesn't require programming knowledge
See More
Top
Con
•••
Free version limited 5 pages/min
Pricing, also no IP rotation on their free plan. However it's open source so you can run it on your machine for free without those limitations.
See More
Top
Pro
•••
Supports Scrapy
https://scrapinghub.com/scrapy-cloud
See More
Top
Pro
•••
IP rotation (rental)
Crawlera allows to use many different IPs and avoid IP-ban when crawling.
See More
Hide
See All
Get it
here
Recommend
1
--
Web Scraper (Chrome App)
My Rec
ommendation
for
Web Scraper (Chrome App)
My Recommendation for
Web Scraper (Chrome App)
All
5
Pros
2
Cons
2
Specs
Top
Pro
•••
Fully support dynamic content
Even one page websites can be scraped as it run in a real fully fetched browser.
See More
Top
Con
•••
Not so easy to use
The interface isn't that intuitive especially if you want to follow links.
See More
Specs
Language:
JavaScript*
Top
Pro
•••
Easy to install and use
It's a Chrome App so if you've Chrome, it's one click to start using it on any platform.
See More
Top
Con
•••
Cannot perform complex operations
Has a basic UI but you cannot script it really to customize certain aspects.
See More
Hide
See All
Get it
here
Recommend
1
1
--
JetOctopus
My Rec
ommendation
for
JetOctopus
My Recommendation for
JetOctopus
Hide
110
Recommend
1
--
Norconex HTTP Collector
My Rec
ommendation
for
Norconex HTTP Collector
My Recommendation for
Norconex HTTP Collector
All
4
Pros
3
Specs
Top
Pro
•••
Easy to run
See More
Specs
Language:
Java
Respects robots.txt:
Yes
Top
Pro
•••
Powerful
Can crawl sites with millions of document
See More
Top
Pro
•••
Flexible
There is a ton of option that can be configured through the XML file.
See More
Hide
See All
Free (Open-source)
Recommend
2
--
simplecrawler
My Rec
ommendation
for
simplecrawler
My Recommendation for
simplecrawler
All
1
Specs
Specs
Language:
Node.js
Hide
Get it
here
Recommend
1
--
pholcus
My Rec
ommendation
for
pholcus
My Recommendation for
pholcus
All
3
Pros
1
Cons
1
Specs
Top
Pro
•••
Distributed and high concurrency
Is capable crawling lots of web pages very rapidly.
See More
Top
Con
•••
Documentation only in Chinese
See More
Specs
Language:
Go
Hide
See All
Get it
here
Recommend
--
scrape-it
My Rec
ommendation
for
scrape-it
My Recommendation for
scrape-it
All
2
Cons
1
Specs
Top
Con
•••
Mostly single page scrapping
Doesn't seem to have built-in support for recursive scrapping of many pages.
See More
Specs
Language:
Node.js
Hide
Get it
here
Recommend
1
Don't see your favorite option? Add it.
--
Netpeak Spider
My Rec
ommendation
for
Netpeak Spider
My Recommendation for
Netpeak Spider
$9.80/mo
Recommend
2
--
JetOtopus
My Rec
ommendation
for
JetOtopus
My Recommendation for
JetOtopus
Logs+crawl+GSC - 120$/month
Recommend
2
--
ProxyCrawl
My Rec
ommendation
for
ProxyCrawl
My Recommendation for
ProxyCrawl
Get it
here
Recommend
2
--
Oxylabs
My Rec
ommendation
for
Oxylabs
My Recommendation for
Oxylabs
$ 400
Recommend
3
See flagged products
Hide flagged products
Built By the Slant team
Find the best product instantly.
4.7 star rating
Add to Chrome
Add to Edge
Add to Firefox
Add to Opera
Add to Brave
Add to Safari
Try it now - it's free
One sec!
Are you sure that you want to abandon your hard work?
Delete Work
Continue working
{}
undefined
url next
price drop