18
User
Recs.
Recs.
13
Updates
Last
Updated
Updated
Activity
15 Options
SpecsUpdate
LanguagePython
Respects robots.txtOptionally (default=yes)
Rate LimitsYes, global or per domain, etc.
Pros
Pro Has a bunch of extensions
Like:
- scrapy-redis: Stores scrapped info in a Redis DB.
- distribute_crawler: Distributed Scrapy cluster using Redis/MongoDB and graphite.
- scrapy-cluster: Distributed Scrapy cluster using Redis and Kafka.
- django-dynamic-scraper: Create scrapers via a Django admin web UI
Know any positive aspects of this option?
Cons
Know any negative aspects of this option?