3
User
Recs.
Recs.
3
Updates
Last
Updated
Updated
Activity
6 Options
SpecsUpdate
LanguagePython
Respects robots.txtOptionally (default=yes)
Rate LimitsYes, global or per domain, etc.
Pros
Pro Comprehensive framework
While a lot of the other tools e.g. BeautifulSoup involve stringing together multiple libraries and custom code to get the job done, Scrapy is very convention-oriented (generates skeleton files and projects for you) and has built-in solutions or plugins for most tasks e.g. IP rotation, pagination, multi-step processing/filtering.
Know any positive aspects of this option?
Cons
Know any negative aspects of this option?