Commonly Compared
SpecsUpdate
LanguagePython
Respects robots.txtOptionally (default=yes)
Rate LimitsYes, global or per domain, etc.
Ranked in these QuestionsQuestion Ranking
Pros
Pro Comprehensive framework
While a lot of the other tools e.g. BeautifulSoup involve stringing together multiple libraries and custom code to get the job done, Scrapy is very convention-oriented (generates skeleton files and projects for you) and has built-in solutions or plugins for most tasks e.g. IP rotation, pagination, multi-step processing/filtering.