Commonly Compared
Norconex HTTP Collector
vs
SpecsUpdate
LanguageJava
Respects robots.txtYes
Ranked in these QuestionsQuestion Ranking
Pros
Pro Easy to run
Pro Powerful
Can crawl sites with millions of document
Pro Flexible
There is a ton of option that can be configured through the XML file.
Commonly Compared
Norconex HTTP Collector
vs