Botify Review

Botify lets you compare what urls googlebot is actually crawling vs the compliant urls you're trying to get indexed. This is an incredibly powerful feature for complex/large websites where you're trying to get longtail content ranked. Using this data you can optimise your sitemaps etc to point google to content it is currently missing.

The data in botify is broken up into current urls, disappeared urls and new urls. This will let you keep track of changes made to your site between crawls.

The botify UI for building reports is very polished and powerful. It supports regex, multiple levels of AND/OR and lets you filter on pretty much any field/data you can think of.

You can set how deep Botify will crawl into your website, as well as how fast and how many URLs. Alternatively, you can set only certain sections of your website to be crawled. Previous crawls can be compared to new crawls to show differences and improvements.

You can easily share reports with co-workers, and set each user to an authorization level (guest/admin/owner) to ensure the right people can make edits, and others can only view the information.

Botify will also crawl through competing sites, allowing you to compare your SEO to theirs, and see where would be most beneficial for you to improve your site's SEO.