Google has now added new details that explain the three categories its Google crawlers fall into, they include Googlebot, special-case crawlers and user-triggered fetchers. In addition, Google now lists a JSON formatted file containing the list of IP addresses each of these different crawler types use. Types of Google crawlers. At the top of this Googlebot page, Google listed these three crawler types: Googlebot – The main crawler for Google’s search products. Google says this crawler always respects robots.txt rules. Special-case crawlers – Crawlers that perform […]