Read Aloud doesn't respect robots.txt rules. User agent tokenĬaution: Feedfetcher doesn't respect robots.txt rules. Mobile Apps Android ignores the * wildcard. (Various mobile device types) (compatible Mediapartners-Google/2.1 +)

Mozilla/5.0 (Linux Android 6.0.1 Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ W.X.Y.Z Mobile Safari/537.36 (compatible Googlebot/2.1 +)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko compatible Googlebot/2.1 +) Chrome/ W.X.Y.Z Safari/537.36.

Mozilla/5.0 (compatible Googlebot/2.1 +).

Through the Google Publisher Center to be used in Google News landing pages. User agent tokenĬaution: Google Publisher Center doesn't respect robots.txt rules.įetches and processes feeds that publishers explicitly supplied Mozilla/5.0 (iPhone CPU iPhone OS 14_7_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Mobile/15E148 Safari/604.1 (compatible AdsBot-Google-Mobile +)Ĭhecks desktop web page ad quality. Mozilla/5.0 (Linux Android 5.0 SM-G920A) AppleWebKit (KHTML, like Gecko) Chrome Mobile Safari (compatible AdsBot-Google-Mobile +)ĪdsBot Mobile Web ignores the * wildcard.Ĭhecks iPhone web page ad quality. Learn how to verify if a visitor is a Google crawler.ĪdsBot Mobile Web Android ignores the * wildcard.Ĭhecks Android web page ad quality. The full user agent string is a full description of the crawler, and appears inĬaution: The user agent string can be spoofed. This list is not complete, but covers most of the crawlers you might see on your One token, as shown in the table you need to match only one crawler token for a rule toĪpply. To match a crawler type when writing crawl rules for your site. The user agent token is used in the User-agent: line in robots.txt

The following table shows the crawlers used by various products and services at Google: Is used to automatically discover and scan websites by following links from one web page toĪbout the common Google crawlers you may see in your referrer logs, and how to specify them in 'Crawler' (sometimes also called a 'robot' or 'spider') is a generic term for any program that

Overview of Google crawlers (user agents)

Firefox mobile user agent