SEO

Googlebot

Googlebot is the primary crawler that Google Search uses to crawl the web, made up of two variants: a smartphone variant that simulates a mobile device and a desktop variant. After crawling the pages it discovers, it renders them with an evergreen Chromium engine and uses the result for indexing.

Googlebot is Google Search's primary crawler. Unlike the generic term "crawler," it refers to the specific bot that Google operates.
It comes in two variants, Googlebot Smartphone (mobile) and Googlebot Desktop, and for most sites Google indexes the mobile version first.
It works in the order discover → crawl → render with evergreen Chromium → index, handling JavaScript-generated content during the rendering stage.
Both variants obey the same robots.txt product token Googlebot, so you cannot selectively block mobile versus desktop through robots.txt.
Genuine Googlebot can be confirmed through reverse and forward DNS lookups or against Google's published IP ranges; the User-Agent string alone can be spoofed, so verification is required.

Overview

Googlebot is the collective name for the two crawlers Google Search uses to gather web pages. One is Googlebot Smartphone, which simulates a mobile user, and the other is Googlebot Desktop, which simulates a desktop user. According to the official documentation, Google Search indexes the mobile version of content first on most sites, so the smartphone variant issues far more crawl requests than the desktop variant.

Googlebot does not simply download HTML; it uses an evergreen rendering engine that always tracks the latest version of Chromium. As a result, content that is generated dynamically with JavaScript is processed during rendering. The Chrome version inside the User-Agent string (the W.X.Y.Z portion) is continually updated to match the most recent Chromium release.

How It Works: Crawl → Render → Index

Googlebot crawls the URLs it discovers to collect their responses, then renders the pages with the Chromium engine and uses those results for indexing. Per the official documentation, Googlebot throttles its requests so that on average it does not hit a single site more than once every few seconds, and it processes the first 2MB of a supported file (up to 64MB for PDFs). When crawling from a US-based IP, Googlebot's time zone is Pacific Standard Time.

User-Agent Strings

The HTTP User-Agent header on a request lets you tell which Googlebot made the visit. The representative strings specified in Google's official documentation are as follows.

Googlebot Smartphone: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot Desktop: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

Both variants obey the same robots.txt product token Googlebot. Because of this, robots.txt rules alone cannot target the smartphone and desktop variants separately to block or allow them.

Verifying Genuine Bots

Because anyone can spoof a User-Agent string, separate verification is needed to confirm that traffic is really Googlebot. The official documentation describes two methods.

Manual verification (DNS): perform a reverse DNS lookup on the connecting IP to confirm the domain ends in one of googlebot.com, google.com, or googleusercontent.com, then run a forward DNS lookup to verify it matches the original IP. The reverse lookup result takes a form such as crawl-***-***-***-***.googlebot.com.
Automatic verification (IP ranges): compare the connecting IP against the CIDR IP ranges in Google's published JSON files (such as common-crawlers.json).

Implementation Checklist

What to Allow

Do not block the JavaScript, CSS, and image resources Googlebot needs to render pages in robots.txt
Check that important pages are not blocked by a robots.txt Disallow or by noindex
Given mobile-first indexing, expose the same content and metadata on mobile pages as on desktop

What to Block and Verify

Do not blindly trust traffic posing as Googlebot; verify it via DNS or the published IP ranges before acting on it
Do not base firewall or access policies on the User-Agent string alone
To exclude specific paths from crawling, write rules with the robots.txt token Googlebot (selective blocking of smartphone versus desktop is not possible)

References and Sources

Related terms

Mobile-First Indexing