Meta Robots Tag
The meta robots tag is a <meta name="robots"> element placed in the <head> of an HTML document that gives search engines page-level instructions on how to index the page and display it in search results. Through directives such as noindex and nofollow, it controls on a per-page basis whether a page is indexed and whether its links are followed.
- The meta robots tag is the
<meta name="robots">element in an HTML<head>, an HTML-based way to tell search engines how to index and crawl an individual page. - Its most common directives are
noindex(keep the page out of search results),nofollow(don't follow links on the page),nosnippet(don't generate a snippet), andmax-snippet(cap snippet length). - It differs from robots.txt, which blocks crawling itself, and from X-Robots-Tag, which carries the same directives but in an HTTP header rather than in the page. On pages blocked by robots.txt, the meta robots tag is ignored.
Overview
The meta robots tag is a <meta name="robots"> element that sits in the <head> of an HTML document and controls how a search engine indexes and displays that single page in search results. The value name="robots" applies to all crawlers; to target a specific crawler, you name it instead, as in name="googlebot". You can combine multiple directives in the content attribute by separating them with commas.
Key Directives
| Directive | Meaning |
|---|---|
noindex | Tells the engine not to show this page in search results |
nofollow | Tells the engine not to follow links on this page |
none | Equivalent to combining noindex, nofollow |
nosnippet | Suppresses text snippets and video previews in search results (also blocks use in AI overviews) |
indexifembedded | Allows indexing of content embedded via iframe even when noindex is present (only works alongside noindex) |
max-snippet:[number] | Limits the text snippet to the specified character count (0 = no snippet, -1 = Google decides the length) |
max-image-preview:[setting] | Controls image preview size (none/standard/large) |
max-video-preview:[number] | Limits the maximum video preview length in seconds (0 = still image only, -1 = no limit) |
noimageindex | Prevents the images on this page from being indexed |
notranslate | Prevents a translated version from being offered in search results |
unavailable_after:[date] | Removes the page from search results after the specified date (RFC 822, RFC 850, or ISO 8601 format) |
Note that noarchive, nocache, and nositelinkssearchbox are directives that Google Search no longer uses.
Code Examples
The basic form that blocks indexing and also stops links from being followed:
<meta name="robots" content="noindex, nofollow">An example that targets only a specific crawler to control the snippet and image preview:
<meta name="googlebot" content="nosnippet, max-image-preview:large">To exclude only part of the body from a snippet, use the data-nosnippet attribute:
<p>Visible sentence <span data-nosnippet>excluded from snippet</span>.</p>Differences from robots.txt and X-Robots-Tag
The three operate at different layers and are delivered in different places. The meta robots tag is page-level index and display control placed inside the HTML <head>; X-Robots-Tag delivers the same directives through an HTTP response header; and robots.txt is file-level control that blocks crawling (fetching) itself.
| Type | Where it lives | Role |
|---|---|---|
| Meta robots tag | <meta> inside the HTML <head> | Per-page control of indexing and search-result display (HTML pages only) |
| X-Robots-Tag | HTTP response header | Delivers the same directives via a header; suited to non-HTML files such as images and PDFs, and to site-wide rules |
| robots.txt | Text file at the site root | Blocks crawlers from fetching the resource at all |
The crucial difference is when each one takes effect. Per Google's official documentation, a page blocked by robots.txt is never fetched by the crawler, so any meta robots tag or X-Robots-Tag directive inside it cannot be read and is ignored. To reliably keep a specific page out of the index, therefore, you should not block it with robots.txt; instead, allow crawling and use a noindex meta robots tag (or X-Robots-Tag). A robots.txt block is only a "don't fetch" signal, not a "don't index" instruction, and a blocked URL can still be indexed if it is discovered through another path.
In addition, the meta robots tag cannot be applied to non-HTML resources such as images, videos, and PDFs, which have no HTML <head>; in those cases you use the X-Robots-Tag header. X-Robots-Tag also has the advantage of letting you apply site-wide rules in bulk through regular expressions and server configuration.