X-Robots-Tag
X-Robots-Tag delivers indexing directives such as noindex and nofollow to search engines through an HTTP response header rather than an HTML <meta> tag. Because it operates at the header level, it can apply indexing rules even to non-HTML files like PDFs and images.
- X-Robots-Tag passes indexing directives such as noindex and nofollow to search engines through an HTTP response header.
- Unlike the meta robots tag that lives inside the HTML
<head>, it can also be applied to non-HTML files such as PDFs, images, and videos. - It can be applied in bulk to specific file types with regular expressions in the server configuration (Apache, Nginx), which is useful for large sites.
- For noindex to take effect, the page must not be blocked by robots.txt and must remain accessible to crawlers.
Overview
X-Robots-Tag is a way to deliver directives that control a search engine's indexing behavior in the form of an HTTP response header. The meta robots tag commonly used for index control is written inside the <head> of an HTML document, but X-Robots-Tag carries the value directly in the header when the server sends its response.
Because of this difference, X-Robots-Tag can apply indexing rules even to non-HTML files. Resources that have no place to insert a <meta> tag, such as PDF documents, images, and videos, still always include an HTTP header in the response, so indexing can be controlled through server configuration alone.
The basic form of the response header looks like this:
HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
X-Robots-Tag: noindexTo apply multiple directives together, separate them with commas or split them across several header lines.
X-Robots-Tag: noindex, nofollow
X-Robots-Tag: unavailable_after: 25 Jun 2010 15:00:00 PSTThe supported directive values are the same as those of the meta robots tag, including noindex, nofollow, none, nosnippet, noimageindex, notranslate, max-snippet, max-image-preview, and unavailable_after.
Server Configuration Examples
Server configuration is effective when you want to block indexing across a specific file type all at once. The examples below exclude every PDF file from indexing.
Apache (.htaccess or httpd.conf):
<Files ~ "\.pdf$">
Header set X-Robots-Tag "noindex, nofollow"
</Files>Nginx (site .conf file):
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex, nofollow";
}Because rules can be applied per file extension with regular expressions, this is easier to manage than inserting a meta tag into every individual page. Nginx requires a restart after a configuration change, whereas Apache applies changes simply by editing the .htaccess file.
Difference from the Meta Robots Tag
The meta robots tag is written inside the <head> of an HTML document in the form <meta name="robots" content="noindex"> and can only be used on HTML pages. X-Robots-Tag, on the other hand, is an HTTP header method, so it can be applied to any response regardless of whether the content is HTML.
- Meta robots tag: written inside the HTML
<head>, for HTML pages only - X-Robots-Tag: delivered via an HTTP response header, applicable to non-HTML files such as PDFs and images, and able to handle them in bulk through server configuration
Both methods support the same directive values, so the outcome is identical. Whichever you use, choose based on the file types you need to block from indexing and your operating environment.
Rationale
Google Search Central documentation states that, instead of a meta robots tag, you can return an X-Robots-Tag HTTP header with a value of noindex or none, and it provides the Apache/Nginx configuration examples and response header format shown above. It also explains that for index blocking (such as noindex) to take effect, the page or resource must not be blocked by robots.txt and must be accessible to crawlers. If it is blocked by robots.txt, the crawler cannot fetch the page and therefore cannot read the X-Robots-Tag directive.