SEO

Noindex

Noindex is a rule that tells search engines to exclude a specific page from their search index. It is applied through a meta robots tag in the HTML <head> or an X-Robots-Tag header in the HTTP response, and any page it covers is kept out of search results even when external links point to it.

Noindex is a directive that tells search engines to drop a page from their search index.
It can be applied in two ways: a meta robots tag inside the HTML <head>, or an X-Robots-Tag header in the HTTP response.
For non-HTML resources such as PDFs, images, and videos, use the X-Robots-Tag header.
The most common mistake is blocking the page with robots.txt: if a crawler cannot reach the page, it can never read the noindex directive in the first place.

Overview

Noindex is a rule that tells a search engine to leave a given page out of its search index. Once Googlebot recognizes the directive, it drops the page from search results entirely, even when external links point to it. Use it to control pages that have no business being indexed, such as login pages, internal search results, duplicate content, and thank-you pages.

There are two ways to apply it. One is to add a meta robots tag to the page <head>; the other is to have the server send an X-Robots-Tag HTTP header in its response. Google states that any rule you can set in a robots meta tag can be specified identically with X-Robots-Tag.

How to Apply It

meta robots Tag

To target every search engine that supports noindex, add the following tag to the page <head>.

<meta name="robots" content="noindex">

To target Google's crawler specifically, set the name value to googlebot.

<meta name="googlebot" content="noindex">

To both exclude the page from the index and prevent links from being followed, combine the rules.

<meta name="robots" content="noindex, nofollow">

X-Robots-Tag HTTP Header

Non-HTML resources such as PDFs, images, and videos have no <head> to place a tag in, so specify the directive in the response header instead.

HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
(…)
X-Robots-Tag: noindex
(…)

On Apache, configure the header to be attached to responses for specific file extensions.

<Files ~ "\.pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

On Nginx, the equivalent configuration looks like this.

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex, nofollow";
}

Common Mistake

The most frequent mistake is blocking the very page you want to noindex with robots.txt. Google states that for a noindex rule to take effect, the page or resource must not be blocked by robots.txt and must remain accessible to the crawler. If you block it with robots.txt, the crawler can never fetch the page, so it never reads the noindex directive inside it, and as a result the page may still surface in search results through external links.

So if you want a page out of the index, do not block it with robots.txt; allow crawling but deliver the noindex directive. After applying it, use the URL Inspection tool in Search Console to confirm that Googlebot actually recognizes the noindex rule, and check the detected noindex status in the Page Indexing report.

Aspect	meta robots tag	X-Robots-Tag header
Where applied	HTML `<head>`	HTTP response header
Applies to	HTML pages	Non-HTML resources: PDFs, images, videos, etc.
Set by	Page markup	Server configuration

Basis

The code examples and rules above are based on Google Search Central's official documentation, "Block Search Indexing with noindex" and "Robots Meta Tags Specifications." The warning that a robots.txt block neutralizes noindex, the note that the meta tag and X-Robots-Tag support the same rules, and the Apache and Nginx configuration examples were all confirmed directly from that documentation.