Meta robots noindex: what it is and how to set it up

How does meta robots noindex work?

When Googlebot fetches a page and finds a noindex directive in its code, it does not add the URL to the index, and if the page is already indexed, it removes it after the next crawl. This is not instant: the delay between adding the tag and the page disappearing from results is however long it takes the bot to revisit the URL.

Two ways to declare noindex

For HTML pages the directive goes in the <head> as a meta tag. For files that have no <head> (PDFs, images, video), the server sends an X-Robots-Tag HTTP header with the response.

Available values

The core value is noindex. It can be combined with other directives, separated by commas: nofollow (do not follow links on the page), noarchive (do not show a cached copy), nosnippet (do not show a description in results). The value none equals noindex, nofollow.

The condition without which it fails

The search engine must be able to crawl the page in order to read the directive. If the URL is blocked in robots.txt, the bot never fetches it, never sees the tag, and the page may stay in the index.

Why do you need noindex?

Noindex removes from search the pages that users need but that should not be found through it: order confirmations, carts, account areas, internal site search, filter pages with no demand of their own, staging versions of the site.

It also keeps the index clean: a search engine judges a site by the full set of its indexed pages, and weak service URLs drag down that overall picture. Unlike canonical, which only suggests a primary version, noindex states plainly that the page does not belong in the index.

Example

<!-- Keep the page out of the index for all search engines -->
<meta name="robots" content="noindex">

<!-- The same, but for Google only -->
<meta name="googlebot" content="noindex">

<!-- Do not index and do not follow links on the page -->
<meta name="robots" content="noindex, nofollow">

For PDFs and other files without a <head>, pass the directive in a header:

# Nginx: noindex for all PDFs
location ~* \.pdf$ {
    add_header X-Robots-Tag "noindex" always;
}

# Apache (.htaccess, requires mod_headers)
<FilesMatch "\.pdf$">
  Header set X-Robots-Tag "noindex"
</FilesMatch>

What are the most common noindex mistakes?

Noindex on a page blocked in robots.txt

The bot cannot fetch the page, so it never sees the directive. If you need a URL out of the index, first open it for crawling, wait until it drops out of results, and only then block it if needed.

Noindex left in place after a release

The most expensive mistake: during development a site or section is closed from indexing, and at launch the tag is forgotten. Traffic drops and the cause is not obvious. After every release, check the code of your key pages.

Noindex together with a canonical to another URL

Canonical says "the primary version is a different page" while noindex says "this page does not belong in the index". The signals contradict each other and the search engine decides on its own. Pick one depending on your goal.

Noindex on pages that should rank

Adding the directive across the board through a template or an SEO plugin can hit categories, pagination or sections you need. Check exactly which page types a rule applies to.

Expecting noindex to save crawl budget

To read the tag, the bot has to fetch the page, which costs a request. If your goal is to reduce crawl load, you need robots.txt, consolidation of duplicates and cleaner internal links.

How do you check noindex?

Open the URL Inspection tool in Google Search Console and enter the page address: if the directive works, the status will read "Excluded by 'noindex' tag". The Page indexing report gives the overall picture: it shows how many pages are excluded because of noindex, and that list is worth comparing against your expectations.

Without Search Console, you can check the directive in the page source (find meta name="robots" in DevTools) or check the header with a command:

curl -sI https://example.com/file.pdf | findstr /i x-robots-tag