Robots meta tag and X-Robots-Tag HTTP header

You can specify for the robots the rules for loading and indexing certain site pages in one of the following ways:
  • Place the robots meta tag inside the head element of the HTML code of the page.
  • Configure the X-Robots-Tag HTTP header for a specific URL on your site's server.
Note. If the page is prohibited in the robots.txt file, the meta tag or header directive does not apply.

By default, the search robots take into account the meta tag and header. You can specify directives for certain robots.

  1. Directives supported by Yandex
  2. Specifying multiple directives
  3. Instructions for specific robots

Directives supported by Yandex

Directive Description Robots meta tag X-Robots-Tag header
noindex Prohibits indexing the page text. The page won't be included in the search results.
nofollow Prohibits following the links on the page. The robot will not follow links when crawling the site, but it can find out about them from other sources. For example, on other pages or sites.
none Similar to the noindex and nofollow directives.
noarchive Prohibits showing the link to the saved copy on the search results page.
noyaca Prohibits using an automatically generated description.
index | follow | archive Disables the corresponding prohibiting directives.
all Allows you to index text and links on the page, similar to the index and follow directives.

The robot uses allow directives by default, so you can omit them if there are no other directives. Allow directives have priority over prohibiting directives if there's a combination of both. Example.

Robots from other search engines and services may interpret the directives in a different way.

Example:

An element that disables page indexing.

<html>
    <head>
        <meta name="robots" content="noindex" />
    </head>
    <body>...</body>
</html>

Specifying multiple directives

You can specify multiple directives separated by commas.

<meta name="yandex" content="noindex, nofollow" />

If conflicting directives are specified for the Yandex robot, it will consider a positive value. Example of meta tag directives:

<meta name="robots" content="all"/>
<meta name="robots" content="noindex, follow"/>
<!-- The robot will choose the value "all", the text and the links will be indexed.-->
<meta name="robots" content="all"/>
<meta name="robots" content="noarchive"/>
<!--The text and links will be indexed, but there won't be a link 
to the saved copy of the page.-->

Instructions for specific robots

You can use the robots meta tag to give a directive to the Yandex robots only. Example:

<meta name="yandex" content="noindex" />

If you list common directives and directives for the Yandex robots, the search engine will take into account all of them.

<meta name="robots" content="noindex" />
<meta name="yandex" content="nofollow" />

The Yandex robot will treat these directives as noindex, nofollow.

If pages don't appear in the search results for a long time or were excluded from them, provide examples of such pages in the form.