Why are pages excluded from the search?
Site pages can disappear from the Yandex search results for a number of reasons. To find out why a page was excluded, open Yandex.Webmaster and go to and select Excluded pages. Learn more about the Excluded pages block
| Reason for excluding a page | Solution |
|---|---|
| The page was considered low-value or in low-demand | The algorithm decided not to include the page in search results because demand for the page is probably low. For example, this can happen if there's no content on the page, if the page is a duplicate of pages already known to the robot, or if its content doesn't completely suit user interests. The algorithm automatically checks the pages on a regular basis, so the decision may change later. To learn more, see Low-value or low-demand pages. |
| An error occurred when the robot was loading or processing the page, and the server response contained HTTP status code 3XX, 4XX, or 5XX. | To find the error, use the Server response check tool. If the page is accessible to the robot, make sure that:
|
| Page indexing is prohibited in the robots.txt file or using a meta tag with the noindex directive. | Remove the prohibiting directives. If you didn't place the ban in robots.txt yourself, contact your hosting provider or domain name registrar for details. Also make sure that the domain name isn't blocked due to the registration period expiry. |
| The page redirects the robot to other pages | Make sure that the excluded page should actually redirect users. To do this, use the Server response check tool. |
| The page duplicates the content of another page | If the page is identified as duplicate by mistake, follow the instructions in the Duplicate pages section. |
| The page is not canonical | Make sure that the pages should actually redirect the robot to the URL specified in the rel="canonical" attribute. |
| The site is recognized as a secondary mirror | If the sites are grouped by mistake, follow the recommendations in the Separating site mirrors section. |
| Violations are found on the site | You can check this by going to the page in Yandex.Webmaster. |
| Reason for excluding a page | Solution |
|---|---|
| The page was considered low-value or in low-demand | The algorithm decided not to include the page in search results because demand for the page is probably low. For example, this can happen if there's no content on the page, if the page is a duplicate of pages already known to the robot, or if its content doesn't completely suit user interests. The algorithm automatically checks the pages on a regular basis, so the decision may change later. To learn more, see Low-value or low-demand pages. |
| An error occurred when the robot was loading or processing the page, and the server response contained HTTP status code 3XX, 4XX, or 5XX. | To find the error, use the Server response check tool. If the page is accessible to the robot, make sure that:
|
| Page indexing is prohibited in the robots.txt file or using a meta tag with the noindex directive. | Remove the prohibiting directives. If you didn't place the ban in robots.txt yourself, contact your hosting provider or domain name registrar for details. Also make sure that the domain name isn't blocked due to the registration period expiry. |
| The page redirects the robot to other pages | Make sure that the excluded page should actually redirect users. To do this, use the Server response check tool. |
| The page duplicates the content of another page | If the page is identified as duplicate by mistake, follow the instructions in the Duplicate pages section. |
| The page is not canonical | Make sure that the pages should actually redirect the robot to the URL specified in the rel="canonical" attribute. |
| The site is recognized as a secondary mirror | If the sites are grouped by mistake, follow the recommendations in the Separating site mirrors section. |
| Violations are found on the site | You can check this by going to the page in Yandex.Webmaster. |
The robot continues to visit the pages excluded from the search, and a special algorithm checks the probability of displaying them in the search results before each update of the search database. So the page may appear in the search within two weeks after the robot finds out about its change.
If you fixed the reason for excluding the page, send it for reindexing. This way you'll inform the robot about the changes.
Questions and answers about pages excluded from the search
- Headers that the robot requests from the server differ from the headers that the browser requests. So excluded pages might open correctly in the browser.
- It is not available to the robot for a certain period of time.
- It is not linked to from other pages on the site and external sources.
In addition to checking the tags on the page, the algorithm checks if the page content is unique, informative, in-demand and up-to-date, as well as many other factors. However, you should pay attention to meta tags. For example, the Description meta tag and the title element can be created automatically and duplicate each other.
If the site contains a lot of similar products that differ only by color, size or configuration, they may be excluded from the search. The same is true about the pagination pages, product selection and comparison pages, and image pages without text content.
This can happen for several reasons:
The Excluded pages list on the Pages in search page shows the pages the robot accessed but didn't index (these may be non-existing pages previously known to the robot).
A page is removed from the excluded list if:
Excluded pages listed in the service don't affect the site position in the search results.
