Site structure

Yandex.Webmaster shows site sections that contain at least 10 pages and make up more than 1% of the total number of pages loaded into the search database. Sections are displayed in a tree hierarchy according to the structure of the site's URLs. Information is available for each site section on the number of pages that were visited by the Yandex robot and that participate in the search.

The data presented helps you determine if the robot downloads the site sections that are meant for the users. You can prohibit loading pages from certain site sections in the robots.txt file.
Note. If you see in Yandex.Webmaster any sections that are not present on the site, it's likely that links to them are placed on other sites. To stop the robot from crawling the links, prohibit indexing the site sections with the Disallow directive in the robots.txt file.

Adding a section

You can get indexing information about any site section, even if it doesn't make up 1% of all pages loaded to the search engine database. To get this data, click the Add section button and specify the section starting with "/". For example, "/catalog1".

You can use the * operator to specify a non-exact match to the name of the site section:

catalog1* Adds all sections with URLs beginning with "catalog1"
*catalog1 Adds all sections with URLs ending with "catalog1"
*catalog1* Adds all sections with URLs containing "catalog1"

Changes will take effect after a few days.


What does “loaded” mean?

The number of pages on your site that the Yandex robot accessed. This may include indexed pages as well as pages that returned an error.

What does “pages available in search” mean?

The pages that participate in the Yandex search. A page might not be used in search results if it duplicates the content of another page, contains spam or viruses, redirects to another URL, or an error occurred when downloading it.

To find out why the page is missing in the Yandex search, use the Check URL status tool.

Why aren't all pages in a section indexed?

Most likely, not enough time has passed since the robot found out about the site. Update the Sitemap file and wait until the robot indexes it.

How can I prohibit the Yandex robot from visiting pages in a particular section?

To prohibit indexing a site section, enter the following in the robots.txt file:

User-agent: Yandex
Disallow: /section/

At the next site indexing, the robot will take this into account and remove the section. Exceptions for the robots.txt file are described in the Yandex robots in server logs section.