The robots.txt file contains strict instructions for the bot about how to index the site. For more information about this file, see the Help section.
If you are using a redirect to change the site's protocol or to move the site to a new address, the indexing bot can't access the robots.txt file at the old address. To make the indexing bot load the file, you need to add the new site address in Yandex.Webmaster and check the robots.txt file for the new address.
The following conditions must be met in order for robots.txt to be loaded by the tool and read by the indexing bot:
- The file must be located in the site's root directory. You can't put the file in internal folders on the site.
- The file name must be robots.txt. You can't use uppercase letters or other names for the file.
- When the file is requested, the server must return the HTTP code 200 OK. In any other cases, the robot can't load the file. You can check the HTTP code that the server returns for a request for the robots.txt file.
Sometimes the robots.txt analysis tool uses outdated file data. This might be related to cached information, or it might take time for changes to be applied by the hosting provider (this is normally no more than an hour).
If the updated information doesn't appear in the robots.txt analysis tool for a long time, please contact the administrator of the server where the site resides, or your hosting provider. This might be related to restricted rights for editing the robots.txt file.