Changing the site's structure and design
Change in site structure
You may need to change the site page URLs in the following cases:
- Change in site structure.
- Introduction of user-friendly URLs.
- CMS changes or upgrades.
Simply creating a new page won't help. The indexing robot will recognize the page by the old and by the new address as two separate pages. They will be indexed and displayed by the search queries independently. To change the page URL correctly in the search results, we recommend the following:
- Set redirection from the old address of the page to the new one (301 server redirect). In this case, the robot indexes the redirection target and removes previously known pages from the search. As the robot crawls the site, the new documents appear in the search results. If needed, you can use a chain of redirects. The final destination page will be index and included in the search. You can check if the page is indexed on the Crawl statistics page in Yandex.Webmaster.
- Check that the new page URL is available for the robot, that the server responds with the 200 OK code and returns the content. To do this, use the Server response check tool in Yandex.Webmaster. Also, make sure that the most important pages are not prohibited for indexing in the robots.txt file. To do this, use the Robots.txt analysis tool.
- If some pages are already missing on the website, but the website topic hasn't changed and you can find similar products (movies, articles, and so on), set up a 301 redirect to the main page of the corresponding section or to a section with a similar topic. If you change the site structure and content, you must configure the server so that the former pages return the HTTP 404 code in the header. In this case, the old pages will be removed from the search output after they are re-crawled by the robot. To speed up the removal of non-existent sections or page types, you can prohibit their indexing in the robots.txt file.
- To speed up the indexing of new pages, use the Reindex pages tool. Pages sent for reindexing appear in the search within two weeks. Old pages with redirections can be also sent for reindexing. This helps the robot detect the redirect faster and exclude the pages from the search (this also takes up to two weeks). Please read our General recommendations on indexing.
- Make sure that the site's navigation menu contains new URLs. On the External links page in Yandex.Webmaster, you can see which third-party resources refer to the old pages. If possible, ask the owners of these resources to update the links.
next to it.Changing the site's design
If you changed the site's design and layout without changing URLs, make sure that the content on the new pages is available for indexing:
- Make sure that the pages return the full content to the robot. If they use JavaScript code, the robot will not be able to index the content generated by the script. The content you want to include in the search should be available in the HTML code immediately after requesting the page, without using JavaScript code. To do this, use HTML copies.
- Make sure that the HTML code of the pages doesn't contain the prohibiting robots meta tag with the noindex directive and therel="canonical" attribute of the link element that specifies another page as canonical (for example, home page or main page of the catalog). These elements don't allow the robot to include new pages in the search.
- Make sure that all opening noindex tags are closed.
- Make sure that the Keywords and Descriptionmeta tags and the title element on the new pages are filled correctly.
