What's wrong with duplicate pages?

What are duplicate pages?

Duplicate pages are pages with nearly or completely identical text content that belong to the same site but have different URLs.

For example, a home page can have multiple site addresses:

• https://example.com/;
• https://www.example.com/;
• https://example.com/index;
• https://example.com/index.html;
• https://example.com/?utm_source=link&utm_medium=source-example&utm_campaign=partner-offer.

For pages with matching text, the indexing bot creates a group of duplicates. It then selects one page from this group to be displayed in search results. Occasionally, the bot may change its choice to another duplicate.

What's wrong with duplicate pages?

• The bot indexes multiple pages instead of one. Crawling duplicate pages wastes time as well as the resources of your site and Yandex Search.
• It may take longer to index new pages.
• Duplicate pages may compete with each other in search results.
• The indexing bot can consider a duplicate and exclude from search results a landing page that is important for your site.

Why are there duplicate pages?

Duplicate pages can emerge due to:

• Specific features of your content management system (CMS). For example, page URLs may have or have not a trailing slash (/).
• Web server settings that make site pages accessible over HTTP or HTTPS and with or without the www prefix.
• Adding GET parameters to links, such as tracking UTM tags used by advertising systems.
• The same page appearing in different site sections under different URLs.

In Yandex Webmaster, can I check which pages Yandex Search considers duplicates?

To get a list of duplicates, use the Indexing → Searchable pages tool: open the Excluded pages tab, find the Status column, and apply the Duplicate filter. For more details, click the three dots.

To see if a specific page is a duplicate, insert its address in the URL filter.

To find duplicates emerged due to adding GET parameters to links, run diagnostics: Website optimization → Site diagnostics. Information about duplicates will appear in the critical issues section.

In addition, Yandex Webmaster flags critical issues on the Summary page.

Learn more:

• Searchable pages
• Critical issues

How to remove duplicate pages from Yandex Search?

• Set up redirects: from alternate site addresses to the primary one, and from duplicates to the desired page.
• In the page code, specify which of the duplicate pages you want to include in search results using the rel="canonical" attribute.
• Use the robots.txt file to prevent the duplicates from being indexed.
• Prevent the duplicates from being indexed by adding the noindex rule to the robots meta tag in the page code.

Learn more:

• How to get rid of duplicate pages
• What is a site move, and how do you do it?
• Primary and alternate site addresses

Ungrouping

The owner of a site that has subdomains and often appears at the top of search results may request to reclassify their domain as a web portal through Yandex Webmaster. To do this, you have to provide a description of the services on the subdomains and their owners.

Learn more:

• Ungrouping