---
metadata:
- name: generator
content: Diplodoc Platform v5.50.6
alternate:
- https://yandex.com.tr/support/webmaster/en/robot-workings/allow-disallow.md
- https://yandex.com.tr/support/webmaster/tr/robot-workings/allow-disallow.md
---
> **Documentation Index:** Fetch the complete configuration index at https://yandex.com.tr/support/webmaster/en/llms.txt
# Disallow and Allow directives
## Disallow {#disallow}
Use this directive to prohibit crawling of sections or individual pages of a site. For example:
- Pages that contain confidential data.
- Pages with site search results.
- Site traffic statistics.
- Duplicate pages.
- Various logs.
- Database service pages.
{% note info %}
When selecting a directive for pages that need to be left out from the search, if their addresses contain GET parameters, it's recommended to use the [Clean-param](https://yandex.com.tr/support/webmaster/en/robot-workings/clean-param.md) directive rather than Disallow. If you use Disallow, you may not be able to identify duplicate link URLs without the parameter and send some metrics of forbidden pages.
{% endnote %}
Examples:
```html
User-agent: Yandex
Disallow: / # prohibits crawling of the entire site
User-agent: Yandex
Disallow: /catalogue # prohibits crawling of pages whose addresses start with /catalogue
User-agent: Yandex
Disallow: /page? # prohibits crawling of pages whose URLs contain parameters
```
## Allow {#allow}
This directive allows crawling of sections or individual pages of a site.
Examples:
```no-highlight
User-agent: Yandex
Allow: /cgi-bin
Disallow: /
# prohibits downloading of everything except pages
# starting with “/cgi-bin”
```
```no-highlight
User-agent: Yandex
Allow: /file.xml
# allows downloading of the file.xml file
```
{% note info %}
Empty line breaks aren't allowed between the `User-agent`, `Disallow` and `Allow` directives.
{% endnote %}
## Combining directives {#simultaneous}
The `Allow` and `Disallow` directives from the corresponding `User-agent` block are sorted according to URL prefix length (from shortest to longest) and applied in order. If several directives match a particular site page, the robot selects the last one in the sorted list. This way the order of directives in the `robots.txt` file doesn't affect the way they are used by the robot.
{% note info %}
If there is a conflict between two directives with prefixes of the same length, the `Allow` directive takes precedence.
{% endnote %}
```no-highlight
# Source robots.txt:
User-agent: Yandex
Allow: /
Allow: /catalog/auto
Disallow: /catalog
# Sorted robots.txt:
User-agent: Yandex
Allow: /
Disallow: /catalog
Allow: /catalog/auto
# prohibits downloading of pages starting with “/catalog”,
# but allows downloading of pages starting with “/catalog/auto”.
```
Common example:
```no-highlight
User-agent: Yandex
Allow: /archive
Disallow: /
# allows everything containing “/archive”, the rest is prohibited
User-agent: Yandex
Allow: /obsolete/private/*.html$ # allows html files
# in “/obsolete/private/...”
Disallow: /*.php$ # prohibits all “*.php” on this site
Disallow: /*/private/ # prohibits all subpaths containing
# “/private/”, but Allow above negates
# part of this prohibition
Disallow: /*/old/*.zip$ # prohibits all “*.zip” files containing
# in “/old/”
User-agent: Yandex
Disallow: /add.php?*user=
# prohibits all “add.php?” scripts with the “user” parameter
```
## Allow and Disallow directives without parameters {#parameterless}
If directives don't contain parameters, the robot handles the data as follows:
```no-highlight
User-agent: Yandex
Disallow: # same as Allow: /
User-agent: Yandex
Allow: # isn’t taken into account by the bot
```
## Using the special characters * and $ {#asterisk-dollar}
You can use special characters when specifying the paths of the Allow and Disallow directives `*` and `$` to set certain regular expressions.
The `*` character indicates any sequence of characters (or none). Examples:
```no-highlight
User-agent: Yandex
Disallow: /cgi-bin/*.aspx # prohibits “/cgi-bin/example.aspx”
# and “/cgi-bin/private/test.aspx”
Disallow: /*private # prohibits both “/private”
# and “/cgi-bin/private”
```
By default, the `*` character is appended to the end of every rule described in the `robots.txt` file. Example:
```no-highlight
User-agent: Yandex
Disallow: /cgi-bin* # blocks access to pages
# starting with “/cgi-bin”
Disallow: /cgi-bin # the same
```
To cancel `*` at the end of the rule, use the `$` character, for example:
```no-highlight
User-agent: Yandex
Disallow: /example$ # prohibits “/example”,
# but allows “/example.html”
```
```no-highlight
User-agent: Yandex
Disallow: /example # prohibits both “/example”
# and “/example.html”
```
The `$` character doesn't forbid the `*` at the end, that is:
```
User-agent: Yandex
Disallow: /example$ # only prohibits “/example”
Disallow: /example*$ # same as “Disallow: /example”
# prohibits both /example.html and /example
```
## Processing the # character {#hash}
According to the standard, you should insert a blank line before every User-agent directive. The `#` character designates commentary. Everything following this character, up to the first line break, is disregarded.
Pages with addresses like `https://example.com/page#part_1` are not indexed by the search bot and will be crawled at the `https://example.com/page` address. Therefore, it is okay to specify the page address without the anchor in the directive.
If you do not take this feature into account and write a disallow directive with the `#` character, it may block the entire site from indexing. For example, a directive in the `Disallow: /#` format will be interpreted by the search system as `Disallow: /`, i.e., a complete ban on indexing.
## Examples of how directives are interpreted {#interpre}
```no-highlight
User-agent: Yandex
Allow: /
Disallow: /
# everything is allowed
User-agent: Yandex
Allow: /$
Disallow: /
# everything is prohibited except the home page
User-agent: Yandex
Disallow: /private*html
# prohibits “/private*html”,
#, “/private/test.html”, “/private/html/test.aspx”, etc.
User-agent: Yandex
Disallow: /private$
# only prohibits “/private”
User-agent: *
Disallow: /
User-agent: Yandex
Allow: /
# since the Yandex bot
# selects entries that include “User-agent:”,
# everything is allowed
```