Для тех кто хочет немного теории - есть отдельная статья на тему "Правильная настройка индексации интернет-магазина на Shop Script". Ссылка на статью.
Инструкция по внедрению robots.txt на основную витрину сайта
- Скопируйте себе весь код для основной витрины сайта.
- Если вы используете страницы тегов в качестве посадочных страниц - удалите строчки "Disallow: */tag/" (3 раза).
- Если вы используете страницы отзывов о товарах в качестве посадочных страниц - удалите строчки "Disallow: */reviews/" (3 раза).
- В строчке "Sitemap: https://site.ru/sitemap.xml" укажите домен своего сайта вместо "site.ru" (3 раза).
- Если вам необходимо добавить дополнительные директивы - рекомендую вывести их в блоке "## individual rules ##" (3 раза).
- Убедитесь, что вы внесли изменения 3 раза для каждого User-agent-а: для Yandex, для Google, и для всех остальных роботов (User-agent: *)
- Введите итоговый код в настройках («админка» --> «Сайт» --> «*выбрать основную витрину из списка*» --> «Настройки сайта» --> «поле "robots.txt"»
- Обязательно проверьте файл robots.txt на основной витрине после внесения изменений (доступен по адресу https://site.ru/robots.txt).
Шаблон robots.txt для основной витрины сайта
- #upd 10.08.2023
- #author: @anatoly_chikurov
- User-agent: Yandex
- ## googleonly usual storefronts ##
- Disallow: */msk/ #1
- Disallow: */moskva/
- Disallow: */spb/ #2
- Disallow: */novosibirsk/ #3
- Disallow: */nsk/
- Disallow: */ekb/ #4
- Disallow: */ekaterinburg/
- Disallow: */eburg/
- Disallow: */kazan/ #5
- Disallow: */nn/ #6
- Disallow: */chelyabinsk/ #7
- Disallow: */cheljabinsk/
- Disallow: */krasnoyarsk/ #8
- Disallow: */krasnojarsk/
- Disallow: */samara/ #9
- Disallow: */ufa/ #10
- Disallow: */rostov/ #11
- Disallow: */omsk/ #12
- Disallow: */krasnodar/ #13
- Disallow: */voronezh/ #14
- Disallow: */perm/ #15
- Disallow: */volgograd/ #16
- Disallow: */saratov/ #17
- Disallow: */tyumen/ #18
- Disallow: */tjumen/
- Disallow: */tolyatti/ #19
- Disallow: */toljatti/
- Disallow: */barnaul/ #20
- Disallow: */izhevsk/ #21
- Disallow: */mahachkala/ #22
- Disallow: */habarovsk/ #23
- Disallow: */ulyanovsk/ #24
- Disallow: */uljanovsk/
- Disallow: */irkutsk/ #25**************
- Disallow: */vladivostok/ #26
- Disallow: */yaroslavl/ #27
- Disallow: */jaroslavl/
- Disallow: */kemerovo/ #28
- Disallow: */tomsk/ #29
- Disallow: */naberezhnye-chelny/ #30
- Disallow: */sevastopol/ #31
- Disallow: */stavropol/ #32
- Disallow: */orenburg/ #33
- Disallow: */stavropol/ #34
- Disallow: */novokuzneck/ #35
- Disallow: */ryazan/ #36
- Disallow: */rjazan/
- Disallow: */balashiha/ #37
- Disallow: */penza/ #38
- Disallow: */cheboksary/ #39
- Disallow: */lipeck/ #40
- Disallow: */kaliningrad/ #41
- Disallow: */astrahan/ #42
- Disallow: */tula/ #43
- Disallow: */kirov/ #44
- Disallow: */sochi/ #45
- Disallow: */kursk/ #46
- Disallow: */ulan-udeh/ #47
- Disallow: */tver/ #48
- Disallow: */magnitogorsk/ #49
- Disallow: */surgut/ #50
- Disallow: */bryansk/ #51
- Disallow: */brjansk/
- Disallow: */ivanovo/ #52
- Disallow: */yakutsk/ #53
- Disallow: */jakutsk/
- Disallow: */vladimir/ #54
- Disallow: */simferopol/ #55
- Disallow: */belgorod/ #56
- Disallow: */nizhnij-tagil/ #57
- Disallow: */kaluga/ #58
- Disallow: */chita/ #59
- Disallow: */groznyj/ #60
- Disallow: */volzhskij/ #61
- Disallow: */smolensk/ #62
- Disallow: */podolsk/ #63
- Disallow: */saransk/ #64
- Disallow: */vologda/ #65
- Disallow: */kurgan/ #66
- Disallow: */cherepovec/ #67
- Disallow: */orel/ #68
- Disallow: */oryol/
- Disallow: */arhangelsk/ #69
- Disallow: */vladikavkaz/ #70
- Disallow: */nizhnevartovsk/ #71
- Disallow: */yoshkar-ola/ #72**********
- Disallow: */sterlitamak/ #73
- Disallow: */murmansk/ #74
- Disallow: */kostroma/ #75
- Disallow: */novorossijsk/ #76
- Disallow: */tambov/ #77
- Disallow: */himki/ #78
- Disallow: */mytischi/ #79********
- Disallow: */nalchik/ #80********
- Disallow: */taganrog/ #81
- ## googleonly opt storefronts ##
- Disallow: */msk-opt/ #1
- Disallow: */moskva-opt/
- Disallow: */spb-opt/ #2
- Disallow: */novosibirsk-opt/ #3
- Disallow: */nsk-opt/
- Disallow: */ekb-opt/ #4
- Disallow: */ekaterinburg-opt/
- Disallow: */eburg-opt/
- Disallow: */kazan-opt/ #5
- Disallow: */nn-opt/ #6
- Disallow: */chelyabinsk-opt/ #7
- Disallow: */cheljabinsk-opt/
- Disallow: */krasnoyarsk-opt/ #8
- Disallow: */krasnojarsk-opt/
- Disallow: */samara-opt/ #9
- Disallow: */ufa-opt/ #10
- Disallow: */rostov-opt/ #11
- Disallow: */omsk-opt/ #12
- Disallow: */krasnodar-opt/ #13
- Disallow: */voronezh-opt/ #14
- Disallow: */perm-opt/ #15
- Disallow: */volgograd-opt/ #16
- Disallow: */saratov-opt/ #17
- Disallow: */tyumen-opt/ #18
- Disallow: */tjumen-opt/
- Disallow: */sochi-opt/ #44
- Disallow: */ivanovo-opt/ #51
- ## system hidden pages ##
- Disallow: *admin/
- Disallow: *helpdesk/
- Disallow: *cml1c/
- Disallow: */vk/
- Disallow: */turbo/
- Disallow: */logs/
- Disallow: *multiform/
- Disallow: *webasyst/
- Disallow: */test/
- Disallow: */old/
- Disallow: *.html
- Disallow: *.php
- Disallow: *.pdf
- Disallow: */amp/
- ## info and service pages ##
- Disallow: *search
- Disallow: *mailer/
- Disallow: *login/
- Disallow: *signup/
- Disallow: *forgotpassword/
- Disallow: *order/
- Disallow: */cart/
- Disallow: *onestep/
- Disallow: *checkout/
- Disallow: */my/
- ## service links ##
- Disallow: *tel:
- Disallow: *skype:
- Disallow: *mailto:
- Disallow: *tg:
- Disallow: *viber:
- ## garbage and dublicates ##
- Disallow: *undefined
- Disallow: *compare/
- Disallow: *?
- Allow: *.*?
- Allow: *config/?
- Allow: *?page=
- Disallow: *?page=*&
- Disallow: *&page=
- Disallow: *?page=1
- Allow: *?page=10
- Allow: *?page=11
- Allow: *?page=12
- Allow: *?page=13
- Allow: *?page=14
- Allow: *?page=15
- Allow: *?page=16
- Allow: *?page=17
- Allow: *?page=18
- Allow: *?page=19
- Disallow: *?page=10*&
- Disallow: *?page=11*&
- Disallow: *?page=12*&
- Disallow: *?page=13*&
- Disallow: *?page=14*&
- Disallow: *?page=15*&
- Disallow: *?page=16*&
- Disallow: *?page=17*&
- Disallow: *?page=18*&
- Disallow: *?page=19*&
- Disallow: *product*/?page=
- ## blog garbage ##
- Disallow: */author/
- Disallow: */2015/
- Disallow: */2016/
- Disallow: */2017/
- Disallow: */2018/
- Disallow: */2019/
- Disallow: */2020/
- Disallow: */2021/
- Disallow: */2022/
- Disallow: */2023/
- Disallow: */2024/
- Disallow: */2025/
- Disallow: */2026/
- Disallow: */2027/
- Disallow: */2028/
- Disallow: */2029/
- Disallow: */2030/
- Disallow: */2031/
- Disallow: */2032/
- Disallow: */2033/
- Disallow: */2034/
- Disallow: */2035/
- Allow: /product/20*/
- Disallow: /product/20*/?
- ## individual rules ##
- Disallow: */tag/
- Disallow: */reviews/
- ## end ##
- Clean-param: from&clid&pclid&promo&utm&utm_referrer&utm_medium&utm_source&utm_campaign&utm_term&utm_content&gclid&openstat&cpc&test&dev&sort&order&tip&tsvet&brend&brand&strana&price_min&price_max&showall&frommarket&nomobile&mobile&crcy&promo_id
- Sitemap: https://site.ru/sitemap.xml
- User-agent: Googlebot
- ## system hidden pages ##
- Disallow: *admin/
- Disallow: *helpdesk/
- Disallow: *cml1c/
- Disallow: */vk/
- Disallow: */turbo/
- Disallow: */logs/
- Disallow: *multiform/
- Disallow: *webasyst/
- Disallow: */test/
- Disallow: */old/
- Disallow: *.html
- Disallow: *.php
- Disallow: *.pdf
- ## info and service pages ##
- Disallow: *search
- Disallow: *mailer/
- Disallow: *login/
- Disallow: *signup/
- Disallow: *forgotpassword/
- Disallow: *order/
- Disallow: */cart/
- Disallow: *onestep/
- Disallow: *checkout/
- Disallow: */my/
- ## service links ##
- Disallow: *tel:
- Disallow: *skype:
- Disallow: *mailto:
- Disallow: *tg:
- Disallow: *viber:
- ## garbage and dublicates ##
- Disallow: *undefined
- Disallow: *compare/
- Disallow: *?
- Allow: *.*?
- Allow: *config/?
- Allow: *?page=
- Disallow: *?page=*&
- Disallow: *&page=
- Disallow: *?page=1
- Allow: *?page=10
- Allow: *?page=11
- Allow: *?page=12
- Allow: *?page=13
- Allow: *?page=14
- Allow: *?page=15
- Allow: *?page=16
- Allow: *?page=17
- Allow: *?page=18
- Allow: *?page=19
- Disallow: *?page=10*&
- Disallow: *?page=11*&
- Disallow: *?page=12*&
- Disallow: *?page=13*&
- Disallow: *?page=14*&
- Disallow: *?page=15*&
- Disallow: *?page=16*&
- Disallow: *?page=17*&
- Disallow: *?page=18*&
- Disallow: *?page=19*&
- Disallow: *product*/?page=
- Allow: *utm_source=
- ## blog garbage ##
- Disallow: */author/
- Disallow: */2015/
- Disallow: */2016/
- Disallow: */2017/
- Disallow: */2018/
- Disallow: */2019/
- Disallow: */2020/
- Disallow: */2021/
- Disallow: */2022/
- Disallow: */2023/
- Disallow: */2024/
- Disallow: */2025/
- Disallow: */2026/
- Disallow: */2027/
- Disallow: */2028/
- Disallow: */2029/
- Disallow: */2030/
- Disallow: */2031/
- Disallow: */2032/
- Disallow: */2033/
- Disallow: */2034/
- Disallow: */2035/
- Allow: /product/20*/
- Disallow: /product/20*/?
- ## individual rules ##
- Disallow: */tag/
- Disallow: */reviews/
- ## end ##
- Sitemap: https://site.ru/sitemap.xml
- User-agent: *
- ## googleonly usual storefronts ##
- Disallow: */msk/ #1
- Disallow: */moskva/
- Disallow: */spb/ #2
- Disallow: */novosibirsk/ #3
- Disallow: */nsk/
- Disallow: */ekb/ #4
- Disallow: */ekaterinburg/
- Disallow: */eburg/
- Disallow: */kazan/ #5
- Disallow: */nn/ #6
- Disallow: */chelyabinsk/ #7
- Disallow: */cheljabinsk/
- Disallow: */krasnoyarsk/ #8
- Disallow: */krasnojarsk/
- Disallow: */samara/ #9
- Disallow: */ufa/ #10
- Disallow: */rostov/ #11
- Disallow: */omsk/ #12
- Disallow: */krasnodar/ #13
- Disallow: */voronezh/ #14
- Disallow: */perm/ #15
- Disallow: */volgograd/ #16
- Disallow: */saratov/ #17
- Disallow: */tyumen/ #18
- Disallow: */tjumen/
- Disallow: */tolyatti/ #19
- Disallow: */toljatti/
- Disallow: */barnaul/ #20
- Disallow: */izhevsk/ #21
- Disallow: */mahachkala/ #22
- Disallow: */habarovsk/ #23
- Disallow: */ulyanovsk/ #24
- Disallow: */uljanovsk/
- Disallow: */irkutsk/ #25**************
- Disallow: */vladivostok/ #26
- Disallow: */yaroslavl/ #27
- Disallow: */jaroslavl/
- Disallow: */kemerovo/ #28
- Disallow: */tomsk/ #29
- Disallow: */naberezhnye-chelny/ #30
- Disallow: */sevastopol/ #31
- Disallow: */stavropol/ #32
- Disallow: */orenburg/ #33
- Disallow: */stavropol/ #34
- Disallow: */novokuzneck/ #35
- Disallow: */ryazan/ #36
- Disallow: */rjazan/
- Disallow: */balashiha/ #37
- Disallow: */penza/ #38
- Disallow: */cheboksary/ #39
- Disallow: */lipeck/ #40
- Disallow: */kaliningrad/ #41
- Disallow: */astrahan/ #42
- Disallow: */tula/ #43
- Disallow: */kirov/ #44
- Disallow: */sochi/ #45
- Disallow: */kursk/ #46
- Disallow: */ulan-udeh/ #47
- Disallow: */tver/ #48
- Disallow: */magnitogorsk/ #49
- Disallow: */surgut/ #50
- Disallow: */bryansk/ #51
- Disallow: */brjansk/
- Disallow: */ivanovo/ #52
- Disallow: */yakutsk/ #53
- Disallow: */jakutsk/
- Disallow: */vladimir/ #54
- Disallow: */simferopol/ #55
- Disallow: */belgorod/ #56
- Disallow: */nizhnij-tagil/ #57
- Disallow: */kaluga/ #58
- Disallow: */chita/ #59
- Disallow: */groznyj/ #60
- Disallow: */volzhskij/ #61
- Disallow: */smolensk/ #62
- Disallow: */podolsk/ #63
- Disallow: */saransk/ #64
- Disallow: */vologda/ #65
- Disallow: */kurgan/ #66
- Disallow: */cherepovec/ #67
- Disallow: */orel/ #68
- Disallow: */oryol/
- Disallow: */arhangelsk/ #69
- Disallow: */vladikavkaz/ #70
- Disallow: */nizhnevartovsk/ #71
- Disallow: */yoshkar-ola/ #72**********
- Disallow: */sterlitamak/ #73
- Disallow: */murmansk/ #74
- Disallow: */kostroma/ #75
- Disallow: */novorossijsk/ #76
- Disallow: */tambov/ #77
- Disallow: */himki/ #78
- Disallow: */mytischi/ #79********
- Disallow: */nalchik/ #80********
- Disallow: */taganrog/ #81
- ## googleonly opt storefronts ##
- Disallow: */msk-opt/ #1
- Disallow: */moskva-opt/
- Disallow: */spb-opt/ #2
- Disallow: */novosibirsk-opt/ #3
- Disallow: */nsk-opt/
- Disallow: */ekb-opt/ #4
- Disallow: */ekaterinburg-opt/
- Disallow: */eburg-opt/
- Disallow: */kazan-opt/ #5
- Disallow: */nn-opt/ #6
- Disallow: */chelyabinsk-opt/ #7
- Disallow: */cheljabinsk-opt/
- Disallow: */krasnoyarsk-opt/ #8
- Disallow: */krasnojarsk-opt/
- Disallow: */samara-opt/ #9
- Disallow: */ufa-opt/ #10
- Disallow: */rostov-opt/ #11
- Disallow: */omsk-opt/ #12
- Disallow: */krasnodar-opt/ #13
- Disallow: */voronezh-opt/ #14
- Disallow: */perm-opt/ #15
- Disallow: */volgograd-opt/ #16
- Disallow: */saratov-opt/ #17
- Disallow: */tyumen-opt/ #18
- Disallow: */tjumen-opt/
- Disallow: */sochi-opt/ #44
- Disallow: */ivanovo-opt/ #51
- ## system hidden pages ##
- Disallow: *admin/
- Disallow: *helpdesk/
- Disallow: *cml1c/
- Disallow: */vk/
- Disallow: */turbo/
- Disallow: */logs/
- Disallow: *multiform/
- Disallow: *webasyst/
- Disallow: */test/
- Disallow: */old/
- Disallow: *.html
- Disallow: *.php
- Disallow: *.pdf
- Disallow: */amp/
- ## info and service pages ##
- Disallow: *search
- Disallow: *mailer/
- Disallow: *login/
- Disallow: *signup/
- Disallow: *forgotpassword/
- Disallow: *order/
- Disallow: */cart/
- Disallow: *onestep/
- Disallow: *checkout/
- Disallow: */my/
- ## service links ##
- Disallow: *tel:
- Disallow: *skype:
- Disallow: *mailto:
- Disallow: *tg:
- Disallow: *viber:
- ## garbage and dublicates ##
- Disallow: *undefined
- Disallow: *compare/
- Disallow: *?
- Allow: *.*?
- Allow: *config/?
- Allow: *?page=
- Disallow: *?page=*&
- Disallow: *&page=
- Disallow: *?page=1
- Allow: *?page=10
- Allow: *?page=11
- Allow: *?page=12
- Allow: *?page=13
- Allow: *?page=14
- Allow: *?page=15
- Allow: *?page=16
- Allow: *?page=17
- Allow: *?page=18
- Allow: *?page=19
- Disallow: *?page=10*&
- Disallow: *?page=11*&
- Disallow: *?page=12*&
- Disallow: *?page=13*&
- Disallow: *?page=14*&
- Disallow: *?page=15*&
- Disallow: *?page=16*&
- Disallow: *?page=17*&
- Disallow: *?page=18*&
- Disallow: *?page=19*&
- Disallow: *product*/?page=
- ## blog garbage ##
- Disallow: */author/
- Disallow: */2015/
- Disallow: */2016/
- Disallow: */2017/
- Disallow: */2018/
- Disallow: */2019/
- Disallow: */2020/
- Disallow: */2021/
- Disallow: */2022/
- Disallow: */2023/
- Disallow: */2024/
- Disallow: */2025/
- Disallow: */2026/
- Disallow: */2027/
- Disallow: */2028/
- Disallow: */2029/
- Disallow: */2030/
- Disallow: */2031/
- Disallow: */2032/
- Disallow: */2033/
- Disallow: */2034/
- Disallow: */2035/
- Allow: /product/20*/
- Disallow: /product/20*/?
- ## individual rules ##
- Disallow: */tag/
- Disallow: */reviews/
- ## end ##
- Crawl-delay: 5
- Sitemap: https://site.ru/sitemap.xml
- User-agent: Mediapartners-Google
- Allow: /
- Crawl-delay: 3
Инструкция по внедрению robots.txt на региональные поддомены сайта
- Скопируйте себе весь код для региональных поддоменов.
- Если вы используете страницы тегов в качестве посадочных страниц - удалите строчки "Disallow: /tag/" и "Disallow: */tag/".
- Если вы используете страницы отзывов о товарах в качестве посадочных страниц - удалите строчку "Disallow: */reviews/".
- Если вам необходимо добавить дополнительные директивы - рекомендую вывести их в блоке "## individual rules ##".
- Введите итоговый код:
- Если вы используете плагин "SEO-регионы", то перейдите на страницу "/webasyst/shop/?plugin=regions&action=robots", вставьте код туда, выберите все региональные витрины (т.е. все витрины кроме основной) и нажмите кнопку "Сохранить".
- Если вы не используете плагин "SEO-регионы", то заходите в настройки сайта и поочередно выбирая все региональные витрины, вставляйте код и сохраняйте изменения, не забывая при этом заменять в строчке "Sitemap: https://{storefront_url}/sitemap.xml" фрагмент "{storefront_url}" на домен вашей региональной витрины (пример: "Sitemap: https://spb.test.ru/sitemap.xml").
- Обязательно проверьте файл robots.txt на региональных витринах после внесения изменений (доступны по адресу https://spb.site.ru/robots.txt, https://ufa.site.ru/robots.txt, https://omsk.site.ru/robots.txt, итд).
Шаблон robots.txt для региональных поддоменов
- #upd 10.08.2023
- #author: @anatoly_chikurov
- User-agent: *
- Disallow: /
- User-agent: Yandex
- ## googleonly pages ##
- ## content pages ##
- Disallow: */blog/
- Disallow: */stati/
- Disallow: */novosti
- Disallow: */seorating/
- Disallow: */rating/
- ## system hidden pages ##
- Disallow: *admin/
- Disallow: *helpdesk/
- Disallow: *cml1c/
- Disallow: */vk/
- Disallow: */turbo/
- Disallow: */logs/
- Disallow: *multiform/
- Disallow: *webasyst/
- Disallow: */test/
- Disallow: */old/
- Disallow: *.html
- Disallow: *.php
- Disallow: *.pdf
- Disallow: */amp/
- ## info and service pages ##
- Disallow: *search
- Disallow: *mailer/
- Disallow: *login/
- Disallow: *signup/
- Disallow: *forgotpassword/
- Disallow: *order/
- Disallow: */cart/
- Disallow: *onestep/
- Disallow: *checkout/
- Disallow: */my/
- ## service links ##
- Disallow: *tel:
- Disallow: *skype:
- Disallow: *mailto:
- Disallow: *tg:
- Disallow: *viber:
- ## garbage and dublicates ##
- Disallow: *undefined
- Disallow: *compare/
- Disallow: *?
- Allow: *.*?
- Allow: *config/?
- Allow: *?page=
- Disallow: *?page=*&
- Disallow: *&page=
- Disallow: *?page=1
- Allow: *?page=10
- Allow: *?page=11
- Allow: *?page=12
- Allow: *?page=13
- Allow: *?page=14
- Allow: *?page=15
- Allow: *?page=16
- Allow: *?page=17
- Allow: *?page=18
- Allow: *?page=19
- Disallow: *?page=10*&
- Disallow: *?page=11*&
- Disallow: *?page=12*&
- Disallow: *?page=13*&
- Disallow: *?page=14*&
- Disallow: *?page=15*&
- Disallow: *?page=16*&
- Disallow: *?page=17*&
- Disallow: *?page=18*&
- Disallow: *?page=19*&
- Disallow: *product*/?page=
- ## blog garbage ##
- Disallow: */author/
- Disallow: */2015/
- Disallow: */2016/
- Disallow: */2017/
- Disallow: */2018/
- Disallow: */2019/
- Disallow: */2020/
- Disallow: */2021/
- Disallow: */2022/
- Disallow: */2023/
- Disallow: */2024/
- Disallow: */2025/
- Disallow: */2026/
- Disallow: */2027/
- Disallow: */2028/
- Disallow: */2029/
- Disallow: */2030/
- Disallow: */2031/
- Disallow: */2032/
- Disallow: */2033/
- Disallow: */2034/
- Disallow: */2035/
- Allow: /product/20*/
- Disallow: /product/20*/?
- ## individual rules ##
- Disallow: */tag/
- Disallow: */reviews/
- ## end ##
- Clean-param: from&clid&pclid&promo&utm&utm_referrer&utm_medium&utm_source&utm_campaign&utm_term&utm_content&gclid&openstat&cpc&test&dev&sort&order&tip&tsvet&brend&brand&strana&price_min&price_max&showall&frommarket&nomobile&mobile&crcy&promo_id
- Sitemap: https://{storefront_url}/sitemap.xml