Guide to setup robots txt and Meta robots tags

SEO Professionals knows the importance of robots txt and Meta tags for a website. These points comes under the Off Page optimization such as content optimization, links, tags, meta tags, image optimization and website structure.

Robots.txt


Robots txt file is used to give instruction search engines, crawlers and spiders how to index a web page and website pages. In General it kept in the top directory of your website.
Some pages you should hide using a robots.txt file:

1. Account pages
2. Chats
3. Thank-you pages
4. Pages with duplicate content
5. Pagination pages
6. Admin pages
7. Shopping cart
8. Dynamic product and service pages

Meta Robots Tags

Meta robots tags are simple to use.

Meta Robots Tags Parameters

1. index, follow: allow search bots to index a page and follow its links
2. noindex, nofollow: prevent search bots from indexing a page and following its links
3. index, nofollow: allow search engines to index a page but hide its links from search spiders
4. noindex, follow: exclude a page from search but allow following its

 Source:https://www.searchenginejournal.com/best-practices-setting-meta-robots-tags-robots-txt/188655/