Robots.txt This is the traffic-cop of search engine bots and instructs those bots which parts of a website they should and should not follow. This file is mainly used to give instructions to search engine crawlers to allow or disallow some part of website for crawling; this is also called as The Robots Exclusive Protocol. Whenever a search engine bot visits a website it checks for robots.txt file and find instruction given in this file. So if it finds instructions not to crawl website it goes on.
Sytanx for Robots.Txt file:
User-agent: *
Disallow: /
Creating the file is simple- Simply open any text editor like Notepad and copy and paste the following:
User-agent: *
Disallow:
Then save the file as “robots.txt” and upload this file into the root directory of website.
The asterisk is a wildcard denoting all search engine robots and leaving the Disallow command blank allows them access to any parts of your site. If there are any areas of your website that for whatever reason you do not want the search engine spiders to visit simply place the file or folder name in the Disallow command like so:
User-agent: *
Disallow: /example folder/
Steps for Creating a robots.txt file
- Open any text editor (notepad)
- Write your robots instruction
- Save file as "robots.txt"
- upload robots.txt file to root of your website
No comments:
Post a Comment