Roles of Robot.txt file in SEO

Robot.txt file plays important role in search engine ranking of the sites.If you have a website and you wants a good ranking in search engines then your sites must have Robot.txt file. When a search engine spider visit your site then will look for a special file called robot.txt file. Robot.txt file is the file which tells
the search engine spider which pages of the site should be indexed and which do not.You can create the robot.txt file in text documents.It is the simple text file and then place it to the root directory of your site. For example.

http://www.mysitename.com/robot.txt

The content of the robot.txt file called "records". The records contains the all the information for the search engine spider. The records consists of user-agent lines and one or more disallow lines.For example.

User-agent: Googlebot
Disallow: /cgi-bin

In user agent line we write the name of the search engine spider. In above example its googlebot, which is the name of google search engine spider. In disallow line we write the file which should not be indexed.So this robot.txt file allow google spider to index the all pages of the site expect "cgi-bin" directory.Lets take another example.

user-agent: googlebot
Disallow: /support

This file will not allow the googlebot to index /support.htm as well as all the other files in support directory.If you want to allow the all the search engine spiders to crawl your site, then following will be your content

user-agent: *
Disallow: /cgi-bin

 If you want that all search engine spiders will index your all the files in site, then following will be the content

user-agent: *
Disallow:

And if you don't want that search engine spiders will index your all the files in site,it will be

user-agent: *
Disallow: /

There is no "Allow" command.

If you have more than one files which you don't want to indexed by spiders then following will be the content

user-agent: *
Disallow: /support
Disallow: /images
Disallow: /contact us

You can make Robot.txt file with google robot.txt file generator tool, i had created robot.txt file with the help of tool ,you can check this as

http://www.dineshthakursem.blogspot.com/robots.txt

You an see the google webmaster tool to create robot.txt video and uploading robot.txt file video to have a more clear idea.

Technorati code  ZZXXABEU54CR.