create a robots.txt file

[automgt]

I keep getting 404 error code on my web site. 95% of the codes is due to robots.txt not being found on my site. Trying to find out how to edit or make a robots.txt file and where to place within my web site. Any help would be greatly appricated.

Hank

[MandyJay]

Hi Hank,
the robots.txt should be in the root folder i.e. residing alongside your home page.

Here is a link that tells you all about the robots.txt file and what you can or can't do and how to.

http://www.robotstxt.org/orig.html

and here is a google link which also explains that if you do exclude pages etc they still maybe found ! i.e. Dont rely on the robots.txt to exclude files, becuase there is no guarantee.

http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

Hope this helps

MJ