Design - Robots.TXT File
How to setup a 'robots.txt' file.
A robots.txt file is a simple text file in your home directory. Before search engines spider your site, they look for this file to see which files and directories they are not allowed to spider.
The file doesn't tell them what they should spider, but what they should not spider. In this case, if you are happy for the search engines to spider all of your site, there is no need to create a robots.txt file.
To create a robots.txt file, use a "text" editor or your "html" editor however, you must save the file as robots.txt with the "txt" extension.
This file should contain coding such as ...
User-agent: *
Disallow: /cgi-bin/
The above would block all spiders from indexing the cgi-bin directory.
User-agent: googlebot
Disallow:
User-agent: *
Disallow: /admin.php
Disallow: /cgi-bin/
Disallow: /admin/
Disallow: /stats/
In the above example googlebot can index everything while all other spiders can not index admin.php, cgi-bin, admin and the stats directory. Notice that you can block single files like admin.php.
More advanced settings can be made to block known browsers that crawl your site to collect information like e-mail addresses. For an example of these, view our own robots.txt file here:
» http://www.saffas-web.co.uk/robots.txt
|
