Explain robots.txt

varun1182 · #1 (**permalink**) 22nd June 2006, 07:19 AM

I found following line in an article regarding robots.txt
"The robots.txt file should be created in Unix line ender mode! Do not attempt to use an HTML editor that does not specifically have a text mode to create a robots.txt file"

Does this restrict me to use text editors like editplus, editpad to create robots.txt??

athmane · #2 (**permalink**) 22nd June 2006, 11:48 AM

Greetings Var,

The robots.txt file is a a text file so naturally you can use any text editor to create it. I think what the Article was trying to get across is that some HTML editors might not render text, so they shouldn't be used.

If you want a hassle-free robots.txt file created on the fly for all major search engines robots out there, then I recommend using the free Robots.txt Generator .

backstage · #3 (**permalink**) 14th September 2006, 06:40 AM

Be careful with robots!

There are robots that visit your site with bad intentions!
There are many, robots whose sole purpose is to scan your website and extract your email address for spamming purposes!

natasha85 · #4 (**permalink**) 16th September 2006, 04:01 PM

oh, that's scary, how do we prevent those from happening?
anti spam? anti virus?
are those helpful in dealing with those bad robots?

backstage · #5 (**permalink**) 17th September 2006, 08:49 AM

www.robotstxt.org/wc/exclusion-admin.html

This site will help you with that.

fredilan · #6 (**permalink**) 3rd January 2007, 08:45 PM

one way on how to prevent using such malicious robot scripts is by using Robots from credible sources.