Robot.txt

[sourabhweb]

The method used to exclude robots from a server is to create a file on the server which specifies an access for robots. This file must be accessible via HTTP .

This method works fine because it can be easily implemented on any WWW server.

Points to be remembered:

The filename should fit in file naming restrictions of all common operating systems.
The filename extension should not require extra server configuration.
The filename should indicate the purpose of the file and be easy to remember.
The likelihood of a clash with existing files should be minimal.


The Format:The format and semantics of the "/robots.txt" file are as follows:
The file consists of one or more records separated by one or more blank lines (terminated by CR,CR/NL, or NL). Each record contains lines of the form "<field>:<optionalspace><value><optionalspace> ". The field name is case insensitive.

Comments can be included in file using UNIX bourne shell conventions: the '#' character is used to indicate that preceding space (if any) and the remainder of the line up to the line termination is discarded. Lines containing only a comment are discarded completely, and therefore do not indicate a record boundary.

[backstage]

Another good thing about the robots.txt file is that enables you to exclude specific robots, so you can inhibit the Googlebot but enable SLURP to crawl a particular page.
This can be useful if you have optimized different pages for separate search engines. This may occur in order to give you flexibility, but a search engine may think you have duplicate pages and may penalize you. Follow these instruction to use the robots.txt file.
You open notepad and type in the following lines:

User-Agent: Slurp
Disallow: whatsisname.html
Disallow: page_optimized_for_google.html
Disallow: credit_card_list.html
Disallow: whatnot.html

Save it as robots.txt and upload it into your root directory. You can disallow as many pages for each crawler robot as you want, but to disallow certain pages for another crawler, you start a new line of code.

User-Agent: Slurp
Disallow: whatsisname.html
Disallow: page_optimized_for_google.html
Disallow: credit_card_list.html
Disallow: whatnot.html
User-Agent: Googlebot
Disallow: page_optimized_for_yahoo.html
Disallow: credit_card_list.html
Disallow: whatnot.html

If you want to disallow all crawlers, you replace the name of the user agent with the wildcard command (*)

Robots.txt is useful for not getting banned on search engines and can also be used to pinpoint crawlers when they come . Only crawlers request Robots.txt, and these requests show up on the server logs.

[sourabhweb]

Many Many thanks backstage for providing such a fruitful conclusion of my introductory discussion.

I hope this will be a helpful discussion for new webmarketers up to a certain extend.

Really nice post.

Keep it up.

Cheers,
Sourabh.

[humpty]

wow, that's really good stuff, thanks a lot for the info,

keep em coming, appreciate you guys a lot!
before this, i don't really have a clue what a robot.txt thing is all about heheheh

[sourabhweb]

It is always satisfactory when somebody learns from you.

I am happy to help you a little bit in my own way, Humpty.

Feel free to ask any question.

Cheers!!!
Sourabh.

[maggots]

yeah man, that's the spirit, love it, keep it up sourabhweb

[mini_0]

There is always a continuous speculation among new , unexperienced webmasters on whether or not they need a robots.txt file ? Also maily people ignore it when they have a smaller website . The main use of a robots.txt file is to give robots instructions to what they can crawl and what they should not crawl. This gives you a little more control over the robots. And since this gives you a little more control over the robots, which means you can issue indexing instructions to specific search engines.The robots.txt file is a simple text file, which can be created in Notepad. It needs to be saved to the root directory of your site-that is the directory where your home page or index page is located.

[wzf851005]

These patterns are based on different foot type, weight, Speed, training programmes, gender and skill level design. These different styles, different prices and multi-purpose products, attracting hundreds of thousands of jogging, making them feel that nike shoes is to provide the most complete variety of running shoes manufacturers, millions of range of ability Running to have that belief. First-generation jordan shoes uppers, there is a conspicuous "Chachi basketball" signs in the use of trapeze signs ago, the first generation and second generation Chachi Jordan basketball shoes are used as signs. First-generationjordan shoes appeared in 1985, 1994 and 2001 NIKE companies produced twice again this shoes. For the classic colors with a red, black, and compared with black. Air Jordans, published in 1991, jordan shoes91-92 season Zhanxue because jordan shoesin the Barcelona Olympic Games 92 wearing this pair of 7 and winning the title, the seventh generation in jordan shoesin the series, it is particularly valuable. 7 and the shape of the shadow of some six generations, the biggest feature is particularly color. Borrow a relatives chinese dress. Sometimes those old chinese dresses are outdated beyond repair, but then again, sometimes theyre not. If you have an (aunt, mom, cousin, sister) who got married in a strapless sheath that would be perfect, theres no reason you couldnt too, assuming theyre open to loaning it out. You can still make the look your own with the veil, jewelry, shoes, a pretty sash. And youll have that “something borrowed” checked off the list.What about the modern cheongsam? With more exchange with the outside world, todays cheongsam combines both Chinese and western characteristics, traditional and modern features. There are bold changes and innovations. Whatever it is, cheongsam on one hand can still create an impression of simple and quite charm, elegance and neatness. On the other hand, blended with modern features, they can also show peoples individuality and distinctiveness. No wonder cheongsam enjoys a growing popularity in the international world of high fashion.

[weiwei]

''There are more than 30 wow gold machines in busy stations while smaller stations have 5 or 6. In addition wow power leveling to the automatic system, we also have personnel serving wow power leveling at ticket outlets. Through wow power leveling this combination, the ticket-selling system can meet the wow power leveling demand of subway passengers in Beijing.''.weiwei1978123

[weiwei]

But a spirit prevailed then which wow gold was quintessentially are a challenge，not a alibi , that men are measured wow power leveling not only wow power leveling by their success wow powerleveling but also World of Warcraft gold by their striving; that it is better to aim grandly than to wallow in mediocre comfort. weiwei1978123