Supporting wildcards in robots.txt

[hassen1]

Product Manager, Yahoo! Search, says:
I was going through my notes from Danny Sullivan's Open Feedback sessions that occur during the ‘Meet the Crawlers’ panel at Search Engine Strategies. One of the items on my list was a request for enhanced syntax in robots.txt to make it easier for webmasters to manage how search crawlers, including Slurp, access your content.

For those who may not be as familiar with search index terminology, webmasters use the robots.txt file to direct robots that visit their site, including search engine crawlers, which files should be crawled and which shouldn't be. You can read about our support for robots directives in the help for Yahoo! Slurp.

Well, we can scratch that one off the list, since we have just updated Yahoo! Slurp to recognize two additional symbols in the robots.txt directives – '*' and '$'. The semantics of these is what is as widely understood for robots.txt files.

[hassen1]

For those who need more information about robots.txt, please visit http://www.robotstxt.org/wc/robots.html