Search engine spider identification

[hassen1]

The following is a basic listing of search engine spider names and their "owners". This is by no means complete, as there are many thousands of search engines on the Internet, but it covers the more common beneficial spiders. Look for these in your traffic reports or search for the names through your server logs to discover which pages they have been spidering. You'll find that many of the entries will also have accompanying numbers or letters e.g Googlebot/2.1 or Slurp.so/1.0

Spider name & Spider owner
Googlebot = Google.com
MSNbot = Search.msn.com
TeomaAgent = Teoma.com
Zyborg = Wisenut.com
Gulliver = NorthernLight.com
Architext spider Excite.com
FAST-WebCrawler FAST (AllTheWeb.com)
Slurp Inktomi.com
Yahoo Slurp Yahoo Web Search
Ask Jeeves AskJeeves.com
ia_archiver Alexa.com
Scooter AltaVista.com
Mercator AltaVista.com
crawler@fast FAST (AllTheWeb.com)
Crawler Crawler.de
InfoSeek sidewinder InfoSeek.com
Lycos_Spider_(T-Rex) Lycos.com
Fluffy the Spider SearchHippo.com
Ultraseek InfoSeek.com
MantraAgent LookSmart.com
Moget Goo.jp
T-H-U-N-D-E-R-S-T-O-N-E Thunderstone.com
MuscatFerret Euroferret.com
VoilaBot Voila.fr
Sleek Spider Search-info.com
KIT_Fireball FireBall.de
WebCrawler Webcrawler.com

If you have spotted any significant activity from these spiders in your reports or logs, there's a good chance that you'll be listed on that particular search engine. But you'll need to be patient; some Search Engines take up to 6 months to refresh their databases!