Home Resources Forum Advertise Contact

Go Back   Webmaster Forums > Search Engines > Search Engine Spiders

Search Engine Spiders Exchange ideas and opinions about search engine spiders here. Discuss about identifying search engine spiders' spidering habits and make it to your advantage here.

Reply
 
LinkBack Thread Tools
  #1 (permalink)  
Old 7th December 2006, 06:12 PM
astros99 astros99 is offline
WD Addict Poster
 
Join Date: 1st November 2006
Posts: 600
Default What are search engine spiders?

I've been reading this search engine spider section of the for a while and I can't really figure out exactly what search engine spiders are. I know they have something to do with search engines, and from the forum I know that they "crawl" (figuratively, not literally) , or something like that. But what exactly is a search engine spider?
Reply With Quote
  #2 (permalink)  
Old 3rd January 2007, 08:41 PM
fredilan fredilan is offline
Super Moderator
 
Join Date: 18th December 2006
Posts: 49
Default

in my knowledge, website spidering/crawling is like looking for some keywords/entries present in a certain website. searching thru google is one example of web spidering/crawling. So google uses Web Spiders to locate some websites with a certain keyword.
Reply With Quote
  #3 (permalink)  
Old 8th May 2007, 01:56 PM
mini_0's Avatar
mini_0 mini_0 is offline
WD Addict Poster
 
Join Date: 10th March 2007
Posts: 1,799
Default

Search engine spiders are the robots that visit the websites that are indexed ( mainly from a internet directory ) and collects information aboutits pages like keywords etc . This is because manually doing a round of the billion websites is not possible . Search Engines use spiders to index websites. When you submit your website pages to a search engine by completing their required submission page, the search engine spider will index your entire site. A ‘spider’ is an automated program that is run by the search engine system.Spider visits a web site, read the content on the actual site, the site's Meta tags and also follow the links that the site connects.
Reply With Quote
  #4 (permalink)  
Old 3rd June 2007, 03:45 PM
astros99 astros99 is offline
WD Addict Poster
 
Join Date: 1st November 2006
Posts: 600
Default

Hmm, I see. So what are some popular search engine spider companies? I'm guessing companies like Google, Yahoo, and MSN, right? Because they have to come up with a list of pagerank, like Google pagerank, so its obvious that they use search engine spiders, correct?
Reply With Quote
  #5 (permalink)  
Old 7th June 2007, 10:05 PM
fredilan fredilan is offline
Super Moderator
 
Join Date: 18th December 2006
Posts: 49
Default

almost all search engines use spiders to crawl the internet. it's how they expand their search results.
__________________
http://www.lhanslair.com
Reply With Quote
Reply



Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads
Thread Thread Starter Forum Replies Last Post
Invite search engine spiders into your dynamic web site arpan911 Search Engine Spiders 5 18th July 2008 06:13 AM
How to set up a robots.txt to control search engine spiders arpan911 Search Engine Spiders 3 20th June 2008 06:56 PM
How do i use search engine? rowenahere SEO Talk 2 13th May 2007 10:51 AM
Search engine spiders list arpan911 Search Engine Spiders 2 19th September 2006 05:55 AM
Search engine spiders: how do thet index a webpage? sourabhweb Search Engine Spiders 0 13th July 2006 01:31 AM


All times are GMT -4. The time now is 08:01 PM.


Powered by vBulletin
Copyright ©2000 - 2008, Jelsoft Enterprises Ltd.
Search Engine Optimization by vBSEO 3.0.0 RC6
vB Ad Management by =RedTyger=