What are search engine spiders?

astros99 · #1 (**permalink**) 7th December 2006, 06:12 PM

I've been reading this search engine spider section of the for a while and I can't really figure out exactly what search engine spiders are. I know they have something to do with search engines, and from the forum I know that they "crawl" (figuratively, not literally) , or something like that. But what exactly is a search engine spider?

fredilan · #2 (**permalink**) 3rd January 2007, 08:41 PM

in my knowledge, website spidering/crawling is like looking for some keywords/entries present in a certain website. searching thru google is one example of web spidering/crawling. So google uses Web Spiders to locate some websites with a certain keyword.

mini_0 · #3 (**permalink**) 8th May 2007, 01:56 PM

Search engine spiders are the robots that visit the websites that are indexed ( mainly from a internet directory ) and collects information aboutits pages like keywords etc . This is because manually doing a round of the billion websites is not possible . Search Engines use spiders to index websites. When you submit your website pages to a search engine by completing their required submission page, the search engine spider will index your entire site. A ‘spider’ is an automated program that is run by the search engine system.Spider visits a web site, read the content on the actual site, the site's Meta tags and also follow the links that the site connects.

astros99 · #4 (**permalink**) 3rd June 2007, 03:45 PM

Hmm, I see. So what are some popular search engine spider companies? I'm guessing companies like Google, Yahoo, and MSN, right? Because they have to come up with a list of pagerank, like Google pagerank, so its obvious that they use search engine spiders, correct?

fredilan · #5 (**permalink**) 7th June 2007, 10:05 PM

almost all search engines use spiders to crawl the internet. it's how they expand their search results.