When do crawler come to a site again?

varun1182 · #1 (**permalink**) 22nd June 2006, 06:37 AM

When do crawler/spider comes to any website URL again for crawling? eg. I update my site after 4 months, now when do googlebot or anyother, comes to my site again and store the updations.. another eg if I create robots.txt file and upload it after say 5 months, when will its usability reflect on SEs?? when do SE read my robots.txt etc..

Basicall I want to know when do SE comes back to my site again?
Thanks.

athmane · #2 (**permalink**) 22nd June 2006, 12:33 PM

Do you have a log file analysis package installed on your website? If you have a basic hosting account, then you should have a free package like Awstats installed by default.
These statistic analysis packages will tell you which search engine robots visited your site, when they did it and how many pages they crawled. It's all displayed in graphical, easy-to-understand format.

backstage · #3 (**permalink**) 14th September 2006, 06:33 AM

little trick for Yahoo

This is done by paying Yahoo's fees and submitting your site. This guarantees you will be added to the index but is no guarantee of your website's ranking in the SERPs.
The URLs are submitted into Yahoo’s index and are then crawled by SLURP every 48 hours.