RSS FeedWho’s Crawling My Site?
With all of the talk about bots and spiders, don’t you ever wonder who’s crawling your site and how often? The answers are in your website’s visitor log reports, which you can easily access from your site’s control panel.
Your log file reports the search engine bots that have visited your site and when they visited, but you might not recognize the search engines’ visits, because their “user-agent” names are often different from their name.
Below are the agent names of the major search engines:
Ask User-agent: Teoma
Google User-agent: Googlebot
Google Mobile User-agent: Googlebot-Mobile
Google Images User-agent: Googlebot-Image
Google Adsense User-agent: Mediapartners-Google
LookSmart User-agent: ZyBorg
Microsoft/MSN User-agent: MSNBot
Yahoo User-agent: Slurp
If you have a lot of search engine bot visitors, this is good, and it probably results from keeping your website updated on a regular basis, adding new links to internal pages, and obtaining new inbound links or backlinks.
If it appears your site is not being crawled on a regular and frequent basis, then you may want to consult your system administrator or SEOP consultant.
Popularity: 93% [?]
admin @ March 27, 2008