Crawler Improvements for Live Search Announced

 Live Search Webmaster Center Blog announced the following updates:

Today we’re pleased to announce several improvements in the crawler for Live Search that should significantly improve the efficiency with which we crawl and index your web sites. We are always looking for ways to help webmasters, and we hope these features take us a few more steps in the right direction.

  • HTTP Compression: HTTP compression allows faster transmission time by compressing static files and application responses, reducing network load between your servers and our crawler…

  • Conditional Get: We support conditional get as defined by RFC 2616 (Section 14.25), generally we will not download the page unless it has changed since the last time we crawled it…

In addition to these two features there are many more improvements in performance that should help further optimize our crawling. As a result, we’ve also upgraded our user agent to reflect the changes, it is now “msnbot/1.1″…..

— Fabrice Canel, Live Search Crawling Team

Full article: Live Search Webmaster Center Blog : Announcing Crawler Improvements for Live Search