What happens when you change the look, or some other aspect, of a site?  Does it increase traffic?  Are you getting results?  A new team at Windows Live, The Microsoft Experimentation Platform,  is developing an experimentation platform to make it easy to measure results based on changes, by creating a control group and a treatment group, and analyzing the differences.

Mary Jo Foley talked to General Manager Ronny Kohavi, who came to Microsoft 3 years ago from Amazon.com, about EXP.  In her email interview with him, he says:

This was not a top-down project, but a bottom-up proposal I made. A few executives, led by (Corporate Vice President of Live Platform Services) David Treadwell, saw the potential and supported the project early on. (Chief Software Architect) Ray Ozzie later said: “We have an unprecedented opportunity to run A/B tests with online users and innovate more quickly based on actual user response. Microsoft needs to shift the culture from planning the exact features to planning a set of possible features, and letting customers guide us.”

On the EXP website (built on Office Live, and nicely done) there are links to a number of talks given about the project.  Here’s an example of an experiment which compared a lesser contrast and higher contrast Live Search page:

The higher contrast site increased the queries per user by .9%, and increased ad clicks per user by 3.1%.  So the platform can not only track changes in site design etc, but also experiment with subtle differences and fine tune a site’s performance.

Very interesting stuff, we’re looking forward to hearing more.