Another project from the Ray Ozzie team: EXP – Microsoft’s Experimentation Platform

Kip Kniskern — Thu, 17 Apr 2008 02:02:19 GMT

What happens when you change the look, or some other aspect, of a site? Does it increase traffic? Are you getting results? A new team at Windows Live, The Microsoft Experimentation Platform, is developing an experimentation platform to make it easy to measure results based on changes, by creating a control group and a treatment group, and analyzing the differences.

Mary Jo Foley talked to General Manager Ronny Kohavi, who came to Microsoft 3 years ago from Amazon.com, about EXP. In her email interview with him, he says:

This was not a top-down project, but a bottom-up proposal I made. A few executives, led by (Corporate Vice President of Live Platform Services) David Treadwell, saw the potential and supported the project early on. (Chief Software Architect) Ray Ozzie later said: “We have an unprecedented opportunity to run A/B tests with online users and innovate more quickly based on actual user response. Microsoft needs to shift the culture from planning the exact features to planning a set of possible features, and letting customers guide us.”

On the EXP website (built on Office Live, and nicely done) there are links to a number of talks given about the project. Here’s an example of an experiment which compared a lesser contrast and higher contrast Live Search page:

The higher contrast site increased the queries per user by .9%, and increased ad clicks per user by 3.1%. So the platform can not only track changes in site design etc, but also experiment with subtle differences and fine tune a site’s performance.

Very interesting stuff, we’re looking forward to hearing more.

LiveSide - News blog : EXP

Another project from the Ray Ozzie team: EXP – Microsoft’s Experimentation Platform