A group of online experimenters at Microsoft has just written a new paper that has the potential to increase the value that A/B and multivariate tests can bring to online businesses.

They have documented methods to combine pre-experiment data with experiment data to yield increased sensitivity for testing the performance of new designs and features. They call their approach CUPED (Controlled-experiment Using Pre-Experiment Data).

This approach will be beyond the capabilities of the typical small website owner using Google Analytics. However, for businesses that employ statistical analysts. CUPED is worth adopting.

By leveraging pre-experiment data, analysts can realize increased sensitivity to

• test effects and interactions that could normally not be detected due to  lack of traffic, such as for high value user segments.
• run more simultaneous experiments.

I won’t go into the statistics here; interested readers can find them in the paper itself. If you have a good statistical background, you will find that authors have a done a good job of clearly explaining the theory behind CUPED, and have provided enough information for advanced experimenters to put it into practice.

The paper is at http://www.exp-platform.com/Pages/CUPED.aspx