CUPED’s Arrow for Analytics Nerds: More Sensitivity For Online Experiments

A group of online experimenters at Microsoft has just written a new paper that has the potential to increase the value that A/B and multivariate tests can bring to online businesses.

They have documented methods to combine pre-experiment data with experiment data to yield increased sensitivity for testing the performance of new designs and features. They call their approach CUPED (Controlled-experiment Using Pre-Experiment Data).

This approach will be beyond the capabilities of the typical small website owner using Google Analytics. However, for businesses that employ statistical analysts. CUPED is worth adopting.

By leveraging pre-experiment data, analysts can realize increased sensitivity to

  • test effects and interactions that could normally not be detected due to  lack of traffic, such as for high value user segments.
  • run more simultaneous experiments.

I won’t go into the statistics here; interested readers can find them in the paper itself. If you have a good statistical background, you will find that authors have a done a good job of clearly explaining the theory behind CUPED, and have provided enough information for advanced experimenters to put it into practice.

The paper is at http://www.exp-platform.com/Pages/CUPED.aspx

Your comments are welcome. Off-topic comments will not be published. If you have a question unrelated to this post, click the "Contact" item in the navigation menu and submit it via email.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s