Hippos are the Most Dangerous Animals on Earth
Read on to see What HiPPOs Have to do With UX Design
Photo Credit: P. Brundel
It’s hard for some people involved with web/app design and development to accept that they can’t rely on intuition, judgment or even qualitative testing methods to reliably predict the success of a new feature. When I worked at Microsoft, one site manager told us his team didn’t need to A/B test their site redesign because “Good design is in our DNA!”
My A/B testing colleagues and I have been tracking expert business, design and technical predictions for years and we have found that the experts, no matter how much experience they have or how highly paid they are, cannot reliably predict user and customer preferences. 
Despite this body of evidence, many people persist in acting as if such a priori opinions are reliable. This is The HiPPO Syndrome. HiPPO = Highest Paid Person’s Opinion: an acronym that was first popularized by Avinash Kaushik and by our Microsoft Experimentation Platform team. I designed HiPPO stress toys, posters and web pages that helped popularize the acronym.
If you hear a designer or site owner confidently declare that a new design doesn’t need A/B or multivariate testing because they created it very carefully using their very best secret sauce design practices and usability testing, run away as fast as you can. You are in the presence of HiPPOs, the most dangerous animals on earth.
Some designers and managers have attacked A/B testing because of a misconception that testing somehow replaces or diminishes design. Nothing could be further from the truth. Good design comes first. The key is that it is the end users who are the final arbiters of design success and A/B testing is the instrument that informs us of their judgment.
When I worked on the Microsoft Experimentation Platform, we carefully tracked the results of A/B and multivariate tests over time and compared actual versus expected outcomes. We found that experienced UX designers and analysts predicted the correct outcomes about 1/3rd of the time, the results were ambiguous 1/3rd of the time and people were dead wrong 1/3rd of the time. Colleagues who had previously worked at Amazon reported that the same approximate ratio was also true there. A quote from a Tnooz article on A/B testing of online Hotel content reinforces our point. A Posadas Hotels’ Sales Manager is quoted as saying:
“We learned that our opinion and the customer’s opinion about our best photo are not the same. We saw a 46% improvement in click-through compared to the image we originally were using. It definitely blew us away.”
If you don’t believe me and think you can predict design success significantly better than 1/3rd of the time, I invite you to visit WhichTestWon.com. Each week, the site presents a completed A/B test and asks its visitors to guess whether they think version A or B won before showing the actual results. Have a look at some of the past tests and see how you do. (The site has put many past results behind a paywall, but there are still some you can browse for free. You just have to click through the link to each test to see if it is paywalled or not.) Of course, if after trying the site out, you find it useful, I encourage you to buy a subscription. In any case, you always subscribe to their free weekly newsletter.
TravelTekker 
I agree that it’s essential to test designs, because the good design is the one that most efficiently and reliably produces the desired outcome. Otherwise, it’s wasted design. Unless, of course you are just designing for design’s sake!
Pingback: Hotel Image A/B HiPPO Fail • Travel Search Semantic Revolution • Ancillary Air Fees Hurt Passengers • WestJet Compassion | TCTReview
Is there a book or information about dealing with the HIPPO?
Start with Avinash Kaushik’s blog post, which I linked to in my post. A simple web search will turn up lots of other useful online content. I am not aware of a book on the subject.
Thanks
–Thomas
Thank you!
Jeroen,
I thought about your question some more following another comment to this post and so I did a quick search on Amazon using the query “Highest Paid Person’s Opinion.” If you try this, you’ll find some books that might help. They’re not exclusively about dealing with HiPPOs, but there is some relevant material.
I wish there was a way to deal with HiPPOs. My only success has been to bring up examples where the HiPPO (or someone he respected) was surprised at the customer reaction to a product or design choice and then leveraging off that memory to push the idea that we must test to find out what the user thinks.
Thanks for your comment Gilbert.
Some HiPPOs are hopeless, but others just need to be taught. Semi-logical HiPPOs can be trained with repeated doses of data. As I mention in the blog post, at Microsoft, we recorded the outcomes of A/B tests. We offered frequent brown bag lunches (promoted with free HiPPO food) where we showed slide decks with A/B test hypotheses and then asked the audience to predict what the outcome would be. We then told them the actual result. Many skeptical HiPPOs started to see the light when they found that their intuition was slightly more accurate than a coin toss……..if they were lucky! You can find a few examples of the tests we ran in papers and PowerPoint decks at http://exp-platform.com, which is maintained by my former boss at Microsoft, Ronny Kohavi. It’s a great resource.
Pingback: 2014-02-04 TravelTech News Review: Seahawks Edition! | TravelTekker