A/B Tests: No narrative for you, my friend

Obama presidential campaign A/B test

Which one of the red buttons prompted the highest portion of website visitors to sign up?

If you move in exciting circles like I do, you will no doubt be familiar with some of the caveats you hear about A/B testing.

For those of you with more sheltered lives, A/B testing is a technique to determine which version of a web page is more effective in influencing visitors’ behavior.

Some visitors will see one version of a page while others see a different version. You keep track on what happened on each version, then proceed with the most effective option.

The image above, for example, shows the 2008 Obama campaign website where the only difference was the wording of the red button. See if you can guess which one was most effective. The results are at the bottom of this article.

Even small changes in A/B test options can lead to big differences in user behavior.

It is especially important on eCommerce sites where a swing of a few percentage points could impact sales. Indeed, for startups, these can be existential differences.

But behind all the altered layouts and tweaked colors, it is important to remember one thing, according to Jennifer Golbeck (@jengolbeck), “We have no idea why one of these performed better than the other.”

Director of the Human Computer Interaction lab at the University of Maryland, Golbeck’s comment came via a Coursera course called Usable Security.

On the tests, she said, “We can come up with theories, but ultimately there’s no insight in to the why.”

The same point was made by Caelen King, (@Caelen) CEO of WhatClinic.com at a recent Entrepreneurs Anonymous meetup in Dublin.

Collecting the data may feel like an exact science, he said. However, it comes with a temptation to build a narrative around the results.

“But there is no knowledge there. There is just a metric,” King warned.

As for the Obama campaign, the results are below. The “learn more” button saw 18.6% more sign ups than the original “sign up” button.

It might be fun to speculate why this was the case. Just remember that you don’t know.


Small changes. Big differences.

Images are partial screen grabs from Coursera. The website developers, Optimizely, explain the results in more detail on their blog.

2 thoughts on “A/B Tests: No narrative for you, my friend

  1. “But there is no knowledge there. There is just a metric,” King warned.

    the definition of knowledge includes facts. If you have facts you have some knowledge.

  2. I don’t know about that, Dave. I view knowledge as falling on this spectrum: Data -> information -> knowledge -> wisdom. Facts are data and information. Knowledge goes a little deeper, IMO.

Comments are closed.