“I have been leading numerous AB Test campaigns for global and local companies, for simple as well as very complex websites, across various industries and countries and eventually it always comes down to the same success factor : get your test planning right or you’ll get confusing stats that will eventually lead you to the wrong decision !
Sure,…. but how to get the AB Test roadmap right ?”
It is probably as simple as complying with the following 3 key rules… :
Limiting the number of simultaneous tests on the user’s path will significantly reduce side effects and therefore the risk of misinterpreting your analytics.
What is a side effect in the AB testing world?
Simply the change of a KPI on a tested page, driven not by the test on the page, but by a test run simultaneously on another part of the user’s path.
Just imagine you set up a test on your website’s homepage to reduce the bounce rate, and simultaneously, a second test on the basket page in order to increase the conversion rate.
As the homepage bounce rate decreases, the conversion rate increases, even if the basket page test has zero impact. This is a “side effect” and it is very likely that some will wrongly conclude that the conversion rate improvement is driven by the basket page test….
If your website offers distinct users’ paths, you can do more than 3 tests at a time. For example, on a website offering a B2C and a B2B user’s path, you can probably set up 3 tests on each path, as there is limited risk that B2C and B2B users interfer and cross each other.
Some AB Test tools offer the option “one visitor, one test”, which prevents the user from running more than 1 test during his session. It limits dramatically side effects, but also reduces the number of tested users on one given test. You can therefore afford this strategy only if you have a lot of traffic.
When you plan to run several tests on a given page, you must activate them one after the other. Otherwise you won’t be able to identify which change is impacting the conversion rate.
If you don’t have a choice, here are some tips & tricks to help you setting up several tests at the same time on a webpage :
(ii) Target a different audience with each test : the first test can target the visitors located in Paris, and the second one, those located in London. Keep in mind this approach may create some bias, as the selection criteria used to create 2 separate audiences may have an impact on the 2 groups’ behavior.
(iii) Opt for a multivariate test rather than an AB Test : multivariate tests allow you to make several changes at a time, and offer as many permutations as possibles. If you test 3 changes, there will be 8 variations (2x2x2).
No matter the bypass technique you choose, it is critical that you have enough traffic so the results you get are meaningful.
Even if your website’s traffic is high, be careful not to stop your tests too early! Your users’ sample may not be broad enough, and you therefore won’t have captured the full story. For example, user’s don’t behave the same way during the week or week ends.
I would advise to run a test for a minimum of one week and if you can afford it, have it running for even 2 or 3 weeks. It will ensure you take the right decision.
“Hope those 3 key rules will prove helpful to your next AB Test campaign planning and please feel free to comment or contact me directly to discuss further the fascinating world of leveraging AB Test and Analytics to run a fully integrated strategy of continuous improvement !”