A/B/n test hypothesis and goal
The first step in any A/B/n test is to formulate a clear hypothesis. This hypothesis outlines the intended outcome of the test and helps you measure it. For example, consider the hypothesis Variant B will lead to at least 20% more page views than the control variant. This hypothesis can result in two possible outcomes:
-
Null hypothesis - the A/B/n test shows no significant difference between the various variants and the control variant.
-
Alternative hypothesis - One variant performs better than the control, confirming the hypothesis. This is the desired outcome of the A/B/n test.
We recommend that you include a measurable goal for each hypothesis to reflect what you aim to achieve. The primary purpose of running an A/B/n test is to evaluate changes in visitor experience to improve specific business outcomes.
To test these changes and track performance, you need to configure a test goal.
For example, when your goal is to increasepage views, you must specify the pages where you want to track your goal. Sitecore uses the metrics of this goal to:
-
Calculate the required number of visits to declare a variant as the winner, or to declare that the test is inconclusive.
-
Measure the performance of each variant to identify the leading variant, or the one that's ahead in achieving the goal before statistical significance occurs.
After the test runs for 24 hours, you can view the analytics to compare the performance of your variants. This allows you to see quantitative data for each variant, eliminating uncertainty about which variant design or content is driving an increase in page views.
We recommend familiarizing yourself with test metrics and calculations to understand how Sitecore determines the winner of an A/B/n test.