1. Get started with A/B/n testing

Configure A/B/n test settings

If you don't want to use the default test values, you must configure your goal and adjust the test settings before you start running an A/B/n test. These settings define how the A/B/n test runs and can impact the accuracy of your test results.

Access the settings

You can manually access the test settings to configure your goal and other parameters. If you start a test without making the required configurations, you will be prompted to edit them first.

To access the test settings:

  1. In the Page builder Editor mode, open the page that contains the A/B/n test you want to edit.

  2. In the right-hand pane, click Configure.

    Click Configure in the right-hand pane to access the test settings.

  3. In the A/B test settings dialog, configure your goal and assign traffic to your variants. You can make any necessary changes in the Optional configuration section to avoid using default values.

Configure the goal

Sitecore uses the goal metrics of an A/B/n test to assess the performance of each variant and determine the winner. To start your A/B/n test, you must configure the goal of your test by indicating the pages where the goal should be tracked.

To configure the goal:

  1. In the A/B test settings dialog, you'll see the name of your test. To edit it, enter the new name in the field provided.

    In the Configure test goal section, click on the goal you want to track.

    Choose the goal of your test in the A/B/n test settings dialog.

    The following goals are available:

    • Increase page views to measure the views of one or more other pages of the site.

    • Increase form submissions to measure which form variant generates the most submissions.

    • Decrease bounce rate to measure the percentage of visitors who enter a specific page on a website and then leave without continuing to view other pages.

    • Decrease exit rate to track how many visitors leave your site from a specific page, after navigating through other pages.

  2. If you selected the Increase page views goal, you must specify one or more pages where you want to track the goal: in the Site tree section, click Down arrow and select one or more pages from the drop-down menu.

    Note

    The goal-tracking pages you selected must be published in order to start the test.

  3. Click Save.

Assign traffic to variants

You can adjust the default traffic percentages while your test is in Draft status. By default, if your test includes only two variants, Sitecore splits the traffic equally, assigning 50% to Variant A (Control) and 50% to Variant B. For any additional variants, the default traffic assignment is 0%.

To assign traffic to variants:

  1. In the A/B test settings dialog, click Assign traffic.

  2. Enter the traffic percentage for each variant using whole, rounded percentages. Each variant must have at least 1% traffic assigned, and the total percentage for all variants must equal 100%.

    Set the percentage of traffic assigned to each variant.

  3. Optionally, if you have more than two variants and want each variant to automatically receive an equal share of traffic, click Evenly distribute. For example, if you have Variants A, B, and C, the traffic percentages will be split into 34%, 33%, and 33%, respectively.

  4. Click Save.

Set automated actions based on the test outcome

You can make the changes to the default automated actions while your test is in Draft status. By default, Sitecore selects what happens after your test reaches statistical significance based on the test outcome.

You can start your A/B/n test without changing the default automated actions.

To set automated actions based on the test outcome:

  1. In the A/B test settings dialog, in the Optional configuration section, click Automated actions.

    By default, the A/B/n test is configured to assign all visitor traffic to the winner if there is one, or to keep running the test if the results are inconclusive.

    Select the actions you want to happen automatically after your A/B/n test reaches statistical significance.

  2. Select the action you want to happen automatically after your A/B/n test reaches statistical significance, based on the outcome.

    • If there is a winning variant:

      • Assign all the traffic to the winning variant - to redirect all visitor traffic to the best-performing variant.

      • Assign all the traffic back to the control variant - to redirect all visitor traffic back to Variant A (Control).

    • If the test results are inconclusive:

      • Keep running the test - to maintain the current traffic assignment and continue the A/B/n test.

      • Assign all the traffic back to the control variant - to redirect all visitor traffic back to Variant A (Control).

  3. Click Save.

Edit the advanced options

You can make the changes to the default values in Advanced options while your test is in Draft status. By default, Sitecore sets the percentage of visitor traffic to include in your A/B/n test and the minimum sample size required for each variant.

You can start your A/B/n test without changing the default values in Advanced options.

To edit the advanced options:

  1. In the A/B test settings dialog, in the Optional configuration section, click Advanced options.

    By default, Sitecore allocates 100% of visitor traffic to the A/B/n test and calculates a minimum sample size of 21,110 visits per variant, required to achieve statistical significance.

    Set the test traffic allocation and adjust the minimum sample size in the Advanced options section.

  2. To edit the default traffic allocation, adjust the percentage of overall visitor traffic you want to include in the test, between 1% and 100%. For example, setting this to 20% means that only 20 out of 100 visitors will participate in the test.

  3. To edit the default minimum sample size, edit the following fields:

    • Base rate - the current conversion rate of the control variant, set at a default of 2%. This rate can be set between a minimum of 0.5% and a maximum of 50%. Increasing the base rate decreases the required minimum sample size.

    • Minimum detectable difference - the smallest amount of change from the Base rate that you want the test to detect, set at a default of 20%. This can be set between a minimum of 1% and a maximum of 50%. Increasing this value decreases the required minimum sample size.

    • Confidence level - the level of certainty required to determine that the differences observed between variants are statistically significant and not due to chance. The default level is 95%, but this can be set between a minimum of 80% and a maximum of 99%. Increasing the confidence level increases the required minimum sample size.

  4.  To revert the sample size parameters to their default values, click Reset to defaults.

  5. Click Save.

If you have suggestions for improving this article, let us know!