Experiment metrics and calculations in Sitecore Personalize
This topic describes how Sitecore Personalize uses metrics to determines a variant winner. Because these metrics are based on the primary goal, we recommend that you familiarize yourself with goals before reading this topic.
For Sitecore Personalize to declare a variant winner, the experiment must meet the minimum values set for these parameters in the Experiment options pane:
-
Sample size - SItecore Personalize automatically calculates the number of sessions that must be reached to prove the experiment conclusive, based on the primary goal. This is the minimum sample size required to determine that the experiment test results are not due to chance.
-
Minimum detectable difference % - the amount of lift or change from the Base Rate you want the test to detect. The default value is 20% but you can adjust it between 1% and 50%.
Increasing this value decreases the required minimum sample size. You can adjust the test sensitivity if you want to change this.
-
Confidence Level % - the amount of evidence your organization requires for proving that the difference between the variants' conversion rates is not due to chance. The default is 95% but you can adjust it between 80% and 99%.
95% is an accepted standard to reach statistical significance. The confidence level you set depends on your organization's risk-tolerance for accuracy in experiment results. Increasing the confidence level increases the required minimum sample size. You can adjust the confidence level if you want to change this.
If the experiment reaches the Sample size but not the Minimum detectable difference % and Confidence Level %, Sitecore Personalize concludes that there's no difference between the variants. The experiment is labeled as inconclusive and no variant winner is declared. If there are multiple variants, Sitecore Personalize determines that the variant with the biggest uplift against the control is the winner.