Experiment metrics and calculations in Sitecore Personalize
This topic describes how Sitecore Personalize uses metrics to determines a variant winner. Because these metrics are based on the primary goal, we recommend that you familiarize yourself with goals before reading this topic.
For Sitecore Personalize to declare a variant winner, the experiment must meet the minimum values set for these parameters in the Experiment options pane:
-
Sample size - SItecore Personalize automatically calculates the number of sessions that must be reached to prove the experiment conclusive, based on the primary goal. This is the minimum sample size required to determine that the experiment test results are not due to chance.
-
Minimum detectable difference % - the amount of lift or change from the base rate you want the test to detect. If you increase the minimum detectable difference, the required minimum sample size decreases. The default is 20%. You can adjust the test sensitivity if you want to change this.
-
Confidence Level % - the amount of evidence your organization requires for proving that the difference between the variants' conversion rates is not due to chance. A confidence level of 95% is an accepted standard to reach statistical significance. The confidence level you set depends on your organization's risk-tolerance for accuracy in experiment results. If you increase the confidence level, the required minimum sample size increases. The default is 95%. You can adjust the confidence level if you want to change this.
If the experiment reaches the Sample size but not the Minimum detectable difference % and Confidence Level %, Sitecore Personalize concludes that there's no difference between the variants. The experiment is labeled as inconclusive and no variant winner is declared. If there are multiple variants, Sitecore Personalize determines that the variant with the biggest uplift against the control is the winner.