Using minimum sample size
When you run an experiment, a healthy sample size enables you to make accurate statistical conclusions. A set sample size must be reached otherwise there is not enough evidence to ascertain that the experiment test results are not due to chance.
Do not stop an experiment before the minimum sample size is reached. If you stop the experiment before the minimum sample size is reached, the test becomes invalid.
Sitecore Personalize automatically calculates the minimum sample size required to meet statistical significance, based on the chosen primary goal. You can edit the sample size calculation if you want to enter different parameters for calculating the minimum sample size.
The following are the parameters for the default minimum sample size calculation:
-
Base Rate % - the base or business-as-usual conversion goal. This is set to 2% by default.
-
Minimum detectable difference % - the minimum conversion goal difference to detect. This is expressed as a relative percent. This is set to 20% by default. You can adjust the test sensitivity if you want to change this.
-
Confidence Level % - the threshold for rejecting the null hypothesis. The alternative hypothesis is what you hope your experiment helps prove. For example, variant A has more conversions than variant B. In relation to the sample size calculation, this is the confidence you have that if the null hypothesis is true, the measured difference is not due to random fluctuations. This is set to 95% by default. You can adjust the confidence level if you want to change this.