Adjusting the confidence level of an experiment

You can increase or decrease the confidence level of the experiment, depending on your organization's risk-tolerance for accuracy in experiment results.

The confidence level is the threshold for rejecting the null hypothesis. In relation to the sample size calculation, this is the confidence you have that the null hypothesis is true and the measured difference is not due to random fluctuations. If you lower the confidence level, the required minimum sample size to reach statistical significance reduces.

Doing this is a trade-off, because your conclusions are more likely to have errors, although the time until the test is conclusive is reduced.

Alternatively, if you increase the confidence level, the required minimum sample size to reach statistical significance increases. Only do this if you want to ensure more accurate results at the expense of lengthening the time until the test is conclusive.