Running an experiment with optimized testing

You can run an experiment with optimized testing when you want to test multiple variants against each other and eliminate the cost of allocating traffic to low-performing variants until statistical significance is reached. If one of your primary objectives is to reach statistical significance or to learn the performance of all variants, we recommend that you run an A/B test.

An experiment with optimized testing uses the multi-armed bandit algorithm to dynamically allocate traffiic. The multi-armed bandit algorithm takes its name from the gambling industry, specifically slot machines, where players might start off equally distributing their bets between several machines, then quickly allocate their bets to only the highest-performing machine(s).

The same concept applies when running experiments on digital devices. Sometimes you can not afford to wait until statistical significance is reached and a winning variant is declared. This is true particularly when running an experiment in a finite amount of time, such as a weekend flash sale. 

When you run an experiment with optimized testing enabled, Sitecore Personalize dynamically allocates guest traffic to high-performing variants. You can easily view which variants are high-performers and see additional analytics including how many additional conversions occurred because you ran an experiment with optimized testing instead of an A/B test.

You can run the experiment with optimized testing enabled as long as you want, or you can stop the experiment, and test the two highest performing variants against each other in a web or interactive experiment. This is ideal if your organization wants to follow a champion versus challenger approach while ensuring that statistical significance has been reached before allocating all traffic to one variant.