Split test results

Using the Split test results report, you can analyze the results using different primary test metrics and their related metrics.

To change the primary test metric, in the Primary test metric drop-down menu in the top right-hand side of the report, select another value.

Screenshot of a split test results report.

For each value chosen from the Primary test metric drop-down menu, the report refreshes and displays the following:

  • A tabular presentation of tested widget variations and related metrics.

  • Statistical confidence at the following levels 80%, 85%, 90%, 95%, and 99%.

  • Standard deviation and probability values at 95% confidence.

  • Line chart plotting the results distribution for each tested widget variation.

The following table lists the related metrics used in our analysis with a primary test metric:

Primary test metric

Related metrics for analysis

Direct CVR

Click-assisted visit, direct conversion, and modeled CVR.

Click-assisted CVR

Click-assisted visit, click-assisted conversion, and modeled CVR.

CTR

Views, clicks, and modeled CVR.

Interpreting split test results

The report provides detailed test results at a commonly accepted confidence level of 95%, which is not conservative nor aggressive.

In a typical split test, the hypothesis is two-sided or run with two widget variations, and the null hypothesis is that the variations are similar. You can analyze the results using the following table:

z-score or standard deviation

Analysis for confidence level of 95%

Less than -1.96 or greater than + 1.96

The null hypothesis is rejected.

The two variations are significantly different.

Between - 1.96 an 1.96

The null hypothesis cannot be rejected.

Metrics and KPIs

Do you have some feedback for us?

If you have suggestions for improving this article,