Personalization tests

Current version: 10.0

After you apply personalization rules to the variants of a component, you can test the effect of the personalization. By default, all tests in Sitecore use the Subgroup Round Robin Sticky test strategy. This topic describes how personalization tests and the default test strategy work so you can understand what contacts will see when they visit your site.

This topic contains the following sections:

Evaluating the effect of content on a specific group

In order to evaluate the effect of new content on a specific target group, you need to compare it to the effect of the default content on the same target group.

During a simple A/B content test, half of the contacts see the original content and half see the new version.

In a personalization test:

  • All contacts who do not meet the personalization criteria see the original content.

  • Of the contacts that do meet the personalization rule criteria, half see the new variant and half see the original content.

Example of a test with one personalization condition

A page that you test has two versions – the original (default) version, and a version with the condition "When the current day is Sunday".

  • All contacts who visit the site from Monday through Saturday see the default version of the variant.

  • Half of the contacts who visit the site on Sunday see the personalized version of the content and the other half see the default version.

Testing multiple personalization conditions

A personalization test can include multiple conditions applied to the same component. In that case, a round robin strategy is used to ensure that all conditions that apply to a group of contacts are presented with equal frequency to that group.

Example of a personalization test with multiple conditions

The following table describes a personalized component with two conditions besides the default condition:

Condition Name

Rule

Experience

Condition 2

When the current day is Monday

Experience 2

Condition 1

When the current day of the month is equal to 1

Experience 1

Default

Always applies

Default

In the Personalize the component dialog, the conditions look like this:

The Personalize the component dialog box with the example conditions

Contacts who visit the website belong to one of four groups based on which conditions they meet. The following table shows the four groups, and describes which experiences are presented to each group:

Group

Description

Matching conditions

Experience shown to contacts from the group

Group 1

Contacts who visit the site when the day is not Monday, and it is not the first

Condition 2 = False

Condition 1 = False

Default = True

All contacts in this group see the default

Group 2

Contacts who visit the site when the day is Monday, and it is not the first

Condition 2 = True

Condition 1 = False

Default = True

Contact 1 - Experience 2

Contact 2 - Default

Contact 3 - Experience 2

Contact 4 - Default

Contact 5 - Experience 2

Group 3

Contacts who visit the site when the day is not Monday, and it is the first

Condition 2 = False

Condition 1 = True

Default = True

Contact 1 - Experience 1

Contact 2 - Default

Contact 3 - Experience 1

Contact 4 - Default

Contact 5 - Experience 1

Group 4

Contacts who visit the site when it is Monday, and it is the first

Condition 2 = True

Condition 1 = True

Default = True

Contact 1 - Experience 2

Contact 2 - Experience 1

Contact 3 - Default

Contact 4 - Experience 2

Contact 5 - Experience 1

As illustrated by the table, contacts from each group see the experiences that apply to that group, including the default experience. For example, group 4 comprises contacts who visit the site on the first of the month, when it is also Monday. Since these contacts meet both condition 1 and condition 2, one third of these contacts see Experience 1, one third see Experience 2, and one third see the default experience.

Testing a page with multiple personalized components

You can run a personalization test on a page that contains multiple components with personalization conditions. The same principles apply as with the simpler tests; all conditions that apply to a group of contacts, including the default condition, are presented with equal frequency to that group.

If, for example, there are two components that have two conditions in addition to the default, then there is a total of 9 experiences. The more experiences that there are in a test, the larger the sample of contacts that is required to run the test in order for the results to be statistically significant/valid.

Testing a component with already tested personalization rules

Sometimes you may want to add new personalization rules to a component that has one or more rules that have already been tested. An already tested rule will not be tested again. When a contact views an experience that was already tested, the test counts the visit as if the contact viewed the default condition.

Do you have some feedback for us?

If you have suggestions for improving this article,