Walkthrough: Running A/B tests on an email campaign message

Version: 10.4

A/B testing lets you to create two or more variants of the same email campaign and test which one receives the best response from the recipients.

We recommend that you test a maximum of two to three variants. You can create as many variants of the message as you like, but when you add more variants, you need a larger pool of recipients for the test results still to be clear and reliable.

You can only perform A/B testing on regular email campaigns, not on automated campaigns. It is not possible to test the messages based on the following templates: Simple HTML message, Plain text message, or Existing page.

Before you run an A/B test, we recommend that you plan the criteria for the test, such as how many recipients you want to send the test to, the timing of the test, and the number of message variations to include.

To prepare for, run, and complete an A/B test, you must carry out the following tasks:

Create a message variant

When you run an A/B test, you test different variations of the same message. For each variation in the message that you want to test, you must create a new variant of the original message. The original message is the first variant (Variant A), and any new message variants that you create are named Variant B, Variant C, and so on.

To create a new variant of a message:

  1. Open the message that you want to perform an A/B test on.

  2. Either add a new variant, or duplicate the original variant and all its content:

    • To add a new variant, on the Message tab, click Add variant . This creates a new message variant based on the same template as the original message but where no content is replicated.

      Note

      Choose this option if you want to test comprehensive differences between the message variations, such as larger components or the layout of the message.

      Email Experience Manager Message tab showing the plus icon next to the variant.

    • To duplicate the original variant, click Actions and then Duplicate this variant. This creates a new variant of the original message by duplicating it and all of its content.

      Note

      Choose this option if you want to test minor differences between the variants, such as the subject line, images, or the name of a button.

      Message tab drop-down menu showing the Duplicate this variant command.

    The message variants are displayed on individual tabs where you can easily toggle between the variants.

    Message tab showing an example of three variants.

  3. Make the necessary changes to the new message variant, for example, you can change the message layout, change the images, or add new components.

    Note

    If you base your A/B test message on an imported HTML template only the Variant A contains the layout. For the other variants, you have to add the message layout manually.

  4. When you have created all the message variants for the test, click Save.

Start an A/B test

After creating the message variants for the A/B test, decide how you want to send the messages, and choose how select the winning message should be selected.

To start an A/B test:

  1. Open the message you want to send and, on the Delivery tab, click the variants that you want to include in the delivery.

    Email Experience Manager Delivery tab showing an example of an A/B test.

  2. In the Size of the test drop-down menu, click the percentage of recipients that you want to include in this test. This set of recipients is randomly generated from the full list of recipients.

  3. Under Select winner, to specify how you want to choose the winner, click one of the following options:

    • Click Automatic to make the system select the winning variant for you, based on either best value per visit or highest open rate. Then, in the Automatically select winner after fields, specify how long you want the test to run before the winner is selected.

      Click Best value per visit to make the winning variant the one that generates the most value on the website per visitor. Alternatively, click Highest open rate, to make the winning variant the one that was opened most often.

      Note

      The value is calculated according to the engagement value points that are set up on your website.

    • Manual – you review the reports of the A/B test and select the winner variant based on your own criteria.

  4. Under Schedule delivery, to specify when you want to start the A/B test, click one of the following options:

    • Click Send message now to start the A/B test immediately.

    • Click Schedule message delivery to schedule the delivery of the A/B test message for a later time. Specify the date and time for when you want to start the A/B test.

    • Click Schedule a recurring delivery to schedule the A/B test message to send recurrently at a certain interval. Specify the interval, the date, and time for when you want to send the recurring A/B test.

      Note

      When you schedule a recurring delivery of an A/B test message, a recurring message is only sent if the winner of the previously sent test message has been selected, either automatically or manually.

  5. If you want to send a notification to certain email addresses when the dispatch of the A/B test has completed, under Notification, select the check box and enter the relevant email addresses. If you chose to have the winning variant selected and sent automatically, the notification is sent after the final message has been sent.

  6. If you want to send the message in the recipients' preferred language, click the More link, then, under Multi-language, click Use preferred language.

  7. Click Start A/B test (or click Schedule test if you scheduled the A/B test).

  8. If you selected Use preferred language in the Multi-language section, review the default language in the Message delivery confirmation dialog that appears.

The A/B test starts and changes the status of the message to Sending. When the system finishes sending the message to the limited set of recipients, the button label changes to Resume, but remains inactive until the winning variant is selected.

If you choose to schedule the A/B test, you can cancel the message up until the date and time it is sent, by clicking Cancel scheduling on the Delivery tab of the message.

Select and send the winning message

If you chose automatic selection of the winner before running the A/B test, the winning message is automatically sent to the rest of the recipients in the recipient list after the test has completed. The winning message is not sent to any of the recipients that were included in the A/B test.

If you chose manual selection of the winner, you need to review the A/B test reports yourself and select the variant that you think is the best. Make sure that you give your recipients enough time to read and react to the message.

To select the A/B test winner manually:

  1. In the navigation menu, under Email campaigns, click In progress.

    In the list, you can see the status of the message. The messages with the Select winner status are ready for you to select a winner.

  2. Open the message and click the Delivery tab. Under Select winner, you can see the results of the A/B test so far.

    Email Experience Manager Delivery options for an A/B test.

    The table lists each variant with the current statistics:

    • Open rate – the percentage of recipients that have opened the message.

    • Click rate – the percentage of recipients that have clicked a link in the message.

    • Value – the number of engagement value points that the recipients have generated on the website after opening the message.

  3. Based on these results, determine which variant you would like to send to the rest of the message recipients and click Choose as winner.

  4. When you are ready to send the winning message to the rest of the recipients, click Resume.

Do you have some feedback for us?

If you have suggestions for improving this article,