How to Setup an AB test with a Rollout
In this lesson, we will cover how to setup a subject line or content AB test that sends the winning version to a rollout group. A “rollout group� is a chunk of the list of contacts you want to send to, that you put aside for after the test. This allows you to run the test on a sample of your contacts, determine which version is the winner and then send that winning version to the rollout group you put aside. This ensures that you are sending the most effective version of your message to the majority of your list.
Setting up an AB Test requires a working knowledge of CONNECT and its basic features. Before reading this article, we recommend that you review the following articles:
1. Creating an AB Test Message
To create an AB test message follow the below instructions :
- Click on Compose
- Hover your mouse on " Create a message"
- Click on A/B Test
2. Setting up the Message
- The process for setting up your AB Test message is very similar to the regular process for setting up a standard message except that you need to define the Test Type and where you wish to get the HTML of your message from.
- Test Type: Subject line This will allow you to define and test various subject lines with the same HTML template. Content This will allow you to define and test various versions of your template with the same subject line.
- Using HTML from: This feature allows you to grab the HTML code from a Template or an existing Standard message and use it for the AB Test message. If you are setting up a Content test, it will appear as the first version of your content.
3. Subject Line A/B Test Message
- To define your subject line versions, navigate to the Envelope tab where can define up to 10 different Subject line versions.
- For each subject line, you can take full advantage of the Dynamic subject line feature and add in personalization or logic to test different subject line scenarios.
- To add a new version, simply click on the "+" icon. You can also remove versions by clicking on the “-“ icon.
4. Content A/B Test Message
- To define your content versions, navigate to the HTML tab where you can define up to 10 different Content versions.
- Content versions require a few extra steps in order to create each version.
- To add a content version, click on the “+� icon next to version changer.
- A dialog will appear that will ask you to either copy the HTML from an existing version, from a Template, or an existing Standard message.
- For each version, you will also need to create a text version on the Text tab.
- Finally, you have option of define different link names for each version on the Click Tracking tab.
5. Previewing each Version
- The Preview tab provides a great opportunity to view each of the different versions of your AB Test message.
- By selecting the version in the drop down menu, you will be able to preview each version of the subject lines or content.
6. Sending Tests of each Version
- Another way to test the versions of your AB Test message is on the Test tab.
- You can select which versions you would like to send to each of the recipients you select (including yourself).
7. Setting up a Deployment
- Now that you have set up your message, you can start setting up the AB Test Plan. To start, navigate to the Deploy section, click on the AB Testing tab and click on "Create New A/B Test".
- The first step of the set-up process will look very similar to the set-up of a regular deployment. The difference here is that you need to provide a name for your Test Plan.
- We have also introduced a new feature where you are able to save and return to an AB Test Plan set-up as long as you define a “Test Name� and click “Create Test Plan�.
- However, in order to move on to the “Test Plan Builder� step, you must define the message and the target before it will become available.
- To finish off this step of the set-up, select your AB test message, define your target and choose your option for scheduling.
8. Setting up your sending Strategy
- Once you are on the Test Plan builder step, you can review the versions from your AB test message and the total size of your target before you split it into test and roll-out groups.
- To get started, select “Rollout� from the Sending Strategy drop down.
- This will cause a slider to appear where you can change the splits for your test groups, ultimately affecting how large your roll-out group is (in red).
9. Reminder Notification
- Another option you have when setting up a test plan with a rollout is to set a reminder notification so that you are reminded that you need to launch a winning version to your rollout group.
- To set up the notification, define when you want the notification sent after the test groups have been sent to.
- You also have access to a few extra options that will affect the notification.
- Include a recommended version for sending to the rollout group in the notification based on Open rate, Unique click rate, or Unique click rate as % of opens.
- Repeat the notification until a winner is chosen Selecting this option will cause the notification to be sent again and again at the interval you set until you select a version to send to the rollout group.
- Send a notification to other stakeholders by default, the notification will be sent to the user that created the AB Test Plan but you can also include other stakeholders so they get the notification as well.
10. Deployment Review
- The deployment review step will allow you to review everything before you begin the test plan.
- Just as a reminder, when you “Start� the test plan, only the version or “Split� groups will be sent to.
- The Rollout group will not be deployed to until you select a winning version.
- Toggle between version to preview the different data points Preview splits in count by version.
11. AB Test Plan Review
- After launching the AB Test plan, you will be taken back to the Deploy section.
- From here, you can : See the status of your AB test Click on the test plan name to view your test plan.