Split Testing

I’m often asked by clients what the best way is to run a split test on ReachMail. Here’s a step-by-step tutorial to running a subject line split test, it takes a bit of leg work but the results can be worth it.

Overview and Goals

Split testing (or A/B testing) is the process of testing varying content to a select portion of a list and comparing the results to see what content performs better. A common method is to run subject line test. In other words does ‘A Special Offer for {{first_name}}’ perform better than ‘{{first_name}}, a special offer for you!’

Our goal is to create a split test of four different subject lines on a list of 10,000 records, each subject line will be distributed evenly amongst the list (2,500 records per subject). Furthermore we will not split the list into 4 smaller lists to do it.

Here’s the short version of how it will all go down.
We’ll create four different mails, each with a different subject line. Then using demographic filters, we’ll create mail filters that will be applied in the sending process. The mail filters will select the recipients which receive the mail allowing use to reuse the main list. Then we’ll use the Compare Mailings report to view the results.

Prepare and Import Your List

The constraint that we’ve applied to this split project is that the list can’t be broken into smaller bits. The list will need to be modified to include an mail key code for each record so that demographic filters can be used to create mail filters for each of the four mails.

So what we need to do is upload a list with a column for mail key code. Each record will need to have a value in that column. For the sake of this example I’ll be using 100, 200, 300 and 400 for the key codes.

Note: If you have an existing list that you want to use, you can export the active records, add the required column, and add it via import to it’s source in ReachMail, saving duplicate values from the uploaded list.

The easiest way to add the column would be to open the list in Excel, insert a column named ‘Mail Key’ and then enter your key value in the first cell. Grab the lower right corner of the cell and drag it down to copy the contents to however many cells you select, 2,500 in our example. Repeat until the whole list is updated.

The list should look something like this when done.


Frank Aker,faker@reachmail.com,100
Bob Jones,bjones@reachmail.com,100
...2,498 more records...
Dave Little,dlittle@reachmail.com,200
...etc...

Upload the modified list and you’re ready to roll.

Create Mail Filters

Once the list is in place, the next step is to make the mail filters for each of the mail keys. The mail filters are built using the demographic filters. You will create one filter for each mail key. Use the filters to select all records in which the value of the mail key code column matches the mail key you want to filter for. In other words, if I’m building the mail filter for mail key ’100′ I would choose to include all records in which the mail key code column value matches ’100.’ Make sure to name the filters something obvious, like ‘MARCHSPLIT-MAILKEY100′ so they can easily be applied when scheduling the mail.

For the sake of brevity, I won’t do a step-by-step here. Check out the Demographic Filters on Send video at ReachMailTV.

Create the Mails

Now that the list is in place and the filters are generated it’s time to make the mail. To keep things organized, make a group for the mail and then name each mail for the key which it will be applied to. For example in the group ‘MARCHSPLIT’ you would have mails MAILKEY100, MAILKEY200, etc.

Don’t forget to customize each mail with a different subject line or any other content that you want to test.

Schedule the Mails

Now it’s time to schedule the mails. Make sure that when scheduling you apply the mail filter to the correct mail. See screen-shot below.

Notice in the screen-shot that the mail filter has been carefully matched to the mail name. That’s why it’s helpful to use the same naming conventions on the filters and the mails.

Compare the Performance

Once the mails have been delivered and you’ve allowed a couple of days for all the data to be reported, it’s time to see what worked and what didn’t. Use the ‘Compare Mailings’ report to see the summarized statistics from each mailing. Don’t forget to move beyond the open rate and check the individual mailing reports to see the click data too. Once you’ve considered all the data you should be able to make an informed decision about what subject lines are most compelling to your subscribers.

Summary and Conclusions

So why do it? Split testing is an easy way to poll your subscribers without asking them to take a survey or a poll. Simply by measuring their reactions you can get a general idea of what works and what doesn’t.

When should I do split testing? It’s easiest to do at the beginning of your subscriber life-cycle. For example, if you’re a retailer with a new store and you’ve been collecting email addresses at said store, your first mailings to those addresses would be a great time to run split-testing. If you’re in the middle of the subscriber life cycle and you’ve got a winning formula already, strong open rate, solid click through, I don’t see a need for any split testing. On the other hand, if your open rates have been flagging and click through is down, split tests would be a good way to re-evaluate your content.

Can I test content beside the subject line? Sure, but don’t go nuts. You might also consider changing your navigation links or the color or link style in the call to action. Keep it simple though and don’t change too much per split test. You’ll actually learn less if the emails are too disparate. Pick one thing to test and stick to it,

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>