Split Testing

I’m often asked by clients what the best way is to run a split test on ReachMail. Here’s a step-by-step tutorial to running a subject line split test, it takes a bit of leg work but the results can be worth it.

Overview and Goals

Split testing (or A/B testing) is the process of testing varying content to a select portion of a list and comparing the results to see what content performs better. A common method is to run subject line test. In other words does ‘A Special Offer for {{first_name}}’ perform better than ‘{{first_name}}, a special offer for you!’

Our goal is to create a split test of four different subject lines on a list of 10,000 records, each subject line will be distributed evenly amongst the list (2,500 records per subject). Furthermore we will not split the list into 4 smaller lists to do it.

Here’s the short version of how it will all go down.
We’ll create four different mails, each with a different subject line. Then using demographic filters, we’ll create mail filters that will be applied in the sending process. The mail filters will select the recipients which receive the mail allowing use to reuse the main list. Then we’ll use the Compare Mailings report to view the results.

Prepare and Import Your List

The constraint that we’ve applied to this split project is that the list can’t be broken into smaller bits. The list will need to be modified to include an mail key code for each record so that demographic filters can be used to create mail filters for each of the four mails.

So what we need to do is upload a list with a column for mail key code. Each record will need to have a value in that column. For the sake of this example I’ll be using 100, 200, 300 and 400 for the key codes.

Note: If you have an existing list that you want to use, you can export the active records, add the required column, and add it via import to it’s source in ReachMail, saving duplicate values from the uploaded list.

The easiest way to add the column would be to open the list in Excel, insert a column named ‘Mail Key’ and then enter your key value in the first cell. Grab the lower right corner of the cell and drag it down to copy the contents to however many cells you select, 2,500 in our example. Repeat until the whole list is updated.

The list should look something like this when done.


Frank Aker,faker@reachmail.com,100
Bob Jones,bjones@reachmail.com,100
...2,498 more records...
Dave Little,dlittle@reachmail.com,200
...etc...

Upload the modified list and you’re ready to roll.

Create Mail Filters

Once the list is in place, the next step is to make the mail filters for each of the mail keys. The mail filters are built using the demographic filters. You will create one filter for each mail key. Use the filters to select all records in which the value of the mail key code column matches the mail key you want to filter for. In other words, if I’m building the mail filter for mail key ’100′ I would choose to include all records in which the mail key code column value matches ’100.’ Make sure to name the filters something obvious, like ‘MARCHSPLIT-MAILKEY100′ so they can easily be applied when scheduling the mail.

For the sake of brevity, I won’t do a step-by-step here. Check out the Demographic Filters on Send video at ReachMailTV.

Create the Mails

Now that the list is in place and the filters are generated it’s time to make the mail. To keep things organized, make a group for the mail and then name each mail for the key which it will be applied to. For example in the group ‘MARCHSPLIT’ you would have mails MAILKEY100, MAILKEY200, etc.

Don’t forget to customize each mail with a different subject line or any other content that you want to test.

Schedule the Mails

Now it’s time to schedule the mails. Make sure that when scheduling you apply the mail filter to the correct mail. See screen-shot below.

Notice in the screen-shot that the mail filter has been carefully matched to the mail name. That’s why it’s helpful to use the same naming conventions on the filters and the mails.

Compare the Performance

Once the mails have been delivered and you’ve allowed a couple of days for all the data to be reported, it’s time to see what worked and what didn’t. Use the ‘Compare Mailings’ report to see the summarized statistics from each mailing. Don’t forget to move beyond the open rate and check the individual mailing reports to see the click data too. Once you’ve considered all the data you should be able to make an informed decision about what subject lines are most compelling to your subscribers.

Summary and Conclusions

So why do it? Split testing is an easy way to poll your subscribers without asking them to take a survey or a poll. Simply by measuring their reactions you can get a general idea of what works and what doesn’t.

When should I do split testing? It’s easiest to do at the beginning of your subscriber life-cycle. For example, if you’re a retailer with a new store and you’ve been collecting email addresses at said store, your first mailings to those addresses would be a great time to run split-testing. If you’re in the middle of the subscriber life cycle and you’ve got a winning formula already, strong open rate, solid click through, I don’t see a need for any split testing. On the other hand, if your open rates have been flagging and click through is down, split tests would be a good way to re-evaluate your content.

Can I test content beside the subject line? Sure, but don’t go nuts. You might also consider changing your navigation links or the color or link style in the call to action. Keep it simple though and don’t change too much per split test. You’ll actually learn less if the emails are too disparate. Pick one thing to test and stick to it,

Get More From Form Wizard

Not too long ago I wrote a post about Form Wizard and how to utilize it and integrate it into your website.

Form Wizard is a handy tool to easily create a simple sign-up form to add to your website. The form writes directly to the ReachMail list of your choosing. If you want to read the how-to instructions, check the old post or watch the video.

Form Wizard has a powerful feature that is sometimes overlooked. It’s the Auto-Responder. Sounds cool, right? It is. What does it do? It sends an automatic email to any new users as soon as they are in your list. Why is that important? It helps you put your best foot forward with all your new list members.

Here’s a sample scenario. Say your company (Acme, Inc.) sends out a monthly newsletter on the first Monday of every month because you’ve wisely heeded the advice of some email marketing experts. A new user joins your list on the tenth of the month. They’ll be waiting a couple of weeks at least for the first email. In the meantime, they’re apt to forget that they signed up. Know what happens when users forget they signed up? Opt-outs. And Spam Complaints.

Enter the Auto-Responder. The auto-responder comes to the user shortly after they’re entered into the list and is a great reminder that they’ve joined and gives you a chance to welcome them and give a preview of what they can expect from your emails. If I were making an auto-responder for this monthly newsletter list I might say something like this: Thanks for joining the Acme, Inc. monthly newsletter! We hope you find it an informative window into our company. You can expect our newsletter on the first Monday of every month. In it, we’ll have recap of the the past month, a look ahead to the new month and an employee spotlight feature. If you’d like to take a look at our newsletter archive, use this link.

The auto-responder shouldn’t be long but you should get across at least the following:

  1. A thank you and welcome.
  2. A content preview and a frequency reminder.
  3. One other piece of content the user can take action on.

Of course, this was just one example. If you’re a retailer you might have a special coupon in the auto-responder; there are a variety of possibilities.

Ready to put an auto-responder on your forms? Here’s what to do. In the Form Wizard section of Libraries and Tools, click the Manage Forms link. Find the form you want to modify in the list of forms and from the Action menu all the way to the right choose ‘Create Auto-Responder.’ From there you be able to use any of the basic mail creation methods to make the auto-responder (template, webpage, file upload, scratch).

If you have any questions about auto-responders or want to discuss your situation individually, drop me a line at dnielsen@reachmail.com

Making Sense of Opens

It’s a pretty safe bet that at some point all new ReachMail clients ask me, ‘How does ReachMail track opens?’

Well here’s the deal; opens, or ‘reads,’ if you prefer, are tracked using a unique image that’s appended to the end of every outbound email. This images is small (1px X 1px) and transparent and unique amongst users and campaigns. That is to say each campaign that goes out has a unique image generated for each user that the campaign is delivered to. When a user loads the images in a mail a request is made of our servers to deliver the image to the user and at that point we record an ‘open.’

Re-read that last sentence. Notice I said ‘loads the images,’ not ‘opens the mail,’ not ‘loads it in the preview pane, just ‘loads the images.’ Loading images in the mail is the only action that will record an open. There’s simply no way of knowing if a recipient double-clicks a mail to open it in a new window or scans it in the preview pane. All those actions happen locally at the user level, we’ve got no way of tracking those actions. All we can do is make note of a user requesting information from us and so we track that as an open.

I know, now you’re probably thinking that opens are a flawed measurement and you’ll never pay attention to them again. Hold on a minute though. Try to think of opens as a gauge of perception rather than an isolated metric. We’ve talked about perception in this blog before and of how to increase open rates and click rates by sending mail perceived as valuable. Well, if someone takes the time to download your images, they perceive some value in your mail and that’s what you want to know. It’s not so much that they’re loading the images, it’s about why they’re loading the images.

So the next logical question is ‘What’s the margin for error in the open report?’ Good question. The fact is, there’s really no way of knowing. We can’t track any actions without the user taking any action, so though there may be people who read the mail word for word, if they never load the images, we’ll never know. Again, don’t dwell on the people who aren’t being recorded, try to view the opens as a measure of how the mail is perceived. Opens won’t increase perception, perception will increase opens.