A/B testing is a key best practice to help you improve your email engagement rates. You execute an A/B test by comparing a test email, changing only one variable such as subject line, against your baseline email to determine which version drives the best results. Our recommendation is to carve off 10% from your initial Segment to use for the A/B test, and then splitting that portion in half, one half to receive the baseline, and the other the test. After determining the "winner", you send that email to the remaining 90% of the initial list.
For Eloqua10 users, there is capability to run an A/B test in Program Builder, but since you're sending emails, we recommend you use the Campaign Canvas. This article provides step-by-step instructions to create and use a Campaign Template for your A/B testing.
- Create your A/B Testing Campaign Template
- Configure the Template for your A/B Test
- Best Practices for Testing
Create your A/B Testing Campaign Template
First, you'll need to create two Shared Filters. These will be used to execute the split of your lists: first, to carve off 10% against which to run the test, and then an even split for the two test emails.
The Criteria in your Filters use the unique Eloqua Contact ID value to determine into which split the Contact will be placed. All Contacts created in Eloqua are given a Contact ID immediately on creation, and the values end in a series of numbers. We're going to use these in an inventive way to create the split -- thank you to Eloqua Product Specialists Ryan Wheler and Anirude Gurcharn for developing this method!
1. Create a Shared Filter for the 90:10 Split
The single Criteria in this Filter is configured like so:
The comparator value is looking for any record with a Contact ID in which the second-to-last number is a 1. The * and the ? characters (an asterisk and a question mark, respectively) are referred to as "wildcards" in this type of application. The asterisk represents any number of any characters and the question mark represents any one character, and your value entered is "*1?" (without the quotes). To illustrate, if the Contact ID is ABC00000000093217, it will match this Criteria, and thus be included in this Filter. NOTE: This is an update from past versions of this article and uses newly-available Filter comparators in Segments.
2. Create a Shared Filter for the 50:50 Split
This Filter has multiple Criteria, and they're joined by ORs -- we want Contact records with an ID that matches any one of these Criteria:
In this case, we're using the last digit of the ID (hence the newly-available comparator of "ends with"), and since we want half of the records, we have Criteria for the odd numbers 1, 3, 5, 7, and 9.
If you're not sure what this means, don't worry. You'll only need to create these Shared Filters once as you see above, as you can reuse them as-is for all your future testing.
3. Create your Campaign.
First, drag the Audience, Assets, and Decisions onto the Canvas and connect them exactly as you see below. We'll go through configuring each following the numbers on the red arrows.
1. Wait
We need to include this "Wait" Action because the Campaign Canvas doesn't allow a Decision as the first action from the Segment. We can configure it for a small amount of time though, like so:
NOTE: If you are entering fractions of an hour in a Wait Action, make sure the value results in a whole number of minutes. For example, 0.25 Hours equals 15 minutes.
2. Shared Filter Member? -- carve off 10% of your Segment
Select the "90:10 Split" Shared Filter that you created in the step above for this Decision. Approximately ten percent of the Contacts in your initial Segment will meet the Criteria in this Shared Filter, and therefore pass down the "Yes" branch from the Decision.
3. Wait -- for the 90% that will receive the winning email
From the "No" branch of the "Shared Filter Member?" Decision, set the Wait for 3 days, or however much time, plus a window, you want to let your test run. You should have a good idea of the winning email in 24 hours after executing your test, and know with certainty by 48 hours after -- at this point, the vast majority of recipients will have responded, if they respond at all.
4. Shared Filter Member? -- split the test group into half
Now let's configure the execution of the split, again using a "Shared List Member?" Decision, but this time with your 50:50 Split, like so:
Approximately half of the Contacts entering this Decision will meet the Criteria in this Shared Filter and move down the "Yes" path, while the remainder go down the "No" path.
5. Save As Template
At this point, you've configured all of the elements you need to turn this Campaign into a Template that you can re-use for your future A/B tests! Your Campaign should look this:
To save your new template, simple select "Save as Template…" from the Settings menu:
Configure the Template for your A/B Test
Before we go to the Campaign Canvas, you'll need to create the following items for your A/B test:
- Your initial Segment, the full audience to which you plan to send the message
- Your baseline Email
- Your Test Email
1. In the Campaign Canvas, create a new Campaign using your Template, like so:
2. Configure your initial Segment:
3. Add your Emails:
4. Change the dates to those appropriate for your Campaign:
5. Activate your Campaign.
6. Wait for the results of your Test. Be sure you do not wait longer than the number of days you configured in the Wait Action for the recipients of the winning Email! Your Campaign members will automatically exit the Campaign otherwise.
7. In Insight, run the report "Campaign Analysis" for your Campaign. You can either select it in the prompt, or create a Filter on the results, or find it by eyeballing the list of Campaigns. Click the value for your Campaign in the column headed "Total Sends" -- this will drill-down to the report with the results for each Email in your Campaign.
The metrics you'll use to determine the winning Email will depend on what variable you tested. For example, if your test was on subject line, you'll likely pick the Email with the higher Open Rate as the winner.
8. De-Activate your Campaign.
9. Add the Asset to send the winning Email:
10. Check that your Campaign Reporting dates are correct, and re-Activate your Campaign. The Contacts sitting in the Wait Action will be sent the winning Email.
Identify one variable you'd like to improve and develop a plan of attack. Here are some tips for testing:
- Frame the problem in the form of a question. Propose a hypothesis: an educated guess as to the cause of the results. Then, determine what you'll test to prove or disprove that hypothesis.
- Try to avoid testing too many elements at once, so as not to cloud your results.
- Be sure you can measure what you are testing.
- A technical note: if you wish to test one audience against another (everyone still receives the same email), you will need to have separate instances of your Email for each. At this time, Eloqua Insight is only able to report on Email engagement, not on a Segment.
Here are some suggested targets for your testing:
- Subject lines
- Highly graphical versus mainly text emails
- Email or landing page layout, including offer placement and form length
- Adding humor
- Additional or less copy
- Including segment-specific content
Looking for some great examples of A/B testing on both emails and landing pages? Check out Anne Holland's site "Which Test Won?" -- see if you answer correctly!