Understanding Email Tests
What Are Email Tests?
Email Tests are a way to test variations of an Email to see how well they perform. With a large enough mailing list you can create Tests on a random selection of your list, and choose the most successful email to send to the rest. If your mailing list is not large enough to carry out this sort of Test you can send different variations of the email to your entire mailing list, and use its performance to finesse future Emails.
Tests like these are often referred to as A/B Tests, as they involve testing options (Option A and Option B) to determine which performs better. You can of course have more than two options. If we were running a Test and wanted to see which subject line resulted in the most Emails being opened we could run a Test where the same Email was sent to a random sample of our mailing list, with three different subject lines. In the example below we can see that Option C had the most opens, and so we can declare it the winner and send this email to the remainder of our list.
How Many Recipients Do I Need For A Test?
For a test where you intend to send a winning Email to the remainder of your list once the Test has completed, you'd ideally have a list of more than a thousand recipients. If you're correctly targeting your emails you likely won't be sending emails to that many recipients very often, but you can still carry out tests on your Emails, and use the results to decide what to include in Emails going forward.
If we return to the example above, we know that Option C performed best. So the next time we send an Email we'd look at what made Option C successful, and use this to inform our next Email. In this example we'll say that the three subject lines were:
- Option A: Open To Find Out How You Can Get Greens Elected
- Option B: We Need Your Help
- Option C: Can You Help? 💚
Option C was the clear winner, but to determine what made it successful we could run another Test when we next send an Email. This time using the subject lines:
- Option A: Can You Chip-In?
- Option B: Can You Chip-In? 💚
We can then theorise that the use of an emoji in the subject line had a greater influence on the number of times the email was opened, than the use of a question. Next time we send an email we may again create two options- one combining a question with an emoji, and one that uses an emoji with no question. This isn't an exact science, so it's always worth rerunning and continuously improving your Email Tests.
What Options Can I Test?
There are three things we can send variations of as part of an Email Test. These are:
- Subject: This is what recipients will see in their inbox before they open an email.
- From: This is the name of the sender, which will be displayed in the recipient's inbox.
- Email Body: This is the main body of your email- the text, images and links that make up the message to your recipients.
How Many Variations Of My Emails Will Be Sent?
If you are testing only one type of variation, then the number of options created will be equal to the number of alternate options you create. So creating three different Email subjects will create three different variations. If you are testing more than one alternate options your options will be multiplied by the alternates created for each variable. For example testing three different subject lines, and two different senders would create six separate Emails during the Test.
Subject A | Subject B | Subject C | |
Sender A | Option A | Option B | Option C |
Sender B | Option D | Option E | Option F |
If you're running tests on a small mailing list, creating this many options may interfere with your ability to determine what options would be successful if tested on a larger population, so be mindful of how many options you're creating when testing alternatives.
Related Articles
Accessing & Understanding Email Statistics
Just like everything we do, an important part of growth is to check on how well something has been done. Hitting send on an email should only be the beginning, and we should all be in the habit of checking how well our emails have performed, in order ...Using & Customising GPEW-Issued Email Layouts
To support Action Network groups in maintaining a consistent brand, the Green Party Digital Support team will occasionally release email layouts which can be customised depending on your needs. Email layouts use the same drag and drip editor as you ...Action Network Email Tips
You're probably quite comfortable sending and checking emails in a personal, or professional capacity– but sending mass emails using a digital campaigning tool is a different kettle of fish entirely, and so here are some tips on sending mass emails ...Understanding The Clip Demo Email
If you've arrived here before first requesting a demo email you'll first need to complete this form. You'll then receive an email that demonstrates how some of the Clip fields will display in an email. You might want to check out this guide before ...How To Run An Email Test
We recommend reading this guide before running your first Email Test: Understanding Email Tests. How To To Run An Email Test: Testing Different Subject Lines 1. Create a new Email as normal, by clicking on PEOPLE from the Start Organising menu, ...