As an email marketer and a newsletter writer, I always keep an eye on my metrics —open rates, click rates, click-through rates— as soon as I hit “send”. I'm always rooting for these numbers to increase with every email I send. I mean, which marketer doesn't want above-average metrics and conversions?
But, there is always this nagging question at the back of my mind: “How do I improve these numbers?”
I had a gut feeling that adding an interesting emoji in my subject line would be colorful and pique my readers’ interest.
I decided to test my gut feeling with email A/B testing. I sent a newsletter issue with an emoji to half of my list, and one without an emoji to the other half. Then, I waited for the results to pour in.
And, guess what?
The subject line with the emoji won! It had the highest open rate. So, I continued using the same emoji for all my subject lines to establish familiarity among my readers.
Since then, my open rates have significantly improved and I haven’t looked back!
My newsletter email open rates and click rates.
In this guide, I’ll share my learnings about email A/B testing. I’ll show you how to effectively run an A/B test, which variables are worth testing for, and techniques to improve your most important metrics – open rates, click-through rates, conversions, and more.
What is email A/B testing?
In simple terms, email A/B testing (or email split testing) is all about comparing two things and ultimately choosing the one that gives you the best results.
Here’s how it works: you send two different variations of an email to two randomly-split groups of the same list. You then measure which variation performs the best.
The winning variation is then sent to the rest of your email list to optimize your email campaign's results. It's therefore a great way to improve the effectiveness of your email marketing, as you can also apply what you’ve learned to future campaigns.
If you’re using email marketing tools like MailerLite (like I do), A/B testing your email campaigns is going to be a breeze! They come with a ton of features to make it easy for you to run A/B tests. Once the A/B test is concluded, the winning version of the test is automatically sent to the rest of your list.
Note: Remember that you should A/B test only single or blast emails like newsletters, where it’s easy to test just one variable at a time. However, tools like HubSpot do let you automate A/B tests with workflows.
Wondering why you should be A/B testing your emails? Read on to find out.
Benefits of email A/B testing
One of the best things about email A/B testing is that you get insights that are backed by data, which can help make your email campaigns more successful.
Remember how I talked about my gut feeling to include an emoji in the subject line for my newsletter readers?
Using A/B testing, I was able to test my hunch and analyze the results to see if it was effective. This way, you can test your own ideas and see what works best for your email campaigns.
You can also run email A/B tests to learn:
- Which type of subject line brings in more open rates
- What time people are more likely to open your emails
- Which CTA button makes people click
- Whether their readers prefer short copy or long copy for a certain campaign
- …and so much more!
With data-backed insights and the ability to pinpoint variables that improve open rates, click-through rates, and conversions, you can take full advantage of A/B testing to make the most out of your next email campaign.
Not sure where to start? Let’s tackle this question in the next section!
10 things to A/B test for your next email campaign
Every time you send an email, there are so many variables you can A/B test that it can quickly become overwhelming.
That's why I've split the variables into two categories: those that increase “open rates” and those that increase “click-through rates and/or conversions”. This way, you can focus on testing the right variables to achieve your specific goals.
Variables to increase open rate:
1. Subject line
Subject lines are one of the first things your subscribers notice as soon as an email lands in their inbox. It has a direct impact on open rate – so it’s one of the most important email components to A/B test.
You can A/B test a subject line with:
- Length of the subject line: According to Campaign Monitor, 41 characters seem to be the ideal length for a subject line. However, you can A/B test for shorter vs longer length subject lines to understand what encourages your subscribers to open your email.
- Personalization: How many times have you received an email from a company with your first name in the subject line? Well, sometimes it works and sometimes it doesn’t. The trick is to not overdo it and to use it for the right type of email campaign. A/B test your emails for personalization based on urgency or if you want your subscribers’ undivided attention.
For example, in the below image, notice how I open emails that mention my name with a sense of urgency from “Namecheap” and “Mariam from Softr” and ignore the ones from “Google Maps” and “Snapchat”.
- Use of emojis: If you send regular newsletters to your readers, using emojis can actually improve your open rates. It's a fun and easy way to add a little personality to your emails and get your readers' attention. Plus, who doesn't love a good emoji? 😉 Don’t believe it? You can A/B test this with your subscribers.
- Question vs. Answer or Statement: If you look at the subject lines from my newsletter below, you’ll most likely notice that they are framed as questions more often than not. During my A/B tests, I found that questions in subject lines make people curious enough to open them and find the answer for themselves instead of writing a plain statement.
- Word order: When you run an e-commerce store, the order in which you present your offers, promotions, or new launches can have a big impact on your email open rate. Try different word orders to see what resonates best with your subscribers. Consider this example: “50% OFF your next purchase using this discount code” vs. “Use this discount code to get 50% OFF your next purchase”.
- Language or Tone: The language or tone of your subject line could potentially impact your email open rates. Example: “How NOT to price your projects” vs. “How to EFFECTIVELY price your projects”. Which one would you consider opening?
2. From name or Sender’s name
Different types of “from name” or email “sender's name”.
In the above image, you can see variations of “From name”:
- [Team member’s names] from [Company]: e.g. Inka & Roberta from EmailTooltester
- [Name] from [Company]: Inka from EmailTooltester
- [Newsletter or Company Name]: EmailTooltester Newsletter
- [Name]: Inka Wibowo
Zapier's “from name”
In the above image, notice that Zapier uses two different sender names for two different types of emails. I’m assuming this works for them in terms of improving email open rates.
You can A/B test the different variations of from name or sender name to test which one works with your subscribers.
3. Pre-header / Preview text
Pre-header text is the text that appears after a subject line in an inbox. Using the pre-header text to provide more context to the reader is extremely important. It's a missed opportunity if you don't take advantage of it. So, make sure to A/B test pre-header text to improve your chances of increasing email open rates.
See below to understand how I tie my subject lines with a pre-header text for more context.
Email pre-header text
4. Sending time
I would be very unlikely to open a newsletter or an email if they get sent during weekends or past my bedtime. So, it's important to consider the best time to send your marketing emails to make sure they get noticed!
From a recent study by CoSchedule, it's well-known that the best days to schedule your email campaigns for maximum open and click-through rates are Tuesdays and Thursdays.
However, these timings may vary depending on your location, industry, and niche.
If you’re still wondering if it matters when you send your email newsletters, then you should conduct an A/B test to understand your audience behaviour and optimize to increase your email open rates.
Source: Mailjet
📌 Note: To understand more about the best timings to send your emails, read our blog post: The Absolute Best Times to Send Email Newsletters (So They Get Read)
Variables to increase click-throughs and/or conversions:
5. Body copy
Yes, everyone likes short copy. But, does it really work? Does the tone of the email matter? How does anyone know for sure?
If you’re A/B testing your marketing emails, it’s important to test the type of email copy that’s working with your audience.
Short copy vs Long copy
In 2017, Copyhackers did something different for their client, Wistia. They went against the usual wisdom that email marketers follow and created A/B test versions that were 3x longer than the emails Wistia was already using.
The result?
The longer-copy emails brought in 350% more paid conversions!
So, is a shorter email really better than a longer one? Think again.
Short copy vs. Long copy from Wistia.
Tone of the copy
To improve your click-through rates, it's important to pay attention to the tone of your email body copy. Using a positive or negative tone, or formal or informal language, can all have an impact. So take a serious look at your body copy and test whether it is engaging your subscribers enough to convert.
Tone of the email copy: positive vs. negative. Source: CampaignMonitor
Bullet points vs. paragraphs
It's a common fact that most people skim through emails instead of reading them thoroughly. However, the reading style may vary based on your industry and audience. For instance, CEOs and business owners may not have much time to spend on each email, making them more likely to skim.
On the other hand, if the audience consists mostly of content writers and copywriters, they're more likely to read the entire email to avoid missing any important information. So, it's essential to keep the audience in mind while crafting the email.
A/B testing your email body copy will help you understand if your readers prefer bullet points or paragraphs, and allow you to optimize your content accordingly.
Bullet points in email newsletter
In my newsletter, I tested using bullet points for an important section, which my readers loved. It also boosted my click-through rates significantly.
6. Personalization
Building a stronger bond with your readers and customers can lead to better engagement with your emails, and ultimately result in increased click-throughs and conversions.
Personalizing your emails by adding a name, company name, or location not only helps your readers feel more connected – it also makes the email more relevant to them.
Try these variations in your email body vs non-personalized text:
- [Reader’s Name], you’re invited to…
- Maybe it’s time to change that. What do you say, [Reader’s name]?
- We know we all love [Reader’s location] because…
A common mistake made by newsletter writers and email marketers is to assume that personalization is limited to adding the reader's name to a sentence or question.
Despite not using names, Ann Handley, writer of Total Annarchy and a WSJ best-selling author, goes above and beyond to personalize her introductions, sections, and questions. Notice how she calls me (the reader) a hotshot in the below example! 🤯 This is followed by a quirky and fun poem that ties back to her audience, i.e., marketers.
Next time you send an email, speak to your readers directly and sound like you’re excited to be in their inbox. See what happens!
Ann Handley's newsletter: Total Annarchy
As I mentioned earlier, when it comes to personalizing your emails, it's important to not go overboard. Make sure to A/B test elements of your email personalization to better understand your audience's preferences.
Remember, the goal is to improve your click-through rates and conversions, so don't be afraid to test different variables and see what resonates with your subscribers.
7. HTML vs text
I tend to get put off when I receive an email from a company that uses too many HTML elements (such as images or buttons) to structure their content. It makes me feel like they're “trying too hard” to win me over, and I end up not taking them seriously.
It's important to strike a balance between a visually appealing email, and one that comes across as genuine and personable.
I'm a fan of text-only emails that are simple and straightforward. They don't have any flashy designs to distract me from the main content, and they get straight to the point. Sometimes less is more, and in this case, I think it applies.
If you're worried that your email's HTML elements might be turning off your audience, don't fret. You can always A/B test with both versions and see which one works best to increase conversions. Remember, it's important to make your emails visually appealing, but not at the cost of losing your audience's attention.
(Mostly) text email
HTML email
Litmus, an email marketing platform, conducted an A/B test where they compared HTML and plain text emails for a webinar registration. They found that a plain text email with a personal touch resulted in 63% more conversions for their customer segment than their typical HTML email. This confirmed their hypothesis that plain text emails work better for their customers.
8. Image vs. No Images
When creating an email marketing strategy, it's important to minimize distractions for your readers in order to increase conversions.
Although images can provide extra context for readers, they may not be as necessary depending on your specific industry and niche.
You can A/B test images for:
- Text only vs. Images
- One Image vs. Multiple Images
- Static Images vs. Animated GIFs
- Image grid patterns
Source: ImpactPlus
9. Design & Layout
If you're an e-commerce company looking to include a comparison between two products, a two-column layout in your email may be the way to go. However, if you want to highlight only a single product, a single-column layout could be more effective.
A/B test your email design to understand whether you’re getting the best results you can.
Here are a few ways you can do that:
- Colors: Test for a different background color for your emails or keep it white by default.
- Layout: Test for single or multiple-column layout structure.
- Style: Test for different styles to display your images and text (screenshots vs. illustrations, font styling, etcetera).
Different layout design of the same email. Source: Wordstream
10. Call-to-action (CTA)
“Sign up for EmailTooltester’s Newsletter” or “Claim your free email marketing guide now!”
Which one would you rather click?
If you want to bring more clicks, conversions, and sales, Call-to-action (CTA) buttons are crucial. That's why A/B testing them is a great way to get data-backed insights for making your next email campaign a success.
A few ideas to test your CTAs:
- Text links vs. buttons: Consider using a button to make your CTA stand out from the rest of your email content. Test it against a text link (which may work better if you want a casual and conversational tone)
- Colors: Test for different button and hyperlink colors to grab your readers’ attention.
- Placement: Should the CTA go above or below the fold? Test different placements of your CTA button to optimize for clicks.
- Copy: Make your copy more action-oriented or show the benefits of clicking on it. For example: “Watch the video” vs. “Level up your skills with this video!”
- Punctuation: Try adding an extra exclamation mark to sound enthusiastic, or remove it to sound more natural.
Remember, the goal is to make your readers feel compelled to take action, so make sure your CTAs are clear and impactful. 
Plain text CTA
Button CTA
10 Best Practices for Email A/B Testing
Now that we understand what email A/B testing is and how to do it, let's follow some best practices to maximize our chances of success.
1. Start with a hypothesis
Make sure to start with a clear hypothesis before you start A/B testing your emails.
Choose an email metric that you want to analyze and improve upon. This way, you’re clearly establishing “what” to test and “why” you’re testing it.
For example, “Add an emoji in the subject line to test if the open rate will increase.”
What: Testing the subject line by adding an emoji
Why: To increase open rates
Here's a chart of all the email metrics you can A/B test:
Source: Rejoiner
2. Only test one variable at a time
When you test, make sure to focus on one variable at a time. That way, if you see your metrics improving, you'll know exactly what made the difference.
For example, if you change both the design of your emails and the CTA copy, you won't be able to tell which of the changes had the biggest impact on your click-through rate and conversions. Stick to testing one variable at a time to get the clearest results.
3. Focus on frequently-sent emails
A/B testing and improving metrics for frequently-sent emails, such as onboarding emails, weekly recommendation emails, and weekly or monthly newsletters, can help create a strong connection with your audience and improve your conversions.
4. Use a significant sample size
At the end of the day, A/B testing is a statistical experiment. To get the most out of the experiment and draw meaningful conclusions, you need to have a significant number of subscribers to do the experiment.
5. Only use statistically significant results
It's not just about testing on a large enough sample size, but it's also about getting results that have statistical significance. There is no point in using results that are not significant enough to move the needle.
For example, if you sent your A/B test email to 2,000 people and 500 of them opened Version A and 490 opened the Version B, you can't just assume Version A won. To understand the kind of results that are statistically significant, you can use a free calculator like Visual Website Optimizer’s.
6. Make sure to split your list randomly
When setting up an A/B test, it's important to ensure you split your list randomly. Most email marketing tools have this feature built-in, so make sure to use it.
Also avoid splitting the list for two different segments of users. For example, if you're testing emails related to paid customers, make sure to only include them in the test, not customers who haven't paid yet or are free users. This will help you get the most accurate results.
7. Wait for enough time
Not everyone will open the email right away, even if you’re sending it to a large group. That’s why it’s important to allow enough time for your readers to engage with your emails. We recommend waiting at least 4-5 days before drawing any conclusions from your A/B test.
8. Prioritize A/B tests
When you're A/B testing your emails, you might get overwhelmed by all the metrics you can test. To prioritize the tests with the most impact, use frameworks like the “ICE Score” or “PIE Score” to help you understand which tests offer the greatest return for the least effort.
9. Use Email Marketing Tools
If you have a large email list, it is highly recommended to use tools like MailerLite or ActiveCampaign to set up A/B testing and gain useful insights from the results. With these tools, you can easily split lists, test different variables, and get valuable data that can help make your email campaigns more successful.
10. A/B test again
Keep up with your customers' behavior and your email list growth by A/B testing regularly. For example, if you've seen success with increased open rates by adding an emoji to your subject lines, why not try A/B testing to improve your email design and layout, or test out a different copy for your CTAs? Doing this will help make sure your email campaigns are optimized to bring in the most conversions.
Email A/B Testing: Conclusion
Start A/B testing your emails today!
A/B testing will enable you to quickly identify which metrics are the most impactful, and can help your business increase its email marketing return on investment (ROI). Also, by constantly learning from A/B testing your email campaigns, you'll be able to form strong relationships with your audience and significantly improve engagement.
Use the data from these tests to understand which design elements, copy, and call-to-actions are most effective in getting your readers to take action. By doing this, you can ensure that your emails are always optimized and that you're getting the best results.
So, what are you waiting for? Start A/B testing your emails today and get the most out of your email campaigns!
> Learn more about newsletter best practices to improve your email campaigns
The authors
Learn more about us
Our Methodology
This article has been written and researched following our EmailTooltester methodology.
Our Methodology