Copywriting & Campaign Strategy·Updated 2025-07-01·9 min read

How to A/B Test Cold Email Copy

Complete guide to A/B testing cold email copy — what to test, how to test it, sample sizes, and how to interpret results for continuous improvement.

Quick Answer

A/B test one variable at a time: subject lines first (highest impact), then opening lines, then value propositions, then CTAs. Send 100+ emails per variant for statistical significance. Test for at least 5 days.

What to Test (In Priority Order)
The Rules of A/B Testing
Subject Line Testing
Opening Line Testing
Value Proposition Testing
CTA Testing
Sample Size and Statistical Significance
Documenting and Learning from Results

Introduction

A/B testing is how you turn cold email from guesswork into science. Instead of debating whether a question or statement subject line works better, you test it and let the data decide. Yet most cold email senders either do not test or test incorrectly.

This guide explains what to test, how to test it, how much volume you need for statistical significance, and how to interpret results. It turns your cold email campaigns into a continuous improvement machine.

Who This Guide Is For

• Teams that want to systematically improve cold email performance
• Anyone who is guessing at what works instead of testing
• People who have tested before but are not sure if their results are valid

Key Takeaways

1. Test one variable at a time — changing multiple variables makes results meaningless
2. Test subject lines first — they have the highest impact on open rates
3. Send 100+ emails per variant for statistically significant results
4. Test for at least 5 days to account for day-of-week variations
5. Track reply rate as the primary metric (not just open rate)
6. Document results in a spreadsheet to build your knowledge base

What to Test (In Priority Order)

Not all variables are equal. Test in this order for maximum impact:

Priority	Variable	Impact	Test Duration
1	Subject lines	Highest — determines opens	3–5 days
2	Opening lines	High — determines if they keep reading	3–5 days
3	Value propositions	High — determines if they care	5–7 days
4	CTAs	Medium — determines if they respond	3–5 days
5	Email length	Medium — affects mobile readability	5–7 days

💡 Tip

Start with subject lines — they have the highest impact and are easiest to test. Once subject lines are optimized, move to opening lines, then value propositions, then CTAs.

The Rules of A/B Testing

A/B testing only works if you follow the rules. Breaking them produces meaningless results:

Test one variable at a time — changing both subject line and CTA makes results meaningless
Keep everything else constant — same time of day, same audience segment, same sending tool
Send 100+ emails per variant — smaller samples produce unreliable results
Test for at least 5 days — day-of-week variations can skew results
Track the right metric — reply rate for most tests, open rate for subject lines only

⚠️ Warning

The biggest mistake in A/B testing is changing multiple variables at once. If you change the subject line AND the CTA, you will not know which change caused the result. Test one thing at a time.

Subject Line Testing

Subject lines are the highest-impact variable to test. Here is how to test them properly:

Test	Variant A	Variant B	Metric
Question vs. statement	Quick question about [topic]?	[Company] + [topic]	Open rate
Personalized vs. generic	[First Name], quick question	Quick question about your outreach	Open rate
Short vs. long	3 words	7 words	Open rate
Curiosity vs. clarity	Thought of you when I saw this	Introduction from [your name]	Open rate

📝 Example

Test: Send 100 emails with 'Quick question about [Company]' and 100 emails with 'Noticed [Company] just launched [product]'. Compare open rates after 5 days.

Opening Line Testing

Opening lines determine whether the reader continues. Test different frameworks:

Test	Variant A	Variant B	Metric
Observation vs. industry	Saw your post about [topic]	Most [role] struggle with [problem]	Reply rate
Specific vs. general	Your post on [specific detail]	Congrats on the growth	Reply rate
Connection vs. research	[Mutual connection] suggested	Saw your [specific achievement]	Reply rate

💡 Tip

Opening line tests require the rest of the email to be identical. Only change the first sentence. This isolates the impact of the opening line.

Value Proposition Testing

Value proposition tests reveal what your audience actually cares about:

Test	Variant A	Variant B	Metric
Outcome vs. social proof	Save 10 hours/week	Acme Corp saved 10 hours/week	Reply rate
Pain vs. gain	Stop losing [metric]	Increase [metric] by X%	Reply rate
Specific vs. broad	Cut meeting prep 60%	Improve sales efficiency	Reply rate

💡 Tip

Value proposition tests often reveal surprising results. Test with your audience — do not assume what works for others works for you.

CTA Testing

CTA tests reveal the right level of friction for your audience:

Test	Variant A	Variant B	Metric
Question vs. yes/no	Would this be useful?	Are you open to exploring this?	Reply rate
Soft vs. hard meeting	10-minute chat?	30-minute demo?	Reply rate
Specific vs. vague	Worth a quick chat this week?	Let me know what you think	Reply rate

💡 Tip

CTA tests often show that lower-friction asks get more replies but lower-quality replies. Track not just reply rate but also meeting conversion rate.

Sample Size and Statistical Significance

How many emails do you need for reliable results? Here are the guidelines:

Sample Size Per Variant	Confidence Level	When to Use
50–100	Low (directional)	Quick tests, low-stakes decisions
100–200	Medium	Most A/B tests, standard practice
200–500	High	High-stakes decisions, major changes
500+	Very high	Enterprise-level testing

💡 Tip

For most cold email A/B tests, 100 emails per variant is sufficient. If the results are clear (10%+ difference in reply rate), you can act on them. If results are close, test longer with more volume.

Documenting and Learning from Results

A/B testing is only valuable if you document and learn from results. Here is how to build your knowledge base:

Create a spreadsheet with columns: Test name, Variable, Variant A, Variant B, Volume, Result, Winner, Notes
Record every test, even inconclusive ones
Review results monthly to identify patterns
Apply winning patterns to new campaigns
Share learnings with your team

📝 Example

Spreadsheet example: Test 'Subject line question vs. statement', 100 emails each, Question variant got 52% open rate vs. Statement 44%, Question wins, apply to all future campaigns.

Best Practices

✓ Test one variable at a time
✓ Start with subject lines (highest impact)
✓ Send 100+ emails per variant
✓ Test for at least 5 days
✓ Track reply rate as primary metric
✓ Document all results in a spreadsheet

Mistakes to Avoid

✗ Testing multiple variables at once
✗ Too small sample size (under 50 per variant)
✗ Testing for too short (1–2 days)
✗ Not documenting results
✗ Only tracking open rate, not reply rate
✗ Not applying winning patterns to future campaigns

Expert Tips

★ The best A/B testers are the ones who test the most — volume produces learning
★ Inconclusive results are still valuable — they tell you the variable does not matter
★ Test with your audience — what works for others may not work for you
★ Build a knowledge base over time — patterns emerge after 10+ tests
★ If results are close (under 5% difference), test longer with more volume

A/B Testing Checklist

☐ One variable being tested
☐ 100+ emails per variant
☐ Testing for at least 5 days
☐ Everything else kept constant
☐ Reply rate tracked as primary metric
☐ Results documented in spreadsheet
☐ Winning pattern applied to future campaigns

Summary

A/B testing turns cold email from guesswork into science. Test one variable at a time, start with subject lines (highest impact), send 100+ emails per variant, and document all results.

Build a knowledge base over time. Patterns emerge after 10+ tests. Apply winning patterns to new campaigns and share learnings with your team.

For more on what to test, see our Copywriting Mistakes Guide. For subject lines, see our Subject Line Guide. For value propositions, see our Value Propositions Guide.

Frequently Asked Questions

How many A/B tests should I run per month?

Run 2–4 tests per month. Each test takes 3–5 days, so you can run about one test per week. Focus on the highest-impact variables first.

What if my A/B test results are inconclusive?

Inconclusive results (under 5% difference) mean the variable does not matter much. Move on to testing a different variable. Do not waste time on low-impact differences.

Can I test subject lines and opening lines at the same time?

No. Test one variable at a time. If you test both, you will not know which change caused the result. Test subject lines first, then opening lines.

How do I know if my results are statistically significant?

With 100+ emails per variant and a 10%+ difference in reply rate, results are generally reliable. For smaller differences, test longer with more volume.