· 6 min read

Email Subject Line Testing: Pick Winners on Orders

Open rate is the wrong way to pick an email subject line test winner. In 38 tests, 22 had an open winner and only 5 a click winner. Judge tests on orders.

Email subject line testing: pick the winner on clicks and orders, not opens. In 38 tests, 22 had a winner on opens and only 5 on clicks.

Updated October 2026.

Pick email subject line test winners on clicks and orders, not on open rate. Opens are inflated by Apple's Mail Privacy Protection, and a subject line that wins the open often loses the click. In 38 subject-line tests at a D2C home brand I ran, 22 found a clear winner on opens and only 5 found one on clicks. Read every test on orders per 1,000 emails delivered, against a holdout, and roll out only what moves orders.

I'm Raviraj Ramavarapu, founder of Compound Lab, a retention agency for D2C brands and consumer apps. Every retention plan I build carries a test in each journey, and every test is read on orders, not opens. This is why, and how to set it up in the tools most Indian brands use.

Why is open rate the wrong metric for subject line tests?

Because a large share of "opens" are no longer people opening the email. Apple's Mail Privacy Protection, launched in 2021, stops senders using invisible pixels to collect information about the reader. Apple Mail now downloads remote content in the background when a message arrives, so the tracking pixel can fire whether or not anyone reads the email.

Email platforms admit it. Klaviyo's own A/B testing guide warns that Mail Privacy Protection pushes reported opens up, and that those automatic opens get counted if you let open rate decide the test. It also points out that a significant result can't tell you whether people opened on purpose.

There's a second problem that has nothing to do with Apple. A curious subject line can win the open and lose the sale. People open to see what the fuss is about, find nothing they want, and leave. You've measured curiosity, not intent.

When I checked in October 2026, Google's own AI answer for this question still said to judge subject line tests by open rate. On a modern list, that advice will often pick the wrong winner.

What did 38 subject-line tests show?

Most open winners weren't click winners. At a D2C home brand I ran, the email team had run 38 subject-line tests. 22 of them produced a clear winner on open rate. Only 5 produced a clear winner on click rate.

Email subject line testing at one D2C home brand. Of 38 subject-line tests, 22 found a clear winner on open rate, and only 5 found a clear winner on click rate.

If those tests had been rolled out on opens, 22 "winners" would have gone to the full list. At most 5 of them had any evidence of making more people click, let alone buy. The rest were changes that looked like progress in a report and did nothing for revenue.

Reading results too late is the same mistake in another form. That brand's WhatsApp click rate collapsed inside one week, and a monthly report blended it into a dull average. I go through those numbers in retention consultant vs AI.

Which metric should pick an A/B test winner?

The metric closest to the money that the test can actually move. For a subject line that's clicks first, then orders. For anything below the subject line, it's orders.

What you're testing Pick the winner on Watch alongside
Subject line or preview text Click rate, then orders per 1,000 delivered Unsubscribes
Email body, layout or button Click-to-order rate Click rate
Offer or no offer Orders per 1,000 delivered and margin per order Discount share of orders
Send time Click rate Orders per 1,000 delivered
WhatsApp template Click rate and replies Blocks
A whole journey Repeat orders against a holdout Days to second order

Orders per 1,000 delivered is the number I come back to most. It works across channels, it doesn't depend on tracking pixels, and a low-volume test can't hide behind a high percentage.

Which metric picks the A/B test winner. Subject line tests are judged on click rate, then orders per 1,000 delivered. Body and button tests on click-to-order rate. Offer tests on orders and margin. Send time on click rate. WhatsApp templates on clicks and replies. Whole journeys on repeat orders against a holdout.

What does your CRM pick by default?

Usually opens or clicks, never revenue. Check the setting before you let any tool send a "winner" to the rest of your list.

Platform How it picks the winner What to change
Klaviyo Opens, clicks or orders placed, whichever you set Choose placed order rate, or click rate on small lists
CleverTap Clicks for most channels, views for email and WhatsApp, then sends the winner to the remaining audience automatically Don't let it auto-send an email winner on views. Read clicks and orders before rolling out
MoEngage Merlin AI shifts traffic towards the better click rate, or opens and clicks for email Use click rate, and read conversions through a control group

All three support a control group for measuring conversion lift. CleverTap's control groups compare conversions and revenue against users held out of campaigns. That's where the real answer lives. I explain how to set one up in global control groups.

How do you run a subject line test properly?

Change one thing, decide the winning metric before you send, and wait for the orders to arrive before you call it.

  1. Test one variable. Two subject lines with the same preview text, body and send time. If you change three things, you learn nothing.
  2. Write down what you expect. "A question will get more clicks than a statement" is a test. "Let's see which does better" is a guess.
  3. Pick the winning metric first. Click rate or orders, chosen before the send, so nobody picks whichever metric looks best afterwards.
  4. Keep a holdout. A small slice gets neither version, so you can see whether the email made any difference at all.
  5. Wait for the order window. Most orders from an email don't come in the first hour. Read the result once the bulk of orders from your usual sends have arrived, not when the opens peak.
  6. Log every result by type. Group subject lines, first lines and buttons by style, such as question, statement, number or name, and rank each style by orders over time. One test is an anecdote. Twenty tests ranked by type is a playbook.
  7. Roll out only order winners. An open win that doesn't move clicks or orders gets noted and left out.

How to run a subject line test properly, in seven steps: test one variable, write down what you expect, pick the winning metric first, keep a holdout, wait for the order window, log every result by type, and roll out only order winners.

What should D2C brands test besides subject lines?

The parts of the journey that decide whether people buy, not just whether they open. These are the tests I put into retention plans, each read on orders.

Where Version A Version B Read on
Exit-intent pop-up First look at new arrivals A small prepaid code Sign-ups, then first orders
Welcome message Founder story first Products first Clicks, first orders in 14 days
Cart reminder Trust and help, no offer Offer in the first message Recovered carts and margin
Second order Another colour of what they bought The piece that completes it Second orders by day 30
Second-order timing Day 7 Day 14 Days to second order
Subject line style Statement Question Clicks, then orders
Send time Each person's best hour One fixed hour Click rate

The second-order tests matter most, because the second order is where repeat customers are won. I cover what to send in that window in repeat purchases without discounts, and how to measure the gap in retention metrics.

How do you know a test result is real?

When it holds against a holdout and holds again the next time you run it. A single test on a small list can swing on a handful of orders, so treat one win as a lead, not a rule.

Three checks before you change the default. Did the winner beat the holdout, not just the other variant? Did it win on the metric you chose before sending? Does it hold when you run the same idea on a different send? If the answer to all three is yes, roll it out and log it. If not, note it and test again.

Where I fit

Compound Lab works with D2C brands and consumer apps on everything after the first order, including the test plan, the CRM setup that reads it, and the weekly report that ranks what actually sells. If your test reports are full of open-rate winners and your revenue isn't moving, book a 30-minute call and we'll look at how your tests are being judged.

Next postRepeat Purchases Without Discounts: 10 Reasons to Return

Your customers already paid once. Make them pay again.

Book a Consultation