Email Subject Line Testing: Pick Winners on Orders
Open rate is the wrong way to pick an email subject line test winner. In 38 tests, 22 had an open winner and only 5 a click winner. Judge tests on orders.

Updated October 2026.
Pick email subject line test winners on clicks and orders, not on open rate. Opens are inflated by Apple's Mail Privacy Protection, and a subject line that wins the open often loses the click. In 38 subject-line tests at a D2C home brand I ran, 22 found a clear winner on opens and only 5 found one on clicks. Read every test on orders per 1,000 emails delivered, against a holdout, and roll out only what moves orders.
I'm Raviraj Ramavarapu, founder of Compound Lab, a retention agency for D2C brands and consumer apps. Every retention plan I build carries a test in each journey, and every test is read on orders, not opens. This is why, and how to set it up in the tools most Indian brands use.
Why is open rate the wrong metric for subject line tests?
Because a large share of "opens" are no longer people opening the email. Apple's Mail Privacy Protection, launched in 2021, stops senders using invisible pixels to collect information about the reader. Apple Mail now downloads remote content in the background when a message arrives, so the tracking pixel can fire whether or not anyone reads the email.
Email platforms admit it. Klaviyo's own A/B testing guide warns that Mail Privacy Protection pushes reported opens up, and that those automatic opens get counted if you let open rate decide the test. It also points out that a significant result can't tell you whether people opened on purpose.
There's a second problem that has nothing to do with Apple. A curious subject line can win the open and lose the sale. People open to see what the fuss is about, find nothing they want, and leave. You've measured curiosity, not intent.
When I checked in October 2026, Google's own AI answer for this question still said to judge subject line tests by open rate. On a modern list, that advice will often pick the wrong winner.
What did 38 subject-line tests show?
Most open winners weren't click winners. At a D2C home brand I ran, the email team had run 38 subject-line tests. 22 of them produced a clear winner on open rate. Only 5 produced a clear winner on click rate.

If those tests had been rolled out on opens, 22 "winners" would have gone to the full list. At most 5 of them had any evidence of making more people click, let alone buy. The rest were changes that looked like progress in a report and did nothing for revenue.
Reading results too late is the same mistake in another form. That brand's WhatsApp click rate collapsed inside one week, and a monthly report blended it into a dull average. I go through those numbers in retention consultant vs AI.
Which metric should pick an A/B test winner?
The metric closest to the money that the test can actually move. For a subject line that's clicks first, then orders. For anything below the subject line, it's orders.
| What you're testing | Pick the winner on | Watch alongside |
|---|---|---|
| Subject line or preview text | Click rate, then orders per 1,000 delivered | Unsubscribes |
| Email body, layout or button | Click-to-order rate | Click rate |
| Offer or no offer | Orders per 1,000 delivered and margin per order | Discount share of orders |
| Send time | Click rate | Orders per 1,000 delivered |
| WhatsApp template | Click rate and replies | Blocks |
| A whole journey | Repeat orders against a holdout | Days to second order |
Orders per 1,000 delivered is the number I come back to most. It works across channels, it doesn't depend on tracking pixels, and a low-volume test can't hide behind a high percentage.

What does your CRM pick by default?
Usually opens or clicks, never revenue. Check the setting before you let any tool send a "winner" to the rest of your list.
| Platform | How it picks the winner | What to change |
|---|---|---|
| Klaviyo | Opens, clicks or orders placed, whichever you set | Choose placed order rate, or click rate on small lists |
| CleverTap | Clicks for most channels, views for email and WhatsApp, then sends the winner to the remaining audience automatically | Don't let it auto-send an email winner on views. Read clicks and orders before rolling out |
| MoEngage | Merlin AI shifts traffic towards the better click rate, or opens and clicks for email | Use click rate, and read conversions through a control group |
All three support a control group for measuring conversion lift. CleverTap's control groups compare conversions and revenue against users held out of campaigns. That's where the real answer lives. I explain how to set one up in global control groups.
How do you run a subject line test properly?
Change one thing, decide the winning metric before you send, and wait for the orders to arrive before you call it.
- Test one variable. Two subject lines with the same preview text, body and send time. If you change three things, you learn nothing.
- Write down what you expect. "A question will get more clicks than a statement" is a test. "Let's see which does better" is a guess.
- Pick the winning metric first. Click rate or orders, chosen before the send, so nobody picks whichever metric looks best afterwards.
- Keep a holdout. A small slice gets neither version, so you can see whether the email made any difference at all.
- Wait for the order window. Most orders from an email don't come in the first hour. Read the result once the bulk of orders from your usual sends have arrived, not when the opens peak.
- Log every result by type. Group subject lines, first lines and buttons by style, such as question, statement, number or name, and rank each style by orders over time. One test is an anecdote. Twenty tests ranked by type is a playbook.
- Roll out only order winners. An open win that doesn't move clicks or orders gets noted and left out.

What should D2C brands test besides subject lines?
The parts of the journey that decide whether people buy, not just whether they open. These are the tests I put into retention plans, each read on orders.
| Where | Version A | Version B | Read on |
|---|---|---|---|
| Exit-intent pop-up | First look at new arrivals | A small prepaid code | Sign-ups, then first orders |
| Welcome message | Founder story first | Products first | Clicks, first orders in 14 days |
| Cart reminder | Trust and help, no offer | Offer in the first message | Recovered carts and margin |
| Second order | Another colour of what they bought | The piece that completes it | Second orders by day 30 |
| Second-order timing | Day 7 | Day 14 | Days to second order |
| Subject line style | Statement | Question | Clicks, then orders |
| Send time | Each person's best hour | One fixed hour | Click rate |
The second-order tests matter most, because the second order is where repeat customers are won. I cover what to send in that window in repeat purchases without discounts, and how to measure the gap in retention metrics.
How do you know a test result is real?
When it holds against a holdout and holds again the next time you run it. A single test on a small list can swing on a handful of orders, so treat one win as a lead, not a rule.
Three checks before you change the default. Did the winner beat the holdout, not just the other variant? Did it win on the metric you chose before sending? Does it hold when you run the same idea on a different send? If the answer to all three is yes, roll it out and log it. If not, note it and test again.
Where I fit
Compound Lab works with D2C brands and consumer apps on everything after the first order, including the test plan, the CRM setup that reads it, and the weekly report that ranks what actually sells. If your test reports are full of open-rate winners and your revenue isn't moving, book a 30-minute call and we'll look at how your tests are being judged.