How Applying Simulated RCT to Ad Testing Can Save You Millions
Published by
Irrational Agency
Irrational Agency
A unique ad testing approach inspired by medical research may have saved two global clients a billion euros or more. Nobody knows for sure if John Wanamaker really said it, but it's one of the most famous quotes in advertising: "Half the money I spend on advertising is wasted; I just don't know which half."
The Simulated RCT
In the digital world, pay-per-click measurement sometimes answers that question. But the dilemma still remains for brand advertising, mass media, and non-digital below-the-line marketing. You can A/B test a website, but not a TV ad.
This challenge motivated two clients to approach us with an interesting question: How can we measure which ads, or which advertising media, are really working for us? Between them, these two companies spend €2 billion on marketing every year. So if they could really find out which half doesn't work: that's a lot of money no longer going to waste.
Irrational Agency brought a unique research approach to these clients: the Simulated RCT. You might have heard of the Randomized Control Trial or RCT. It's regarded as the "golden standard" of scientific research and is used in every clinical trial and many other scientific contexts. In an RCT, a group of subjects is divided randomly into two or more groups, and each one receives a particular medicine or other intervention. Crucially, the subjects do not know which group they are in and whether they got a real medicine or a placebo. Each group is observed over time, and the scientists can measure if more people in one group recover from the disease, or have other positive health outcomes.
Applying RCT to Ad Testing
We realized we could apply the same principle to testing ads and marketing channels. For the first client, a European insurance company, we showed the respondents 15 different marketing touchpoints – ranging from sponsorship of a major sports event to in-store promotions and customer service improvements. Each person saw 3 different touchpoints in different combinations, and/or a "decoy" message as a placebo. For the second client, an American consumer goods company, we showed three new TV ads. Each person saw one ad for the brand, plus two placebo ads so they did not know exactly what was being tested.
The second part of the test was just as important. After viewing the ads, we asked each respondent to make choices and tradeoffs between real products. For the insurance client, we provided a simulated online shop where participants could choose between different car or life insurance policies. For the consumer goods client, an innovative twist: we gave away a discount voucher and linked participants to an e-commerce site – where they could buy the real product, with their own money. Then we sit back and measure how many vouchers were redeemed for the group who saw ad #1, ad #2, and ad #3.
With the help of these two imaginative clients, we developed a unique ad test based on true consumer behavior. It gave our clients the robustness of A/B testing, extended into the non-digital world.
What We Found Out
Our insurance client found that two particular marketing channels were more powerful than any other at changing consumer choices, making respondents more likely to pick their branded products from a competitive lineup. And the consumer goods brand discovered that – for their key target segment – one particular ad drove real purchasing behavior more than any other. (And as a bonus, our System 3 tool told them why this ad performed best.)
For the first time, these clients were able to go to their boards with robust, real-world data showing one ad definitively winning the contest, or two marketing channels truly worth investing in. Thus allowing them to identify John Wanamaker's "wasted half" of the ad budget.
That half could be redirected into a more effective place – or they could reduce "marketing waste" without sacrificing growth, winning the CFO's approval and banking some trust for the next battle. Interested in trying out a Simulated RCT? Contact us and we'll be happy to talk through how it could work for you.