Marketers can no longer rely on a single source of truth when it comes to marketing attribution. The majority of users opt out of tracking since Apple’s iOS14 release, punching a big hole in the accuracy of digital tracking that every ecommerce business has felt the effects of. In tandem, as DTC brands mature and expand into offline channels, as Harry’s did while I built the marketing science team there, it becomes impossible to rely on digital tracking alone.
Depending on who’s counting there are 20 attribution methods available, each with their own strengths and weaknesses. While most of us have had time to come to terms with this new reality, and are exploring multiple solutions, not much has been written about how to use multiple methods together to triangulate the ‘truth’ about what’s driving performance. What’s more, the pursuit of foolproof deterministic methods has left much of a brand’s competitive advantage on the cutting room floor; one model to rule them all surely points to a truth, but whose?
At Recast, the media attribution startup I co-founded after leaving Harry’s, we use Marketing Mix Modeling (MMM) to determine the impact each media channel has on performance. As well as Harry’s, we’ve built models for modern consumer brands like MasterClass, Mockingbird and Away. Google and Facebook have also gotten into the game, releasing their open source MMM solutions.
Historically MMM has primarily been used by larger brands, due to the cost and complexity of building a model, and the assumption you need lots of data to do this type of analysis. Contrary to common belief, MMM can work just as well if not better for ‘small’ brands (<$50k spend per month), so long as you take advantage of modern data pipelines, machine learning algorithms and data transformation techniques to automate the process, protect against implausible results, and remove human bias.
In the example we cover in this post, you can use the result of a checkout survey (as administered by a tool like Fairing) to calibrate your model. This acts to make the model and checkout survey consistent with each other but closer to the ‘truth’ about what’s driving performance. You end up with results that are better than any one method arrives at individually.
Do you believe what Facebook / Google / TikTok are telling you?
Put yourself in the shoes of a marketer or ecommerce store owner advertising across Facebook, Google and TikTok. Do you trust these ad platforms to grade their own homework? And even if you trust their intentions, do you believe they’re capable of telling a full story? Here’s the story they’re telling you:
|
Channel |
CPA |
|---|---|
|
|
$50.00 |
|
TikTok |
$55.00 |
|
Google Brand |
$4.50 |
|
Google Non-Brand |
$30.00 |
If utilizing a last-click attribution model, most of your sales from Google come through your brand campaign, i.e. people searching for your brand term. And yet, if customers are searching your brand name, then clearly they heard about you somewhere else – attribution credit belongs to another channel! On the other end of the funnel, you have Facebook and TikTok video ads driving hundreds of thousands of views and lots of engagement, but nowhere near as many sales as you expected. Do you turn TikTok off or cut spending on Facebook ads? As a savvy marketer you decide to go for a second opinion, and add a post-purchase survey that asks “How did you hear about us” using Fairing. Here’s what the survey tells you:
The results confirm your suspicions: many people say they heard about you from TikTok despite UTM code-based analytics data saying they came via “direct / none”, “google / cpc” or “google / organic”. A colleague suggests the survey might be biased towards more ‘memorable’ experiences like watching a video, and in reality it was the more functional Google ads that really sealed the deal.
Google isn’t separated out in the survey (we wanted to keep the survey simple so completion rates remained high), but it’s roughly inline with what Google reported for brand terms, which takes up most of the spend. Facebook is a somewhat muddled picture in that the survey results show a worse CPA than Facebook itself is reporting: potentially this is because you use more static ads and less prominent placements in this channel than TikTok.
Rather than being relieved the picture is only getting messier. Now do you believe analytics, your survey, or the ad platform reports? You need to solve this problem before setting your budgets for next quarter. What can you do?
Build an MMM in Excel with LINEST
You’ve heard about Marketing Mix Modeling as a potential solution to find out what channels are driving incremental sales. You were told it was possible to build a simple model in an afternoon, using the LINEST formula in Excel to handle the linear regression. After an hour or so gathering the data and following MMM tutorials, you arrive at the following model:
It seems to have captured your suspicion that Facebook and TikTok were driving results better than what was being tracked by the platform. This is likely the result of being unable to track users who opted out of tracking post iOS14. Facebook in particular has had a huge role reversal, it’s now performing better than TikTok. Google Brand is inflated versus what Google reports, but it’s still suspiciously low. Here’s what the model is telling you:
|
Channel |
CPA |
|---|---|
|
|
$34 |
|
TikTok |
$44 |
|
Google Brand |
$7 |
|
Google Non-Brand |
-$16 |
The main result that doesn’t make sense is Google Non-Brand: it says that the channel drove negative sales, i.e. one less sale for every $16 spent, which is unlikely. This is a relatively small channel spending just $5,300 total, which explains why the model didn’t have enough data to get this right. However, showing and trying to explain an implausible result like this to executives and decision-makers can seriously harm the credibility of our model. Furthermore this model now offers a third opinion rather than unifying the results we got from survey data, analytics and ad platform reports. Fortunately we can correct for this by calibrating our model.
Calibrate your MMM with Bayesian priors
We use a Bayesian model at Recast, which lets us incorporate what we’ve learned from other attribution methods and make the results of our model more plausible. For example we can tell the model that a marketing channel can’t drive negative sales, because that scenario is unlikely. Setting priors using the results of other attribution methods, like survey data, makes MMM the glue that sticks multiple attribution models together. This allows you to reconcile differences of opinion and make more accurate predictions to base budget allocation decisions on.
In our case we want to calibrate our model with the results of the survey. It’s important to note that all of our existing models are likely to be wrong. Facebook tends to underestimate the true number of conversions due to iOS14. Google is likely to be overestimating the impact of brand keywords. The survey overstates more ‘memorable’ channels like TikTok. The key is using the priors to set an intelligent starting point for the model to map and determine the most likely estimate for what true performance is.
We’ve put together an example script in R, to show how a Bayesian marketing mix model would work using our example data. It’s in a Google Colab notebook which lets you execute the code in a virtual environment rather than having to download and set everything up on your local computer, and it’s shareable like any Google doc.
For simplicity we’ve interpreted our survey data using upper and lower bounds, that we feed into the model. Approximately 30% of orders each came via Facebook and TikTok according to the survey, which translates to $61 and $41 CPA respectively. We suspect the survey is most generous to Facebook and TikTok being engaging video ad formats, so we set the lower bound (minimum CPA) at $10 for these channels to give the model some wiggle room.
Google Search is a problematic variable, as we want the model to correct for the spurious negative $16 CPA we got from our MMM. So we’ve set a $20 minimum for this channel as it’s likely to be more expensive than Facebook and TikTok based on what we know. We have a strong belief that Google Brand isn’t truly incremental, though might drive a few incremental conversions that otherwise would go to a competitor, so we put the minimum CPO at $100. We created upper bounds for each channel at $500 maximum because that’s high enough to be unbelievable based on what we know.
The model is then set up using Stan, a Bayesian modeling library, which forms the core of how we build our models at Recast. This toy example sets up a simple Bayesian model using the data and bounds we specified, in order to demonstrate how powerful Bayesian priors can be in joining multiple attribution methods together. There’s a lot more the tool can do (Recast models typically have thousands of parameters) but this should give you a good sense of its flexibility.
Don’t worry if you don’t understand all the code, it’s something you can pass to your data science team to build a custom MMM solution on top of (or it can be handled for you if you use [Recast]https://getrecast.com/demo-request/). The part we should be most interested in is the output: how does it compare to our simple model in GSheets?
In this model ‘mean’ is the equivalent of the CPAs in the GSheet model: how many dollars do we have to spend in that channel to get an order? Here’s what the Bayesian model says:
|
Channel |
CPA |
|---|---|
|
|
$35 |
|
TikTok |
$31 |
|
Google Brand |
$160 |
|
Google Non-Brand |
$39 |
In this model you can see the results are now far more plausible. We have incorporated what the survey told us about TikTok and Facebook, and our intuition about Google Brand being mostly non-incremental. Let’s see the model results side by side:
|
Channel |
Platform CPA |
Survey CPA |
GSheet CPA |
Bayesian CPA |
|---|---|---|---|---|
|
|
$40 |
$61 |
$34 |
$35 |
|
TikTok |
$55 |
$41 |
$44 |
$31 |
|
Google Brand |
$4.50 |
$9 |
$7 |
$160 |
|
Google Non-Brand |
$30 |
N/A |
-$16 |
$39 |
By incorporating the more generous survey measurement, we’ve shifted the CPA for TikTok and Facebook downwards from what the platforms are reporting, which we know suffers from undercounting due to users opting out of tracking. The CPA is right on the money for Facebook compared to our first model, but the new Bayesian model has revealed a far stronger result for TikTok: this is a competitive advantage as now we can double down our investment in this channel while competitors mistakenly think it’s under-performing! We’ve corrected for the ‘Google Brand Effect’ so are no longer treating brand campaigns as fully incremental. We also have a more reasonable estimate for Google Non-Brand, though the CPA will likely come down as we scale up spend and collect more data. It’ll also be possible to feed these results in as priors to the next model we build, so that what we tell management is consistent over time, but allows for performance improvements.
The end result is far more realistic and flexible than relying on ad platform reports, analytics data or survey results alone, and it’s all governed in one central model for the business. The model can be used to forecast future performance at different spend levels, and can be updated as performance changes through optimization. Management aren’t seeing any completely implausible results from our model, so will be less likely to question it. As more data comes in, uncertainty gets solved as the model continuously learns and improves. No model is right, but by incorporating everything we know about a business, we’re less likely to be wrong.
One thing you may have noticed is that setting the bounds on the CPO gives the analyst or data scientist a lot of power over the results of the model. If we wanted to tell a story that TikTok was really really effective, we could set the upper bound on the CPO for TikTok at $5 and the model will spit back an answer that includes the CPO of TikTok at $5. As they say: with great power comes great responsibility. When building a model this way we need to be careful not to impose our own unjustified biases on the model. If we just tell the model the answer we want to see, then we aren’t getting an unbiased additional perspective!
Michael Kaminsky is the Founder of Recast, eliminating wasted marketing spend and optimizing performance across all online and offline marketing channels by accurately measuring the true incrementality of marketing in real-time. Learn More
