I won’t be standing on this soapbox for long, but while I’m here, I want to take a second to support something counterintuitive coming from someone with the title of senior paid search analyst: you should suspend paid search.
Not all at once. Not all at once. But with the right structure and parameters, suspending paid search can be an invaluable source of data for us PPC professionals.
With the right structure and parameters, suspending paid search can be an invaluable source of data for PPC professionals.
The question of incremental growth
When it comes to showing profitability, I would say that most of us measure some variation of total revenue over total cost. If that figure is high, then we’re doing well. If not, we rethink and refine until we reach our desired ROAS.
It’s not inherently bad. It’s a quick measure of performance, and I believe I do the same thing. The problem is that this number doesn’t tell us much. It doesn’t tell us whether we lost money on buying users who would have gone to us anyway. And it doesn’t tell us what might have happened if we simply hadn’t spent the money.
Admittedly, it’s usually the branded spending that gets the job done here. We, as an industry, intuitively feel that branded search doesn’t bring in many new users. But we should take the same comb for non-branded search, which can bring in far less revenue than we might assume.
To really answer these questions, we need to run an A/B incremental test.
How to structure an incremental test
For starters, I’m sure many of us know what an A/B test is, but let’s do an overview.
An A/B test is an experiment in which we take two groups, a control group and a test group, and measure the change in behavior or outcome resulting from a change in a variable.
In any good test, it is important to define the parameters of the test. For our purposes, we should try to think through the following questions:
- How long will you run this test?
- What are we comparing and what are we measuring?
- What method will you use to analyze the results?
- What attribution model will you use to measure the impact of paid search?
The answer to the first question is up to you. We recommend running the test over at least a few months to account for possible seasonal variations. The second question is also quite simple. The purpose of this experiment is to measure two things:
- When we suspend spending on branded ads, how much does our organic ad traffic change?
- When we suspend spending on paid search (branded or unbranded), how does that affect our core revenue or number of leads?
When it comes to the method of analysis, there’s a bit more nuance, but we’ll cover our options in a few paragraphs. As for the attribution model, it’s important to keep it consistent across all tests. Pick one and stick to it.
Once you have it all mapped out, there are two important things to do before you start testing.
1. Create a test group and a control group
To measure the impact of paid search, the variable we change is cost. To measure effectively, we need two groups: a test group and a control group. Instead of sorting our groups at the user level, we can create these groups using geographic data.
The methodology outlined in the article by Tadelis et al. offers a great example of how this can be done using Nielsen DMA regions to make sure that our two groups are made up of regions with similar sales and seasonal trends.
Start by selecting a subset of geographic regions to test. The article uses a subset of 30%, but that number is up to you and your attitude about the risks associated with suspending paid search.
We want to measure the impact of our spending on core revenue. To do this, it’s doubly important to consider seasonal fluctuations in performance across geographic regions. There are some quick and dirty ways to get an idea of seasonality in your data. There are also SEO seasonality tools that can be adjusted for PPC purposes. If you’re curious, feel free to explore some of the more technical but still understandable methods of seasonality analysis.
Next, within your subset of geographic regions, sort and pair them based on sales and seasonality data. Split this list down the middle so that both groups look about the same and are easy to compare. These are your two groups.
2. Double, triple and quadruple check measurements.
This is very important. If you can’t trust the measurements on your site, all of your results here, and indeed all of your digital efforts, will be marked with an asterisk in a cloud of misbehaving tags and skewed revenue figures.
Before you start any testing, we recommend doing a tracking audit of your site. This will avoid bad results and make sure you have a clear understanding of how ad traffic, engagement, and revenue are measured.
3. Start testing.
Once you’ve divided up the groups and are confident in the accuracy of your measurements, you can begin testing. Deploy the pause in the areas you specified and start collecting data. Pay special attention to.
- Brand ad spending.
- Clicks and impressions of the brand’s ads
- The brand’s organic traffic
- Total number of conversions on the site
4. analysis of the results
Once the test is complete, we move on to the most interesting part. The structure of this test allows us to use the “Difference in Difference” (D-I-D) test, which compares the impact of changes between two groups to evaluate their effects.
While I won’t detail the specifics of the D-I-D test – and why every marketer should use it more often – in this article I’ll provide some interesting resources to help you with it. Below are a few articles that look at examples of the process.
- D-I-D Overview.
- An Example of D-I-D in Marketing
- A Walkthrough Of D-I-D
I’ve also created a Google Colab notebook that you can use to do this analysis yourself. It contains instructions and additional information about the test results. Find this notepad and make a copy here.
You can then follow the instructions in the document and you’re all set!
Pause the paid search for the sake of science
Much of the work of building a paid search program focuses on strategy. We study, predict, plan, launch, and test new tactics. But the broader question remains: once we’ve launched our program, how can we show that the results actually help the customers we serve?
Intentionally suspending paid search to evaluate its impact is, in my opinion, a vital part of any interaction. It’s validation, another test we have to do along with regular search query reports. It keeps us mindful of incrementality and ensures that we’re using budgets and efforts as efficiently as possible.
That said, it’s important to note that the tests and analysis listed here are not the only ones to make sense of this issue. They are cursory, and there are many variables that these models don’t account for, as discussed in the Google Colab notebook. But hopefully, this approach offers a good first step toward incorporating deeper, more structured testing into our PPC strategies.
