Did the Campaign Lift Brand Awareness? How to Design a Geographic Holdout
Measure awareness in markets with and without the incremental campaign. Plan the markets, survey, and decision rule before buying the media.
To estimate brand awareness lift without person-level ad exposure, vary the campaign by market and survey the same kind of buyer in the markets you compare. If you can randomize comparable markets to receive or withhold the extra campaign, the market assignment gives you a clearer way to separate campaign effects from shifts that would have happened anyway.
Imagine a fictional meal-kit brand adding connected TV in selected metros. All metros continue their normal search campaigns. The decision is whether to expand the extra TV buy. The test estimates the effect of adding that buy under those conditions.
Choose markets the media team can keep separate
Start with eligible markets where the brand sells the same offer and can vary the incremental media. Look at baseline awareness, trend, customer mix, distribution, and unusual local promotions. Form comparable pairs or groups using pre-campaign information, then randomize within them where feasible. Ask the media team to confirm its targeting and report actual delivery by geography. Google's geo-experiment research
Watch for broadcast overlap, commuting, national placements, and campaign sharing that could expose the holdouts. Decide what spillover would trigger a review before the buy. Specify the smallest lift that could justify expansion, then plan how many independent markets and survey completes the study needs. More people in two markets cannot substitute for independent markets when the treatment is assigned geographically.
Keep the survey comparable
Take a baseline and a follow-up in both groups. Use the same target population, recruitment rules, survey mode, and field windows. Ask an unaided question before showing a list of brands. Keep aided awareness as a separate secondary outcome. Collect survey geography consistently and keep recent category use visible in the readout. Sapien's brand awareness research guide
Suppose treated markets rise from 20% to 26% while holdouts rise from 19% to 22%. The simple change-in-changes is three percentage points. Those are illustrative numbers. A real readout needs market-level estimates and uncertainty suited to the assignment and sampling design. It should also say whether the agreed campaign was delivered.
Use this before approving the media plan
Item | Record |
|---|---|
Incremental treatment | Extra connected TV, where it runs, dates, and what media stays in both groups. |
Assignment and leakage | Eligible markets, matching inputs, randomization, and the delivery report used to detect spillover. |
Measurement | Same audience and unaided awareness wording in both windows. Define the scoring rules and primary outcome. |
Decision | Minimum lift to expand, uncertainty method, and what to do if delivery or survey precision falls short. |
If the result clears the prespecified decision rule and delivery matches the plan, consider extending the tested buy. If the plausible effect still spans the hurdle, retain the uncertainty in the decision memo. A test with heavy spillover or disrupted delivery can point to a better design for the next campaign.
