What to Do When Synthetic and Human Tests Pick Different Winners
Before you choose between conflicting readouts, check whether they answered the same question. Then scope the smallest human retest that can change the decision.
When a simulated audience and a recruited human sample favor different concepts, put both studies on the same page before you pick a winner. Start with the audience, what each group saw and what the reported metric actually measures. If the difference survives that check and the decision is costly, use a focused human test that matches the choice buyers will face.
Consider a hypothetical ready-to-drink coffee launch. The simulated screen favors a smaller can positioned for the afternoon. The human survey favors a larger morning can. The launch team can afford one packaging run. The conflict sounds decisive until you inspect both readouts. The simulated screen asked which option fit a stated occasion. The human test presented both prices and asked which product people would buy on their usual shop. The two teams can be reporting their respective results accurately.
Reconcile the studies before reconciling the scores
A choice between concepts changes when the audience, stimulus or task changes. Ask for the original stimuli and exact questions alongside both summaries. Put the studies in adjacent columns and flag every mismatch that could change the recommendation.
Check | What to compare and what to do |
|---|---|
Audience | Compare target definitions, category use, occasions, exclusions, and weighting. Recheck the decision audience if either study represents a broader pool. |
Stimulus | Match pack size, label, price, placement, competitor set, and offer terms. Restore the version that shoppers would actually see. |
Task | Compare unaided understanding, occasion fit, stated choice, and behavior as separate outcomes. Agree which outcome decides the next commitment. |
Precision | Get the human group bases and an interval appropriate for the design. Review variation across simulation runs separately and label its source. |
Meaning | Record the proposed mechanism and what would change your mind. Check whether each conclusion survives reasonable treatment of uncertain or missing answers. |
Simulated responses can help identify language or trade-offs to probe. Re-running a model can show whether its output changes under that run configuration. It does not estimate sampling error among recruited humans. For how to set up a comparison in advance, see Sapien's guide to blind validation pilots.
Retest the disagreement that matters
For the coffee launch, recruit current buyers who shop the intended channel and are eligible to buy either size. Assign one offer per participant at first exposure, using the complete pack and price that would actually be offered. Hold the buying context constant. Collect purchase choice against the same realistic alternatives and ask people why they chose, including why they would choose an alternative or leave without buying.
If morning and afternoon buyers would receive different distribution or media, plan those groups before fieldwork. If both would face the same launch, keep the overall target and agreed mix as the primary read. Ask for enough human responses to distinguish the smallest difference that would change the packaging order. A small or imprecise split can leave the team with two viable options. Concept testing sample-size guide
Write down what the next decision requires
If the retest clears the decision threshold, advance that offer into the next planned human or market check. If fit favors one can and choice favors the other, keep both observations and state which one controls the order. If the evidence remains inconclusive, change the commitment. A smaller store pilot or reversible print run may be a better decision than commissioning another broad study.
Close the readout with the difference you found, the check that resolved it, and the commitment the evidence can support. Keep the unresolved mechanism visible so it can be tested later.
