Your Concept Test Has a Higher Score. Is It Enough to Pick a Winner?
The higher score can be a clear lead, a difference too small to matter, or an inconclusive result. Read the interval against the decision you face.
Use the size and precision of the lead to decide whether the higher-scoring concept earns the next commitment. Set the smallest difference that would justify that commitment before you see the scores. A point estimate by itself cannot tell you whether the lead is large enough.
A fictional beverage team must choose which pack to take into a limited run. It compares two concepts on the proportion of eligible buyers who select the assigned offer in a realistic purchase task. The team decides that Concept B needs a lead greater than five percentage points to justify its extra production cost. That is an illustrative business rule for this run.
Read the difference and its interval
Calculate the difference as B minus A on the agreed measure. Use an interval appropriate to the study design, accounting for features such as paired assignment, clustering, or weighting when they apply. Keep the group bases and exact outcome beside the estimate. For multiple variants or repeated looks, use the inference plan agreed before fieldwork.
These three hypothetical intervals illustrate a prespecified reporting rule. A formal equivalence claim calls for an equivalence procedure with the margins, confidence level, and analysis selected up front. Equivalence testing primer
Illustrative result | Decision under the stated rule |
|---|---|
B leads by nine points, with interval six to twelve. | The whole interval clears the five-point hurdle. Advance B if the other gates, including feasibility, pass. |
B leads by two points, with interval one to four. | The entire reported interval stays within the five-point practical margin. Choose using production cost or another agreed criterion. |
B leads by six points, with interval minus two to fourteen. | The interval spans a small disadvantage and a worthwhile lead. Retest if a sharper answer would change the commitment. |
Match the next step to what you still need to learn
If a result is inconclusive, ask whether another study can be precise enough around the hurdle and whether the answer would arrive before the decision. Set the new sample from that question. If a bigger test cannot change the decision in time, choose a smaller reversible commitment and name what will be checked during it. Sapien's concept testing sample-size guide
Keep any separate disqualifier visible. A concept may clear the choice hurdle and still create a misunderstanding that has to be fixed before a launch. The recommendation should name which evidence releases the next step and which conditions still need to be met. Sapien's concept-test results guide