Concept Testing Results: Turn the Report into a Go, Revise or Stop Decision
A completed concept-test report shows why the appeal winner needs revision, with response counts, comprehension checks, realistic alternatives and a clear next investment.
The concept with the highest appeal score doesn't automatically deserve the next investment. If people are enthusiastic about a product you aren't offering, its lead gives you a presentation problem to solve before a development recommendation.
A concept-testing report should make that distinction visible on its first page. It needs a recommendation, the findings that support it and the conditions that would change the call. A chart showing which bar is tallest leaves the hardest work to the meeting.
The completed report below follows a household-gardening team choosing between two $24 herb kits for apartment residents. The offers, counts and business conditions are fictional teaching inputs; no human study or Sapien simulation was run.
The executive recommendation
Decision: advance the established-plant kit to a small product check; revise the seed-kit card before comparing demand again. Neither offer is ready for launch.
The seed kit leads on positive appeal, 72 of 120 answers versus 60 of 120 for the established-plant kit. But 30 of its 120 first explanations describe receiving established plants, which the kit doesn't include. Twenty-four of those 30 answers are positive on appeal. A third of the seed kit's positive appeal comes from people imagining a materially different offer.
The established-plant kit communicates its contents more clearly and receives more stated selections against the alternatives shown. That supports investigating the actual product. It doesn't establish a reliable market winner, future sales or viable fulfillment.
Route and call | Work permitted now | Condition before a larger commitment |
|---|---|---|
A-v1 seed kit: revise | Prepare a clearer card and retest it | Buyers must understand that they receive seeds and will wait for growth |
B-v1 established plants: go to product check | Check the plants, tray, collection process and care instructions | Product performance and costs must support a paid trial |
Either offer: launch remains open | No production commitment from these results | Observed buying, growing experience and viable unit economics |
This is the page an executive can use without reading every response. The rest of the report lets the research and product teams challenge its reasoning.
The offer and evidence record
The intended buyers are US apartment residents within the proposed pickup area who have a suitable sunny growing space, help choose household herbs and bought fresh culinary herbs in the past four weeks. Both concepts are offered at $24 before tax through the same proposed local pickup point. Neither includes recurring deliveries or a promise about harvest size.
A-v1 contains seeds for three herbs, growing medium, three small pots, a reusable tray and instructions. Its headline says “Fresh herbs ready for dinner,” with a picture of mature plants. The contents list names seeds but doesn't explain the wait for growth.
B-v1 contains three established herb plants, three pots, the same reusable tray and care instructions. Its card shows the plants supplied. The comparison changes the product's starting state and the work left to the buyer. It doesn't isolate a headline effect.
The report's study record is:
Field | Completed entry |
|---|---|
Response source | Illustrative human online survey design using an opt-in panel; unweighted |
Screening | 300 screened; 260 eligible; 40 ineligible |
Assignment | Random allocation: 130 assigned to A-v1 and 130 to B-v1 |
Exposure and completion | 10 in each group leave before seeing a card; 120 in each group see their assigned card and complete all three measures |
Comparison | Independent groups; one concept per person; matching layout and questions |
Answer bases | 120 per concept for comprehension, appeal and choice; no item missingness |
Result scope | Descriptive responses within these groups and shown conditions; no population margin of error supplied |
The 20 people who leave before exposure aren't negative answers. The report keeps them in the disposition record and outside the answering bases. Our monadic-testing guide explains why independent groups and repeated evaluations need different analyses.
A real report should also identify the panel supplier, field dates, screening instrument, actual cards, question order, quality checks and exclusions. AAPOR's disclosure guidance calls for this information so readers can inspect how findings were produced, including the sample source, instruments, weighting and limitations. AAPOR Disclosure Standards.
Three measures, three different answers
Comprehension comes first: “What would you receive, and what would you need to do before using the herbs?” The coding key distinguishes the correct starting state, a partial explanation, an explicit wrong starting state and an answer that can't be classified. The original answers stay beside their codes.
First explanation | A-v1 seed kit | B-v1 established plants |
|---|---|---|
Correct starting state and buyer task | 70/120, 58.3% | 100/120, 83.3% |
Partial account without an explicit contradiction | 10/120, 8.3% | 8/120, 6.7% |
Materially wrong starting state | 30/120, 25.0% | 4/120, 3.3% |
Can't classify | 10/120, 8.3% | 8/120, 6.7% |
For A, “I get three plants and can start picking herbs” is materially wrong. “Pots for growing herbs” is partial because it doesn't reveal whether the person expects seeds or plants. Counting both as correct would conceal the problem.
Appeal is asked next, without correcting misunderstandings: “How appealing is this kit at the price and terms shown?” Positive appeal means extremely or very appealing, defined before reviewing answers.
Appeal response | A-v1 | B-v1 |
|---|---|---|
Extremely appealing | 20 | 15 |
Very appealing | 52 | 45 |
Moderately appealing | 28 | 35 |
Slightly appealing | 15 | 20 |
Not at all appealing | 5 | 5 |
Positive appeal | 72/120, 60% | 60/120, 50% |
The choice question then asks which route the person would select for their next herb purchase: the shown kit at $24, their usual herb option or no purchase. Each group sees only its assigned kit. This is a stated choice against those alternatives, rather than an A-versus-B vote.
Stated next-purchase choice | A-v1 group | B-v1 group |
|---|---|---|
Shown kit | 42/120, 35% | 48/120, 40% |
Usual herb option | 54/120, 45% | 48/120, 40% |
No purchase | 24/120, 20% | 24/120, 20% |
A report that removes “no purchase” would report 42/96 = 43.8% for A and 48/96 = 50% for B. Those are shares among people selecting either purchase route, a different denominator. The primary readout retains all 120 eligible answers, including people declining a purchase.
The response options define the comparison. A kit's 40% share here says nothing about its share against every possible nursery, supermarket, online seller or home-growing alternative. Pew Research Center explains how wording, offered responses and earlier questions can change survey answers. Keep those conditions beside the result. Pew Research Center: Writing Survey Questions.
Our concept-testing questionnaire provides a fuller instrument for collecting understanding, relevance, intentions and alternatives separately.
Why the apparent appeal winner needs revision
Linked answers expose A-v1's central problem. Of its 72 positive appeal answers, 24 come from the 30 people who explicitly expect established plants. The calculation is 24/72 = 33.3%, not 24/120. It describes the composition of the positive group.
The remaining 48 positive answers come from 35 correct explanations, seven partial explanations and six unclassifiable explanations. Removing the 30 materially mistaken answers would produce 48/90 = 53.3% positive appeal. That filtered figure still doesn't tell us how the original audience would rate a clear seed offer: it removes a selected group rather than showing them the corrected concept.
The practical repair is specific. A-v2 should show the seed packets and included equipment, replace the headline with “Grow three herbs from seed,” and state a credible growth timeline that the product team has checked. Price, pickup terms, contents and buyer eligibility stay fixed. The revised card needs fresh first reactions; explaining it to the original group and asking again would answer a different question.
B's five-point stated-choice lead is descriptive. This exercise supplies no validated precision model for its opt-in sample, so the report doesn't label that lead statistically established. AAPOR requires nonprobability precision measures to explain their model and assumptions. More answers alone wouldn't repair A's misleading presentation. AAPOR Disclosure Standards.
The follow-up has a decision attached
The research owner prepares A-v2 and a matched B card for a fresh independent comparison. Before collection, the team records its comprehension rule, the choice difference worth acting on, sample plan and treatment of inconclusive results. The rule must address the specific risk of buyers expecting harvest-ready plants from the seed kit; a favorable appeal score doesn't override that requirement.
The product owner checks whether B's supplied plants and tray survive the proposed pickup and home-growing conditions. The finance owner obtains actual costs. For this exercise, the business requires at least $9.60 per kit before acquisition costs and overhead, equal to 40% of the $24 pretax price. That leaves at most $14.40 for the costs included in that contribution calculation. The report must list those costs and account for losses and refunds consistently.
If B can't meet that requirement, stop this version or change its economics before a paid trial. If A-v2 is understood but still gives intended buyers little reason to choose it, reconsider the seed offer rather than repeatedly rewriting its explanation. Missing cost quotes or unresolved product checks mean the decision pauses; they don't establish rejection.
Sapien's team can run concept comparisons with simulated audiences and help interpret the findings. For that work, the report should name the simulated response source and retain the audience setup and study conditions. Our comparisons with human research let you inspect the relevant evidence; generated response counts don't become a human sample. Actual growing performance, paid purchases and fulfillment costs need their own evidence.
If your report has an appeal winner but no defensible development call, discuss the concept comparison with Sapien. The useful brief is the unresolved choice: which offer needs a clearer explanation, which merits a product check, and what result would stop the next investment.
