Concept Testing Benchmarks: Choose a Reference That Fits the Decision

Concept testing benchmarks compare a result with a reference, such as earlier concept tests, an existing offer tested alongside the new one or a business requirement. Choose the reference before interpreting whether a concept is “above average.” A product team deciding whether to develop a $24 desk organizer needs to know what that comparison contains before it treats the label as permission to invest.

A norm can put a score in context. It doesn't answer every question in the development decision. A high percentile doesn't show that buyers understand the offer, that its parts work or that the business can make money supplying it.

Three references answer different questions

A historical norm summarizes earlier research. It can show how a concept compares with a defined set of previous concepts measured in a compatible way. The reference may be a category database or your company's own archive. Its usefulness depends on what entered the archive, not just how large it is.

A concurrent benchmark is an existing or otherwise relevant offer included in the current study. It can provide a present-day comparison under matched audience, question and exposure conditions. It isn't automatically a forecast of what would happen if the new offer replaced it in market.

A development requirement states what the business needs before the next investment. It might concern comprehension, a meaningful improvement over the current offer or a product condition that must be met. It comes from the decision, rather than the average of a database.

Keep these separate in the report. “Higher than our old concept,” “above the category norm” and “ready for a prototype trial” are different conclusions. A benchmark can inform the third without deciding it alone.

The priced offer determines the match

Consider a hypothetical organizer: a freestanding desk tray measuring 30 × 20 cm, supplied with two removable dividers, sold online for $24 before tax with shipping included. It needs no attachment to the desk. The proposed buyers are US adults who bought desk-storage products in the past six months and help choose them. This is a planning example, with no research results or tested norm.

The team wants to choose whether to develop this complete offer. Its proposed primary measure is the share saying they definitely or probably would buy within four weeks at the displayed terms. It also needs unaided explanations of what is included and how the tray fits the intended workspace.

A database of unpriced rough ideas could contextualize early appeal, but it wouldn't supply a matched reference for this price-specific intention. Nor would a norm for employees evaluating software subscriptions answer the same buying question merely because both studies involve office work.

Ask for the closest relevant reference and record the remaining differences. If the only available norm measures general appeal, keep its conclusion about appeal. Don't relabel it purchase intention to obtain a benchmark.

A norm needs an inspectable definition

Zappi's norm documentation distinguishes country/category and customer references, describes norms for consistently asked measures and shows the number of tested assets in the reference. It also notes that proprietary customer assets aren't individually visible. Those details show why the buyer should request a method description and coverage information even when the underlying cards can't all be shared.

For the organizer, use this request before accepting the proposed benchmark:

The number of reference concepts and the number of people answering each are different facts. A database described as having many respondents may still contain few relevant concepts. Ask whether each concept contributes equally or whether results pool people across studies. Don't assume that an average concept score and an average across all respondents are the same calculation.

If the provider can't document a compatible reference, request a proposal for the study results without that benchmark claim. Price any additional comparison separately. The missing norm should change the scope, not become an unexplained endorsement in the report.

A percentile needs its own interpretation

A percentile describes position within a reference distribution. An 80th-percentile concept isn't an offer that 80% of buyers want, and the percentile isn't an 80% chance of commercial success. Ask what is being ranked, how ties are handled and which reference was selected.

Keep the original response measure beside the rank: its wording, answer categories, eligible base and result. If the report calls the comparison statistically significant, request the actual comparison method and uncertainty treatment. A favorable rank by itself doesn't provide that analysis.

The research record matters here. AAPOR's disclosure standards call for instruments, population and recruitment methods, sample sizes, weighting and limitations. They also require nonprobability precision measures to explain their model and assumptions. A norm label doesn't replace those facts for either the new study or the reference evidence.

A fixed reference and a current reference serve different purposes

For recurring work, decide whether the aim is comparison with a stable past baseline or with the current set of category concepts. A moving reference can change a relative position even when the underlying concept score doesn't change.

Zappi's fixed-norm documentation describes a stable reference distinct from changing organic norms. Neither type is automatically the right choice for every task. A fixed reference supports a consistent historical comparison; it may become less representative of current offers. A refreshed reference can improve relevance while changing what “above average” means.

Save the reference name, filters, version or date, asset count and calculation with the decision. If someone later applies a different norm, show it as a new analysis rather than quietly replacing the basis of the original recommendation.

Changes to your own instrument can also break continuity. Pew Research Center's questionnaire guidance explains why wording, mode and earlier questions matter when comparing results over time. An old intention question asked after a benefit explanation isn't a clean reference for a new unaided reaction.

No compatible norm is a design choice to resolve

If the organizer has no suitable historical reference, the team can consider testing a relevant current offer alongside it. Keep the actual offers complete, including their prices and terms, and plan the comparison explicitly. A concurrent control answers the question posed by those offers; it doesn't create a category-wide norm from one comparison.

Another option is to develop a consistent internal reference over time. Retain both promising and weak tests, identify eligibility and instrument changes and preserve the reason each concept entered the archive. An archive containing only ideas that survived an internal selection process describes that selected set, not every idea the category could have tested.

For the immediate decision, it may be better to use a defined absolute requirement and acknowledge that external positioning is unknown. Missing norm coverage doesn't mean the concept failed. Our concept-testing sample-size guide helps plan the human evidence needed for the comparison you actually choose.

The development gate still needs a reason

For the organizer, the decision record should state whether the complete offer is understood, which reference supports its relative performance and what the next investment would resolve. A misunderstanding about supplied dividers calls for a clearer card. A promising, understood offer may justify a product check. Neither conclusion settles physical fit, production cost or observed purchases.

The worked concept-test report shows why an appeal leader can still need revision. The concept-testing tools guide helps compare how a proposed supplier will return the evidence. Agree the benchmark and progression rule in that scope before results arrive.

Sapien's team can compare complete offers with simulated audiences and help interpret their reactions. Label the response source and agree which human comparison would support the intended use. A simulated score compared with a human norm doesn't by itself validate the simulation, and a larger generated audience doesn't supply a human-survey margin of error.

If your team has an “above norm” result but the development call is still unclear, discuss the concept with Sapien. Start with the tested card, the reference definition and the investment the score is supposed to support. That makes the comparison useful beyond its label.