Concept Testing Sample Size: Plan the Answers Your Decision Needs

Size a concept test per concept and per buyer group, using the difference you need to detect or the precision required for a score. There isn't one response count that fits every concept decision. A coffee team choosing between delivery offers might need a broad shortlist, a reliable ten-point comparison or a separate decision for occasional buyers; each buys a different study.

A quote for “400 responses” leaves that choice unresolved. Two independent concepts would receive 200 answers each. The same 400 evaluations could also come from 200 people seeing both concepts, with different implications for exposure and analysis. Set the question before accepting the count.

The decision behind the sample

Consider a hypothetical coffee-subscription team choosing an offer for a limited paid trial. A supplies one 200 g bag every 28 days for $15 per delivery. B supplies two 200 g bags every 56 days for $28. Both use the same medium-roast ground coffee, include standard shipping, exclude tax and allow pausing or cancellation at least three days before the next charge. These are planning inputs, not study results.

The audience is adults in the contiguous US who bought ground coffee in the past four weeks and help choose it for their household. Each eligible person sees one complete offer. Quantity, delivery interval and payment change together, so the comparison concerns the whole offer.

The primary measure is the share saying they definitely or probably would start the subscription within the next four weeks at the displayed terms. The team decides that a ten-percentage-point difference would matter when choosing the trial offer. It checks understanding separately so enthusiasm for a misunderstood delivery schedule doesn't become a reason to advance it.

Those decisions give the analyst something to size. “Find a winner” doesn't say how small a difference the study must distinguish.

A smaller difference costs more answers

The table shows three possible planning choices for one binary outcome in two independent, equally sized groups. It assumes 80% power, a two-sided significance level of 0.05, independent observations and a normal approximation without a continuity correction. The anticipated shares are assumptions, not predictions for the coffee offers.

We calculated these targets and checked the rounded counts using R's two-sample proportion power procedure. Its sample parameter is the number per group. The approximation isn't an exact guarantee for every achieved sample or survey design.

Power describes the chance of detecting the specified difference when the assumptions apply. At 200 valid answers per concept, the same calculation gives about 52% power for 50% versus 60%. That study could miss a ten-point difference nearly half the time under this scenario. It isn't a universal argument against 200; it shows what that budget can distinguish here.

The commercial choice is whether a five-point difference deserves a larger study or whether the team can treat near ties alike. Change that rule because the business can tolerate it, before collection. Don't choose a larger detectable difference afterward to make the existing budget look sufficient.

Precision for one score isn't power for a comparison

A buyer may instead need to estimate how many people understand each offer. That calls for a precision target for each concept, rather than a power calculation for their difference.

For one share near 50%, the familiar large-population, simple-random-sampling approximation needs 385 valid answers for a 95% interval with about five percentage points on either side. Splitting 385 people between two concepts doesn't give each concept that precision.

As a separate arithmetic illustration, 100 positive answers out of 200 would give a 95% Wilson interval of approximately 43.1% to 56.9% under its binomial assumptions. NIST describes the Wilson interval and why interval choice matters. That roughly seven-point half-width isn't a five-point target, and the uncertainty of a difference requires its own calculation.

Our sample-size worksheet supplies the formulas and copyable cells. Use it after deciding which concept measure and comparison must support the next investment.

The important buyer group can set the budget

Suppose occasional coffee buyers make up an expected 30% of the eligible audience and their A-versus-B result could change the trial plan. Define that frequency group in the screener before requesting quotes. A study with 388 valid answers per concept would contain only about 116 occasional buyers in each group if that share holds.

The overall count doesn't give that subgroup the original power. With 116 per concept, the same 50%-versus-60% assumptions give about 33% power. A descriptive subgroup chart may be useful, but it won't support the same detection claim as the full study.

If the team requires 388 occasional buyers per concept, natural recruitment at an assumed 30% share implies about 1,294 valid answers per concept, or 2,588 across both. That is an expected-count allowance, not a guarantee the quota will fill. An analyst may propose targeted recruitment or a separate subgroup study instead. Oversampling changes the composition; the overall estimate then needs an appropriate analysis and weighting plan.

Ask whether the subgroup really changes the action. If it doesn't, the extra fieldwork may add little. Also distinguish detecting A versus B within occasional buyers from proving that their response differs from regular buyers. The latter is a comparison of differences and needs its own plan.

The quote needs a recruitment funnel

The 776-answer plan refers to valid primary answers, not invitations sent or people opening a screener. Eligibility, incomplete questionnaires and unusable primary answers can each reduce the achieved base.

For an illustrative allowance, assume 60% of screening starts qualify and 90% of eligible starts produce a valid primary answer. The required screening starts are 776 ÷ (0.60 × 0.90), rounded up: 1,438. Invitations need a further allowance for people who never start. Use relevant recruitment evidence for these rates; none is a claimed coffee-buyer response rate.

Ask the provider to quote the required valid base in each concept group, with the screening and replacement rules visible. Track actual qualifying and valid counts during fieldwork. A shortfall in occasional buyers doesn't disappear because another group fills the total.

A human opt-in panel also doesn't become a probability sample when it reaches 776. AAPOR's disclosure standards distinguish probability sampling error from nonprobability precision measures, which require the underlying model, validated assumptions and calculation to be explained. Recruitment quality, coverage and measurement still matter after the target is met.

Design changes can reopen the calculation

Showing both offers to the same people creates paired answers and possible order effects. It doesn't reproduce two independent groups with fewer recruits. Our monadic and sequential monadic guide explains the exposure records and linked outcomes needed for that design.

Adding a third concept also adds possible comparisons. Three concepts have three pairwise comparisons; applying the same unadjusted test to all of them changes the study's false-positive risk. Name a primary comparison or agree a suitable multiple-comparison plan. Don't multiply the two-concept sample mechanically and assume every possible claim is covered.

Weighting, household or account clustering, several primary outcomes and important missing answers can change the requirement too. Keep those features with the calculation so the final report isn't held to an assumption the actual design broke.

The final sample plan should change the purchase

For the coffee team's ten-point comparison, the starting plan is 388 valid human answers per offer, with one complete card per person and a separate understanding measure. If a critical subgroup needs the same comparison, revise the recruitment plan before accepting the quote. If the team can only fund 200 per offer, choose a decision that can tolerate the weaker detection ability or reconsider the study scope.

Sapien's team can compare complete offers with simulated audiences and help interpret the reactions. A generated count of 388 or 10,000 isn't a human sample and cannot be inserted into these calculations to claim equivalent power or a human-survey margin of error. Judge the modeled comparison through task-relevant human validation and inspect the conditions behind its findings.

When discussing a concept study with Sapien, identify the difference that would change the coffee trial, the buyer group that could reverse the choice and the human evidence still required. The useful sample plan makes those costs and limits visible before anyone buys the study.