How to Code Open-Ended Survey Responses After a Concept Test
Give each objection a definition and check it with a second coder. Then separate what needs a clearer explanation from what needs a product change.
Code open-ended survey responses around the decision the concept team can make. The same comment can raise an objection about price and reveal that the shopper misunderstood the offer. Preserve both ideas, then separate what needs a rewrite from what needs a change to the product.
A fictional pantry delivery service asks shoppers why they would pass on a monthly box. “I thought I'd have to pay again for delivery, and I'm already stretched” is an illustrative response. It mentions an assumed fee and affordability. Put both in the record. Fixing the delivery line may resolve one objection while leaving the other in place.
Build the codebook from a first pass
Read a mix of comments from the comparison cells and audience groups, including short, uncertain, and contradictory answers. Draft code definitions with inclusion and exclusion rules. Give each one an example and a boundary case. Keep the prompt and concept version attached to the response.
Code | What counts in this example |
|---|---|
Fee confusion | The shopper assumes an extra fee that the offer says is included. Questions about the delivery date belong in a separate code. |
Total price too high | The shopper explicitly objects to the cost. A comment about an ambiguous fee can take both codes if it also raises affordability. |
Contents unclear | The shopper cannot tell what is included or expects an item outside the offer. |
No usable reason | Blank or unrelated text. Record blank responses and unusable written responses separately in the final readout. |
Check the rules with a second coder
Have a second person code the same trial subset independently using the written definitions. Compare assignments and inspect disagreements. If one coder treats “I don't know what it costs” as affordability and the other treats it as confusion, refine the definitions and recode affected records. Once the rules are stable, use them for the remaining comments. Record whether they were human-coded, machine-assisted, or both.
For a simple readout, if five of 20 usable replies mention delivery confusion, report 5 of 20. If 25 eligible respondents received the question, also show five missing or unusable responses. Count the number of people with each code and make clear that the same person can appear under more than one code. Sapien's market research data analysis guide
Make the readout lead to a repair
If confused shoppers repeatedly assume a fee the offer does not include, move “delivery included” into the offer line and check comprehension on a fresh exposure. If people who understand the fee still object to the total, test a different quantity or price. Include one or two representative examples of each code when permitted so the product team can see the distinction it is acting on.
An open response can reveal a reason the team omitted from the questionnaire. Use a follow-up question or experiment to find out how much that reason matters to the purchase decision. Attest’s open-ended survey guide