Synthetic market research vs ChatGPT: what are you paying to learn?

If your team already uses ChatGPT to review concepts, analyze files and challenge a campaign brief, hiring a synthetic research provider raises a reasonable question: what would the additional work tell you?

The answer should be more specific than “better personas” or “AI trained for research.” You need to know how the audience is defined, which evidence informs its responses, how alternatives are compared and what has been validated against people or behavior. A useful provider should make those differences inspectable.

Start by separating two jobs. An assistant can help your team think through a decision and work with evidence. A synthetic study produces modeled responses from a defined audience under a research design. Both can use language models, retrieval, uploaded documents and analytical tools. The value of the study depends on how those components are assembled and checked.

ChatGPT is a capable research assistant

An ordinary prompt can be useful: “What might shoppers misunderstand about these two pack designs?” It can expose ambiguous language, suggest objections and help a team prepare better questions. Give an assistant customer interviews and it can help organize the accounts, compare explanations or inspect a dataset.

Current official OpenAI documentation describes web research, file reading, data analysis and connected tools in ChatGPT Work, with availability depending on the plan, platform and workspace settings. A fair comparison therefore cannot assume that ChatGPT is restricted to an isolated text prompt with no sources.

Nor does a research label automatically make another system more reliable. A provider can produce detailed personas and an attractive report while leaving the audience assumptions or comparison procedure unclear. Evaluate the work behind the output.

Follow one decision through both approaches

Consider Wren Tea, a fictional unsweetened iced-tea brand preparing a convenience-store launch. Its marketing team is choosing between “A calmer afternoon” and “Real tea. No sweet finish.” The product, pack size and price are fixed. The intended buyers already purchase a cold drink during their workday.

A ChatGPT discussion could help the team notice that “calmer” may suggest a functional ingredient or a physiological benefit. It could propose clearer copy and identify questions about taste, caffeine and sugar. Those are valuable hypotheses to investigate.

The market question is more demanding: which message creates the intended understanding among the buyers Wren wants, and how does that understanding affect their choice against their usual drink or buying nothing?

A study needs to keep those outcomes separate. A message might communicate “less sweet” clearly without making the drink more appealing. Another might attract attention by creating an expectation the product cannot meet. Research can reveal an interpretation; the team still has to verify the underlying product claim.

That is where the audience, task and evidence begin to matter more than how persuasive the assistant's explanation sounds.

Grounding should change something you can examine

“Grounded in data” is useful only when you can connect the data to the research question. Wren's past interviews, drink-purchase patterns and category evidence could inform different needs: avoiding a sweet aftertaste, seeking caffeine, managing price or choosing something familiar.

Demographics alone would leave much of that choice unexplained. Two office workers of the same age can buy very different drinks for different occasions. A study should preserve relevant differences instead of producing a single average shopper with several names.

Ask how the workflow handles missing information. Does it distinguish an observed purchase pattern from an assumed preference? Does it use evidence about the intended market and occasion? Can the researcher explain which inputs shaped a segment and which uncertainties remain?

Retrieval is one way to supply context. It does not establish that the retrieved material measures the target population or that the resulting answers are accurate. The same applies when a team uploads files to ChatGPT. The quality of the sources and their use remain research decisions.

At Sapien, we build simulated populations grounded in purchase, behavioral and research data, with customer data and past research able to add context. Our team builds the audience, runs the study and helps interpret the findings. The useful conversation is what that approach can add to your particular decision and existing evidence.

More generated respondents do not settle the question

Asking a model to create 100 personas gives you 100 generated profiles. It does not, by itself, give you a representative sample or 100 independent observations from customers. The profiles may share assumptions, omit a relevant group or respond similarly because of the underlying model and instructions.

Academic findings show why the evaluation needs to be specific. Argyle and colleagues' 2023 study found that conditioning GPT-3 on human survey backstories could reproduce important patterns in the US survey settings they examined. That is evidence for a particular research approach and task, rather than a general guarantee for product choices.

In Bisbee and colleagues' 2024 study, persona-prompted ChatGPT responses could approximate average sociopolitical ratings while showing less variation and different relationships between variables than the human survey data. The work tested models and configurations from that period. It does not rank today's ChatGPT against Sapien, but it illustrates why matching an average is an incomplete validation target.

For Wren, a reassuring overall score would be weak evidence if the system missed the group that mistakes “calmer” for a health promise. Inspect the segment disagreement and the reasons behind the response, as well as the aggregate ranking.

A fair comparison needs a fair question

There are two useful tests, and they answer different buying questions.

One compares complete workflows. Give the ChatGPT-assisted team and the research provider the same business brief, relevant permitted source materials, budget and deadline. Compare what each delivers: audience rationale, question quality, interpretation, disagreements and usefulness for the decision. Record the human work needed to reach that result.

The other tests predictive agreement. Use a separate human or behavioral reference that fits the audience, stimuli and question. Keep its target results out of the generation inputs, define the scoring before viewing the answers and evaluate every agreed item, including misses.

If one condition gets detailed customer history and another gets a one-sentence prompt, a difference in output could reflect the supplied context. It would not isolate model capability. That comparison might still describe the practical value of a managed workflow, provided you report what each side received.

Use a metric that matches Wren's actual question. Correctly identifying a pack's meaning, reproducing a preference distribution and predicting a purchase are different tasks. A ranking match does not tell you the size of the difference, and a close average can hide an important reversal. Our synthetic-user accuracy article shows why the original task, scale and individual comparisons need to remain visible.

Choose the amount of research the decision deserves

Use ChatGPT directly when the immediate job is to organize existing evidence, critique a draft, explore possible objections or prepare a research brief. Keep the outputs tied to their sources and treat generated customer reactions as hypotheses.

Consider a managed synthetic study when you need a defined audience, consistent comparisons across alternatives, segment-level interpretation and follow-up exploration that your team would otherwise have to design and operate. Ask to see how the provider handles your audience, evidence and decision before assuming the additional service is worthwhile.

Use appropriate customer or behavioral research when the decision requires observed experience, actual payment, physical or sensory feedback, representative estimates or consequential external proof. Synthetic exploration can help narrow the question and compare candidate changes; its output should retain the conditions under which it was produced.

For Wren, the next step could be a sharper copy test, a revised product explanation or a live choice experiment. The useful result is knowing which uncertainty remains and what evidence would resolve it.

If you are deciding whether a synthetic study would add value to your current ChatGPT workflow, bring Sapien the decision and the evidence you already have. We can discuss the audience, alternatives and findings that would make the work useful before choosing the study.