Where Synthetic Respondents Go Wrong, and What Helps
A practical checklist for spotting weak synthetic evidence and choosing the next research step, with a specific published Sapien miss.
Synthetic respondents can give plausible answers while missing the pattern that matters to your decision. Check for excessive positivity, compressed differences, weak audience coverage, wording sensitivity and unsupported claims about new or lived experiences before using a result.
The practical question is whether a particular failure would change what you do next. If a team is choosing a message to develop, a stable relative comparison may be useful. If it is committing to a production order, a model that overstates willingness to buy can create a much more expensive mistake.
This guide is a study-review checklist. Some risks have been documented in research on general language models; their size depends on the model, grounding and task. A finding about ChatGPT is not a measurement of every synthetic research provider.
Answers can be too positive to help you choose
Consider an illustrative subscription service called Weeknight: five dinner recipes and a consolidated shopping list for $8 per month, with groceries purchased separately. The team wants to decide whether convenience or less food waste should lead its campaign.
If every modeled profile praises both messages, the study may leave the decision exactly where it started. The problem becomes clearer when you ask what the person would give up, choose instead or find unsuitable. A statement that a service sounds useful carries a different commitment from selecting it over an existing routine at the displayed price.
What to check: Retain negative and uncertain responses. Show the current alternative and a realistic choice to buy nothing. Inspect whether explanations describe an actual trade-off or simply repeat the benefits in the stimulus.
What helps: Ask for unaided comprehension before favorable attributes. Compare complete offers under the same conditions, and inspect rejection reasons alongside appeal. Treat a high positive share as something to explain, rather than proof of demand. This is a design check, not a claim that every model systematically overpraises every offer.
An average can match while differences disappear
A study can reproduce an overall mean and still miss how responses vary. That becomes a problem when the business needs to identify a reluctant group, a divided market or a reason that matters to a minority of buyers.
In Bisbee and colleagues' study of ChatGPT-generated public-opinion data, average ratings resembled the human survey, but synthetic answers had less variation and estimated relationships often differed. The study also found sensitivity to prompt wording and changes over time. It concerned specified models and sociopolitical questions; it is evidence for checking those properties, not a current audit of commercial platforms.
What to check: Look at the full answer distribution and relevant relationships, not only the mean. For Weeknight, do profiles with a tightly constrained food budget react differently to the subscription price? Does the readout preserve rejection as well as enthusiasm?
What helps: Compare distributions and consequential audience differences with an appropriate held-back reference. A ranking match and an accurate response share answer different questions. Set the tolerable error around the decision you need to make.
A large synthetic count can conceal weak subgroup coverage
Generating many profiles does not establish that the evidence behind a narrow audience is rich enough to support a conclusion. The important gap may concern a buying role, a local market, an unusual constraint or a recently changed behavior.
For Weeknight, “households with dietary requirements” may combine very different situations. A model grounded mainly in general grocery preferences may be a weak basis for deciding how the service should work for a household with several competing restrictions.
What to check: Ask what sources support the subgroup, their dates and their coverage. Keep grounded attributes, assumptions and unknowns visible. Check whether the subgroup result would actually change the offer or research plan.
What helps: Narrow the claim to the audience the evidence supports. Use direct customer work to investigate an important missing experience. If a supplier proposes conventional human-sampling confidence intervals based solely on the generated count, ask how those intervals account for model error. More generated records do not resolve a missing source.
Our synthetic vs human respondents guide explains how to assign each source a clear job.
Small wording changes can move the result
“Would this save you time?” points attention toward a benefit. “What, if anything, would change about your meal planning?” leaves more room for the service to be irrelevant or inconvenient. Human research also needs careful questionnaire design; leading wording is a design problem before it becomes a model problem.
Bisbee and colleagues' findings give another reason to record the actual prompt and model version when evaluating language-model research. A result that cannot be reproduced under its stated conditions is difficult to use as a baseline.
What to check: Keep the complete stimulus, instructions, answer options and exposure order. Ask whether plausible neutral rewordings reverse the decision, rather than expecting every individual response to remain identical.
What helps: Fix the main instrument before reviewing its result. Where wording sensitivity could change the action, agree a separate check in advance and retain all its outcomes. Do not keep trying prompts until the preferred concept wins. Our guide to leading and double-barreled questions helps identify avoidable wording problems.
Novel products and lived experiences need different evidence
A model can react to a description of a folding phone or meal-planning app. That does not establish what happens after someone uses it, encounters a fault, receives a bill or decides whether to renew.
For a new product, ask which familiar needs and alternatives the grounding can support and which new interaction remains uncertain. Weeknight's shopping list could sound convenient while proving awkward in the supermarket. A simulated account of that visit would not supply an observation of product use.
What to check: Separate expected reactions, recalled human experiences and recorded behavior in the readout. Check whether the target answers or close equivalents were available during model construction when a study is presented as a prediction.
What helps: Hold the evaluation evidence out where feasible and document any overlap or uncertainty. Use an appropriate human task for physical handling, usability or a fresh lived episode. Use transactions or a suitable live experiment for a behavioral outcome. Our AI research vs A/B testing guide explains the move from modeled comparisons to measured response.
A useful overall result can hide the wrong individual recommendation
Our fourteen comparisons against human research include a concrete example. Across eight 2025 Ad Meter ads, Sapien's scores differed from the human ratings by about 0.14 points on a five-point scale on average. Both series put Budweiser first. But Uber Eats scored 2.92 and ranked fifth in Sapien, compared with 3.38 and second in the human results.
That 0.46-point gap matters if Uber Eats is the creative your team might discard. It shows why a useful shortlist still needs inspection of the exceptions. The ranks cover the eight tested spots, not the entire Ad Meter field, and ad liking does not establish sales impact.
What to check: Ask for the largest consequential miss, the exact reference and the unit of error. Check whether an overall score masks a different decision for a specific item or audience.
What helps: Review exceptions alongside the aggregate. Investigate why they could matter, and use the relevant evidence to decide whether the result is strong enough for the next commitment.
When a pilot misses, buy the repair that fits the failure
Suppose Weeknight's pilot looks encouraging overall but gets one important group wrong. The next purchase shouldn't automatically be a larger run of the same study. First identify whether the disagreement concerns the offer, the audience evidence, the questionnaire or the outcome the model was asked to predict.
For the $8 meal-planning service, imagine that the proposed comparison is convenience versus less food waste. If the card makes some readers think groceries arrive with the recipes, repair that misunderstanding before buying a more precise appeal estimate. If the group with competing dietary restrictions lacks relevant source coverage, a clearer card alone won't fill that gap. These are proposed failure scenarios for the existing example.
Pilot problem | Work to scope before another decision |
|---|---|
Buyers or the model evaluate an imagined grocery-delivery offer | Correct the exact card and check unaided understanding with fresh first exposure |
The overall pattern fits but a decision-critical audience doesn't | Inspect that group's reference and source coverage; collect the missing human evidence where needed |
Neutral wording variants reverse the message choice | Keep every result, investigate the wording and repeat the locked comparison against appropriate held-back answers |
Appeal looks strong but the decision requires renewal behavior | Plan the actual service or transaction evidence; another appeal run doesn't supply renewals |
This diagnosis should produce a revised scope with an owner and an acceptance rule. If the missing evidence concerns households with multiple dietary restrictions, describe those constraints in the recruitment or source brief. Keep the actual recipes, grocery responsibilities, price and alternative routine fixed. Identify what the human comparison must answer and what simulated exploration will do alongside it.
Ask for the outputs that let your team inspect the repair: the changed input, the reason for changing it, the human and modeled bases kept separate, the relevant discrepancies and the resulting recommendation. If only the affected group is repeated, say which earlier comparisons remain usable and which are no longer matched. A narrow repair can be efficient, but it shouldn't silently turn an old overall result into a newly validated study.
Don't use the reference answers revealed during diagnosis as a fresh blind evaluation. Those answers can help explain the error. A later claim that the revised process predicts unseen answers needs evaluation evidence that wasn't supplied to make that repair. Our blind-pilot guide explains how answer custody and a prespecified comparison support that separation.
Compare proposals on the work still required to make the decision. A cheaper rerun that leaves the important subgroup unsupported hasn't completed that job. A new human study of every segment may also be unnecessary if only one missing experience blocks the next step. Ask each provider to separate the diagnosis, revised study, human follow-up and interpretation it is actually quoting.
The result can be a narrower use of the method. Weeknight might use the modeled study to investigate how to explain the shopping-list service while reserving the dietary-restriction claim for direct customer work. Or it may pause the message choice until the missing comparison is complete. Agree that disposition before expanding the project, so the next invoice buys evidence the team can act on.
What Sapien's documented process provides
Our audience-building guide describes consumer and research grounding, an audience built around the decision, team-run studies, segment comparisons and responses you can inspect. Follow-up questions can explore a surprising result or unclear interpretation.
Those features give the research team places to investigate a problem. They do not make the risks vanish or establish that every study receives every validation check described above. Agree the relevant sources, comparisons and evaluation work for your project.
A dataset that resembles source records and a simulated respondent that answers a new question need different checks. Our synthetic data vs. synthetic respondents guide explains that distinction before you choose a validation claim.
Bring us an offer that appears promising and the finding that would make you change course. Discuss a study with Sapien so the comparison can focus on the decision you need to make.
How to validate a synthetic study yourself
Use this short review before acting:
- Name the decision. State what would advance, change or stop if the result crossed a threshold.
- Match the evidence. Use the same eligible audience, offer, question and response scale for the comparison where possible. Record meaningful mismatches.
- Separate inputs from evaluation. Identify which human or behavioral evidence informed the model and which was reserved for checking it.
- Define the score first. Use answer-share error for proportions, rank agreement for ordering or another measure suited to the output. Preserve the original units.
- Inspect failures. Read the weakest items and important subgroups, including uncertainty and missing coverage.
- Choose the next evidence. Advance a robust direction; repair an unclear stimulus; commission customer research where the remaining gap affects the commitment.
Copy our synthetic research brief template to record those requirements. A useful validation result tells you what you can do next and which uncertainty still matters.
