How Accurate Are Synthetic Users? 14 Human Comparisons

Synthetic-user accuracy depends on the task, audience and measure. In the 14 Sapien comparisons below, the leading menu message matched human research, but one EV scenario reversed the preferred alternative. Compare results in their original units and check whether an error would change the decision.

Our company built Sapien. These comparisons put its estimates beside published human research and identify each audience, question and scale. Twelve cases report normalized accuracy scores. Two others report dollar differences and matches to interview findings. Read the paired results to see both close agreement and misses hidden by a high overall score.

Sapien validation evidence

Use the paired cases below as a record of agreement and consequential misses. They distinguish observed actions, stated intentions, probabilities reconstructed from a published model and matches to qualitative findings. Check the original units, audience and outcome before applying a result to another study.

For a shorter route through the record, compare ad liking and its Uber Eats miss, the pension-satisfaction gap and the savings offer’s action-rate difference. The full source links, paired results and limitations remain with each case. Use our blind-pilot guide to define an evaluation for a new decision, with reference answers held back and rules chosen before scoring.

Creative and message testing

1. Ad testing

USA TODAY’s Ad Meter ranks Super Bowl commercials using scores from registered viewers, who rate each ad from one to five. Sapien scored eight ads from the 2025 game to see how closely it could recreate those human ratings.

Sapien reached a reported 96.5% normalized accuracy across the eight tested ads. Its ratings differed from the human scores by about 0.14 points on average. All eight were within half a point; seven were within 0.2.

Scores are out of five. Parentheses show ranks within the eight tested spots, rather than all 57 Ad Meter ads. The human scores come from USA TODAY’s full ranking and winner announcement. The Homes.com sequel ranked 40th overall.

Uber Eats was the only break in the relative order. Remove it from both lists and the remaining seven ads have exactly the same order. Its lower Sapien position shifts Novartis, Mountain Dew and Disney+ in the displayed ranks. Those shifts are not three separate ordering errors.

Both groups chose Budweiser. This comparison supports using the model to shortlist ads for this task. The 0.46-point Uber Eats gap needs investigation. Liking an ad does not tell you which one will generate more sales.

Our agency example shows why the team would keep the spot in contention and how it could test a clearer Uber Eats cue.

2. Message testing

Can a sentence above a menu change what people order? Trial 2 of the World Resources Institute’s menu study tested five environmental messages and a no-message control with 2,902 people. Each person chose lunch and dinner.

Sapien recreated the conditions and counted vegetarian selections across both meals. Every condition was within 0.7 percentage points of the human result, and the complete message ranking matched.

These are shares of menu selections across lunch and dinner, not shares of participants. The reported normalized accuracy score was 97.8%.

“Small changes, big impact” led in both comparisons. It increased vegetarian selections over the control by 12.8 points in Sapien and 13.0 points in the human trial. A messaging team would have chosen the same line to advance.

3. Claim and label testing

An environmental label can improve a product’s perceived sustainability without producing an equally large change in willingness to buy it.

The Taillie and colleagues experiment randomized 1,013 US parents to see eight products with an eco-score label or a barcode control. They rated sustainability and future purchase intention on 1–5 scales. Sapien recreated those conditions in the US-parent context.

Eco-score minus barcode control, on 1–5 rating scales. Selected effects from the eight-product comparison.

The veggie-pizza result shows the difference. The sustainability-rating gain was more than twice the purchase-intention gain in both sets. A packaging team needs to know which result a label changes. Neither rating is a purchase.

Product decisions and pricing

4. Purchase intent testing

A customer who will try an unfamiliar food once may still decline to buy it regularly. Bryant and colleagues’ 2019 study asked about regular purchase of plant-based meat and “clean meat,” its term for cultivated meat. Sapien recreated the US sample context.

Stated regular-purchase intentions under the study’s 2019 product descriptions.

The high-intent cultivated-meat estimate was within 0.6 percentage points of the human result. Sapien underestimated outright rejection of plant-based meat by 6.6 points. That matters when planning for regular use rather than a first try. These results concern the study’s 2019 descriptions; they do not forecast current sales.

5. Feature prioritization

Which EV benefits deserve the most attention? The 2019 Dutch proEME survey recorded weighted responses from 1,553 participants, who could select several purchase priorities. Sapien recreated the same multi-select factor list.

These are selected factors from a 15-answer question. People could choose several, so shares do not add to 100%.

Both sets have the same order, and each displayed share is within 0.7 percentage points. Either would put price and range first in the brief. These were priorities for that audience in 2019. A launch today needs a current audience and offer.

6. Willingness to pay

A pricing test becomes more informative when buyers can choose nothing. Our recreation of the Van Loo, Caputo and Lusk burger experiment offered four burger types at different prices without brand names, alongside a no-purchase option.

These are choice-model estimates in dollars per pound, relative to buying nothing. They are compared with the human study’s unbranded MNL control. They are not prices buyers named or paid at checkout.

Across the four products, the average absolute difference was $0.24 per pound. The largest gap was $0.39 for farm-raised beef. Both sets ranked the products in the same order. This gives a pricing team a close estimate for this task. Test an actual offer before choosing a shelf price.

7. Product trade-off testing

Averages can conceal a reversal that changes the decision. Using Zou, Khaloei and MacKenzie’s EV scenarios, we varied the walk to slow charging and access to fast charging within a 15-minute drive. The selected new- and used-car scenarios specified a 40-year-old man who did not own an EV.

These are selected scenarios from a 36-scenario comparison. The human probabilities are approximate calculations from the published choice model. They are not raw observed choices or EV sales.

Without fast charging, Sapien estimated higher EV choice among used-car buyers than new-car buyers. The human-study model had the reverse order. The used-car gap was about 18.4 percentage points. A charging-access decision needs this detail even if the overall comparison has high agreement.

View an example study to see the audience, offer and findings before planning your own.

Customer behavior and segmentation

8. Shopping behavior

How do shoppers respond to rising food prices? We compared four reported behaviors from the 2024 IFIC Food and Health Survey. The audience was people who noticed food and beverage price increases; each share combines “always” and “often.”

Human eligible base: 2,728 people who noticed price increases. Human shares are from the IFIC report.

Sapien was within 1.9 points for switching to cheaper products or brands. It underestimated choosing less-premium products by 7.8 points. A consumer-goods team needs to distinguish those responses to price pressure. Both sets record what people said they did, rather than basket transactions.

9. Audience segmentation

Do people with EV experience value the same benefits as those who have never driven one? Sapien recreated the six-benefit question in the 2019 Dutch proEME survey. Each result below subtracts the never-driven share from the experienced share. It is not either group’s own percentage.

Weighted human responses from the published survey data: 212 EV-experienced and 1,323 never-driven participants, excluding 18 unsure responses. pp means percentage points.

Sapien matched the price gap and followed the human direction for range, charging stations and running costs, each within two points. The climate-effect gap had the opposite sign. These differences can inform which benefits to emphasize for each audience, but the groups contain different respondents: the comparison cannot establish that driving an EV caused the differences. See the original survey and questionnaire.

10. Brand perception

Knowing a brand and liking it are separate judgments. The BRAND dataset measures both. We tested eight food, beverage and restaurant names from its 2024 US brand-name study, using a 1–7 scale for each judgment.

Brand-name liking, rather than logo liking. Each mean in the human 2024 BRAND data is based on 39–41 ratings.

Sapien put Cheerios slightly above Chick-fil-A; people put Chick-fil-A slightly above Cheerios. Budweiser had a larger gap of 0.66 points. Distinguish a near tie from a larger miss rather than treating every rank change alike.

11. Customer satisfaction

We tested customers’ ratings of their own providers across eight product groups in FCA Financial Lives 2024. Sapien’s averages stayed between 7.57 and 7.91 out of 10. The FCA’s published results vary more sharply.

Ratings from 0 to 10. Each category covers holders of that product; FCA averages exclude “don’t know” answers.

Sapien ranked pension satisfaction highest in its tested set. The FCA found it comparatively low. Sapien’s pension estimate was 1.21 points above the human average. A team deciding which customer experience to improve should resolve that gap before spending resources.

Offers and decision clarity

12. Offer testing

Could a simpler reply turn an attractive offer into action? Trial 3 of FCA Occasional Paper 19 sent longstanding savings customers a letter offering a higher-rate account. One version added a prefilled form and prepaid envelope.

The human study recorded action within four weeks: converting or closing the account, or withdrawing at least 95% of its starting balance. Sapien simulated answers corresponding to those outcomes.

The displayed rates imply 84 additional actions per 1,000 recipients in Sapien, close to the human study’s 87. Both would point the team toward the simpler response package with a similar estimated benefit. The human result records action; the Sapien result simulates it.

13. Customer interviews

Can synthetic interviews surface the concerns found in human qualitative research? We recreated the early interviews from FSA/Ipsos research into online food ordering, before discussion of hygiene-rating display requirements.

Human participants focused on cost, delivery time, reviews and familiar outlets. Many assumed food was safe without actively checking hygiene. Unfamiliar outlets or bad experiences made hygiene more important to them. The simulated sample had 40 participants: 20 in England and 20 in Northern Ireland.

Reviewers compared simulated interviews with published human findings. These percentages count matches to findings. They do not measure how common a concern is or supply a normalized accuracy score.

Both reviewers found much of the human pattern, but they chose different sets of five matches. The comparison supports using simulated interviews to develop questions about familiar concerns. It cannot show how common those concerns are across the population.

14. Comprehension testing

Explaining an offer more clearly may help with one benefit while leaving another unchanged. The BIT/Citizens Advice experiment compared a standard home-battery offer with one explained through a plain FAQ.

We focused on understanding cheaper off-peak electricity, solar energy use and “flexibility services,” which let an energy supplier use the battery.

Each change is the share answering correctly after the FAQ minus the share after the standard offer. The human report did not test whether these differences could be due to chance.

Sapien’s correct answers for flexibility services rose from 21.8% to 37.0%, closely matching the human improvement. Its solar result fell from 66.0% to 62.5%, also in the human direction. Either would identify the same largest gain in understanding. That does not tell you whether clearer wording increases purchases.

Rerun stability: repeatability is different from accuracy

If the winner changes when you repeat a synthetic study under the same conditions, keep both offers in play. Check whether the reversal crosses the margin you agreed would change the decision and which buyer groups account for it. If you still need to pick one offer, resolve a meaningful reversal with evidence from the people who would make that choice.

Decide what would count as a repeat

Fix the audience definition and source data, the offer cards, question wording and order, answer options, response counts by segment, and rules for unusable outputs. Record the study protocol, model or service version, and any generation settings the provider can confirm. Use fresh executions under those conditions. Where a material setting is unavailable, mark the repeat conditions as partly verified.

A deliberate change to wording, audience inputs, or model version answers a sensitivity question. Run it as a separate condition so you can see which change moved the result. If a supposed repeat used a different offer card, correct its label and keep it in the readout. Source: Our silicon sampling guide.

Agree on the margin before the first run

Consider a bike-share team that needs to choose one weekday pass for a field trial. Pass A is $18 for three rides. Pass B is $24 for five. Both cover weekday rides of up to 30 minutes, expire after 30 days, and require a fresh purchase to renew. Riders can also keep paying $8 per ride under the existing terms. The protocol, prices, and responses below are illustrative.

The study asks occasional riders, who took one to three weekday rides in the prior month, and regular riders, who took at least four, to choose a pass or keep buying single rides. The team gives both groups equal weight in the combined comparison. Regular riders are the priority group for the trial.

The team plans three runs. To shortlist a pass by itself, it must lead the next most chosen option by at least eight percentage points overall and among regular riders in every run. A smaller lead keeps both offers open. If two runs show leads of at least eight points in opposite directions overall or in the priority group, take both passes into a human comparison. The eight-point margin reflects how large a difference the team would need to drop a pass from the shortlist.

Put every run in the readout

Each run generates 100 attempted choices per segment, all valid in this example. Entries show counts for Pass A, Pass B, and single rides, in that order. The combined lead uses all 200 responses in each run.

In run 1, A gets 99 of 200 choices and B gets 80. The gap is 19 divided by 200, or 9.5 percentage points. Among regular riders, A leads by 12 points. In run 2, B gets 99 choices to A’s 80 overall and leads by 22 points among regular riders. Both runs clear the margin in opposite directions. In run 3, A leads by 0.5 points overall, which is too close to shortlist a pass. Keep both offers in play and include regular riders in the human check.

Use the same readout for your own comparison. Keep attempted and usable counts, exclusions, and exact conditions with each row. Report segment results even if the combined ranking looks steady. A change in the group that drives the decision can matter more than a stable result overall.

Use human evidence to resolve the choice

Repeatability asks whether the decision holds under the same protocol. Sensitivity asks how it changes when an input changes. Accuracy requires an independent comparison suited to the decision. The same audience inputs can produce the same error in every run. Source: NIST AI Risk Management Framework.

Show eligible riders the same pass terms and the single-ride alternative. Agree on a meaningful lead and how you will handle an inconclusive result before reviewing answers. If paid uptake is the decision, measure actual purchases in the field. Keep the human outcome separate from any adjustments to the synthetic setup so it can serve as an independent check. Source: Our guide to synthetic research validation.

Check the decision as well as the score

Ask whether the model works for your question. Close ad-liking scores do not establish accuracy for a different audience’s willingness to pay. Choose human comparison data that fits the decision and compare results in the units you will use to make it.

Start with the original study. Record the population, recruitment, exact questions, alternatives and measures. Keep the conditions comparable: selecting several features is different from choosing one favorite, and understanding is different from buying. Label any result calculated from a published model. The EV trade-off figures above are approximate model probabilities, not observed shares.

For a new validation, define success before reading the human results. For a message choice, check the winner and size of its lead. For a target-group decision, check whether the model identifies the same group. Keep the human answers out of simulation inputs and save what the model received. With published studies, this does not prove the answers were absent from model training data. Record what was controlled and what remains unknown.

If you’re evaluating a provider for your own study, our blind pilot guide shows how to hold back the reference answers, compare baselines and set task-specific acceptance rules.

Show the miss that could change the recommendation

Show results side by side with their differences and any disagreement that could change the choice:

• In the used-car scenario without fast charging, Sapien estimates EV choice at 53.1% while the reconstructed human estimate is 34.7%. That puts EV choice above versus below the halfway point and reverses the preferred alternative, even though other EV comparisons are closer.
• In the financial-product comparison, a narrow range of synthetic satisfaction scores conceals the pension gap. Sapien's 7.91 and the human 6.7 differ by 1.21 points on the ten-point scale, affecting which product looks most in need of attention.
• In the savings-offer comparison, the useful quantities include the 8.4- and 8.7-percentage-point increases. Readers can see the intervention's estimated size without having to infer it from a normalized score.

Check buyer-group differences too. The climate answer in the EV comparison has opposite directions in the simulated and human results. A close overall pattern does not justify using that particular group recommendation.

Keep different measures and studies separate

Ask how an accuracy score was calculated, which results it includes and how studies were combined. A rank, percentage-point error, dollar difference and interview-finding match answer different questions. Keep each with its task. Combining them requires a justified common measure.

The twelve normalized scores are reported results, but the complete formulas and inputs for each measured outcome aren’t provided here. The selected tables alone aren’t enough to reproduce them. Ask Sapien whether those materials are available; until then, judge the decisions supported by the paired results and errors shown here.

Use a disagreement to choose the next step. You may need better audience data, clearer research materials, a narrower claim or a focused human study. Keep the original result when changing the setup so readers can see what changed. Useful validation shows which recommendation the team can act on and which still needs evidence.

What these comparisons show

The closest matches included menu-message selection, EV feature priorities, burger pricing and the simpler savings offer. Sapien chose the same leading menu message and ranked the displayed EV priorities in the same order. Burger willingness to pay differed by an average of $0.24 per pound. Uber Eats was the only break in the relative order of the eight ads.

Among the 12 cases with reported normalized scores, the lowest was 95.3% for brand familiarity and liking. The highest were about 98.3% for EV priorities and offer testing, rounded to one decimal place. Each concerns its own task and measured outcomes. These are not percentages of people predicted correctly. The other two cases use dollar differences and interview matches. Use the paired results to understand agreement in practical units.

The cases cover consumer goods, food and restaurants, cars, financial services and home energy. Choose a similar case to define what a proposed study needs to get right: message order, a price gap, a buyer-group difference or an offer’s effect on action. Resolve important disagreements before committing resources based on a simulated result.

For a product-concept example, see our illustrative Red Bull protein study. It uses unofficial concepts and simulated consumers; it was not a project commissioned by Red Bull.

Our validation evidence overview offers a shorter route through selected agreement and misses in their original units. Use the full paired cases here to inspect the source, audience and outcome behind each comparison.

If you’re considering Sapien, start with the closest paired case above. Ask which audience, stimulus, outcome and failure check a proposed study would use for your decision. Explore the research you can scope with Sapien.