How Accurate Are Synthetic Users? 14 Tests Against Human Research

Can synthetic consumers pick the same winning ad as people? Can they identify the strongest message, estimate willingness to pay, or spot a confusing offer?

We gathered 14 common research use cases, covering ads, messaging, product choices, pricing and customer experience. We recreated tasks from published human studies with a synthetic model, then compared its answers with the original results. The question was how closely simulated consumers could reproduce findings that research teams use to make decisions.

The model is Sapien, built by our company. The comparisons below put its estimates beside the human evidence, with the original audience, question and measurement scale made clear. Twelve cases report normalized accuracy scores; the other two report dollar differences and matches to interview findings. The paired results show where the agreement was strongest and where a useful overall score concealed a specific miss.

Creative and message testing

1. Ad testing

USA TODAY’s Ad Meter ranks Super Bowl commercials using scores from registered viewers, who rate each ad from one to five. Sapien scored eight ads from the 2025 game to see how closely it could recreate those human ratings.

Sapien reached a reported 96.5% normalized accuracy across the eight tested ads. Its ratings differed from the human scores by about 0.14 points on average. All eight were within half a point; seven were within 0.2.

Scores are out of five. Parentheses show ranks within the eight tested spots, rather than all 57 Ad Meter ads. The human scores come from USA TODAY’s full ranking and winner announcement. The Homes.com sequel ranked 40th overall.

Uber Eats was the sole exception to the relative ordering. Remove it from both lists and the remaining seven ads appear in exactly the same order. Its lower Sapien placement shifts the displayed ranks of Novartis, Mountain Dew and Disney+; those are not three independent ordering errors.

Both groups chose Budweiser as the winner. For a creative team, the comparison supports using the model to shortlist ads, while the 0.46-point Uber Eats gap identifies the spot to investigate. Ad liking alone does not establish which commercial will generate more sales.

2. Message testing

Can a sentence above a menu change what people order? Trial 2 of the World Resources Institute’s menu study tested five environmental messages and a no-message control with 2,902 people. Each person chose lunch and dinner.

Sapien recreated the conditions and counted vegetarian selections across both meals. Every condition was within 0.7 percentage points of the human result, and the complete message ranking matched.

Shares of menu selections across lunch and dinner, rather than shares of participants. The reported normalized accuracy score was 97.8%.

“Small changes, big impact” led in both comparisons. It increased vegetarian selections over the control by 12.8 points in Sapien and 13.0 points in the human trial. A messaging team would have chosen the same line to advance.

3. Claim and label testing

An environmental label can improve a product’s perceived sustainability without producing an equally large change in willingness to buy it.

The Taillie and colleagues experiment randomized 1,013 US parents to see eight products with an eco-score label or a barcode control. They rated sustainability and future purchase intention on 1–5 scales. Sapien recreated those conditions in the US-parent context.

Eco-score minus barcode control, on 1–5 rating scales. Selected effects from the eight-product comparison.

The veggie-pizza effects illustrate the distinction: the sustainability improvement was more than twice the purchase-intention improvement in both sets. Packaging teams can use that separation to assess what a claim accomplishes. These are ratings and stated intentions, rather than observed purchases.

Product decisions and pricing

4. Purchase intent testing

A customer who will try an unfamiliar food once may still decline to buy it regularly. Bryant and colleagues’ 2019 study asked about regular purchase of plant-based meat and “clean meat,” its term for cultivated meat. Sapien recreated the US sample context.

Stated regular-purchase intentions under the study’s 2019 product descriptions.

The high-intent cultivated-meat estimate was within 0.6 percentage points of the human result. The larger miss was outright rejection of plant-based meat: Sapien estimated 6.6 points fewer rejectors. That distinction matters for a product team planning around repeat use rather than trial alone; the comparison does not forecast current sales.

5. Feature prioritization

Which EV benefits deserve the most attention? The 2019 Dutch proEME survey recorded weighted responses from 1,553 participants, who could select several purchase priorities. Sapien recreated the same multi-select factor list.

Selected factors from the 15-answer question. Respondents could choose several answers, so shares do not add to 100%.

The same order appears in both sets, and each displayed share is within 0.7 percentage points. Price and range would lead the brief in either comparison. The results describe priorities in that audience in 2019; a present-day launch needs a current audience and offer.

6. Willingness to pay

A pricing test becomes more informative when buyers can choose nothing. Our recreation of the Van Loo, Caputo and Lusk burger experiment offered four burger types at different prices without brand names, alongside a no-purchase option.

Choice-model estimates relative to buying nothing, compared with the human study’s unbranded MNL control. These are dollars per pound, rather than prices named or paid at checkout.

The average absolute difference was $0.24 per pound across the four products, with a largest gap of $0.39 for farm-raised beef. Both sets put the products in the same order. That gives a pricing team a close starting estimate for this task, while an actual offer test remains necessary before choosing a shelf price.

7. Product trade-off testing

Averages can conceal a reversal that changes the decision. Using Zou, Khaloei and MacKenzie’s EV scenarios, we varied the walk to slow charging and access to fast charging within a 15-minute drive. The selected new- and used-car scenarios specified a 40-year-old man who did not own an EV.

Selected scenarios from a 36-scenario comparison. Human probabilities are approximate reconstructions from the published choice model, rather than raw observed choices or EV sales.

Without fast charging, Sapien put used-car buyers above new-car buyers; the human-study model put them below. The used-car gap was about 18.4 percentage points. A team assessing charging access needs this scenario-level view, even when the broader comparison reports high agreement.

Customer behavior and segmentation

8. Shopping behavior

How do shoppers respond to rising food prices? We compared four reported behaviors from the 2024 IFIC Food and Health Survey. The audience was people who noticed food and beverage price increases; each share combines “always” and “often.”

Human eligible base: 2,728 people who noticed price increases. Human shares are from the IFIC report.

Sapien came within 1.9 points on switching to cheaper products or brands. Its larger miss was trading down from premium products, at 7.8 points below the human result. For a consumer-goods team, those are different responses to price pressure and should remain separate in the research. Both sets measure reported behavior rather than basket transactions.

9. Audience segmentation

Do people with EV experience value the same benefits as people who have never driven one? Sapien recreated the six-benefit question in the 2019 Dutch proEME survey. Each result below is the experienced-versus-never-driven difference, rather than either group’s absolute share.

Weighted human responses from the published survey data: 212 EV-experienced and 1,323 never-driven participants, excluding 18 unsure responses. pp means percentage points.

Sapien matched the price gap and followed the human direction for range, charging stations and running costs, each within two points. The climate-effect gap had the opposite sign. These differences can inform which benefits to emphasize for each audience, but the groups contain different respondents: the comparison cannot establish that driving an EV caused the differences. See the original survey and questionnaire.

10. Brand perception

Knowing a brand and liking it are separate judgments. The BRAND dataset measures both. We tested eight food, beverage and restaurant names from its 2024 US brand-name study, using a 1–7 scale for each judgment.

Brand-name liking, rather than logo liking. Each mean in the human 2024 BRAND data is based on 39–41 ratings.

Sapien placed Cheerios slightly ahead of Chick-fil-A; people placed Chick-fil-A slightly ahead of Cheerios. Budweiser had a wider gap, at 0.66 points. The results help separate a near tie from a more substantial difference, rather than treating every rank change alike.

11. Customer satisfaction

We tested customers’ ratings of their own providers across eight product groups in FCA Financial Lives 2024. Sapien’s averages stayed between 7.57 and 7.91 out of 10. The FCA’s published results vary more sharply.

Ratings from 0 to 10. Each category covers holders of that product; FCA averages exclude “don’t know” answers.

Sapien ranked pension satisfaction highest in its tested set, while the FCA found it comparatively low. Its pension estimate was 1.21 points above the human mean. A financial-services team choosing where to improve customer experience would need to resolve that difference before allocating resources.

Offers and decision clarity

12. Offer testing

Could a simpler reply turn an attractive offer into action? Trial 3 of FCA Occasional Paper 19 sent longstanding savings customers a letter offering a higher-rate account. One version added a prefilled form and prepaid envelope.

The human study recorded action within four weeks: converting or closing the account, or withdrawing at least 95% of its starting balance. Sapien simulated the corresponding responses.

The displayed rates imply 84 additional actions per 1,000 recipients in Sapien, close to the human study’s 87. The model would have led an offer team toward the same simpler response package, with a similar estimate of its benefit. Here the human reference is recorded action; the Sapien result is simulated.

13. Customer interviews

Can synthetic interviews surface the concerns found in human qualitative research? We recreated the early interviews from FSA/Ipsos research into online food ordering, before discussion of hygiene-rating display requirements.

The human report found that participants focused on cost, delivery time, reviews and familiar outlets. Many assumed food was safe without actively checking hygiene until an unfamiliar outlet or bad experience made it salient. The synthetic sample also included 40 participants: 20 in England and 20 in Northern Ireland.

Reviewers judged matches between synthetic interviews and published human findings. These percentages describe finding matches, rather than population prevalence or a normalized accuracy score.

Both reviewers found much of the human pattern, but they did not identify exactly the same five matches. The comparison supports using synthetic interviews to develop questions around familiar concerns. It cannot establish how common those concerns are in the wider population.

14. Comprehension testing

Explaining an offer more clearly may help with one benefit while leaving another unchanged. The BIT/Citizens Advice experiment compared a standard home-battery offer with one explained through a plain FAQ.

We focused on understanding cheaper off-peak electricity, solar energy use and “flexibility services,” which let an energy supplier use the battery.

Changes in the share answering correctly, FAQ minus standard offer. The human report gave these changes without testing whether they could be due to chance.

Sapien’s flexibility-services result rose from 21.8% to 37.0% correct, closely matching the human improvement. Its solar result fell from 66.0% to 62.5%, also following the human direction. An offer team would have identified the same strongest gain in comprehension. That is separate from whether the clearer explanation increases purchase.

What these comparisons show

The strongest agreement appeared in practical tasks such as menu-message selection, EV feature priorities, burger pricing and the simplified savings offer. The model chose the same winning menu message, kept the displayed EV priorities in the same order, and estimated burger willingness to pay within an average of $0.24 per pound. In ad testing, Uber Eats was the only departure from the relative order of the eight spots.

Across the 12 cases with reported normalized accuracy scores, the lowest reported score was 95.3% for brand familiarity and liking. The highest reported scores were about 98.3% for EV feature priorities and offer testing, rounded to one decimal place. Each score describes its own task and endpoints, rather than the share of individual people predicted correctly. The dollar and interview-match measures in the other two cases remain separate. The paired results above show what agreement means in the units a team actually uses.

The coverage spans consumer packaged goods, food and restaurants, automotive, financial services and home energy. Across those decisions, Sapien gives research, brand and product teams a way to compare concepts, messages, features and prices before committing to the next test. The opportunity is concrete: narrow the options, identify the audience to study, and focus human research on the decisions or discrepancies that matter most.

For a product-concept example, see our illustrative Red Bull protein study. It uses unofficial concepts and simulated consumers; it was not a project commissioned by Red Bull.

Bring Sapien the decision your team needs to make, whether it is a product concept, a campaign message, a price or an offer. Explore what to test with Sapien.