Sapien Validation Evidence: Results and Decision Limits
Inspect close matches and meaningful misses from our published comparisons, then choose the evidence your proposed research decision needs.
Sapien’s published comparison cases put simulated results beside human research for specific tasks. They show close agreement in some decisions and misses that could change others. Use the evidence to define what your proposed study must get right, rather than treat one headline score as a guarantee.
Our company produced these comparisons. The full case page names the original studies, audiences, measures and important qualifications. It also identifies where results are reconstructed from a published model, where they concern stated intentions, and where human research measured an action.
A close response effect, with the outcome identified
Our recreation of Trial 3 in FCA Occasional Paper 19 concerns longstanding savings customers offered a higher-rate account. The human study recorded conversion or closure of the account, or withdrawal of at least 95% of its starting balance, within four weeks. The displayed action rates were 3.0% for a letter alone and 11.7% with a simplified response package. The corresponding simulated rates were 2.9% and 11.3%.
The increases are 8.7 percentage points in the human study and 8.4 points in Sapien. Both point toward the simpler response package. Keep the distinction visible: the human result recorded actions under that study’s conditions; the Sapien result simulated them. This is a relevant comparison to inspect for an offer-response question, not a current forecast for every savings product.
The individual miss can matter more than an average
Across the eight 2025 Ad Meter ads tested in our published comparison, Uber Eats scored 2.92 out of five in Sapien and 3.38 in the human results. It ranked fifth and second respectively within that tested set. An ad team considering whether to discard that execution would need to investigate the disagreement.
These are ad-liking ratings, not measured sales effects. The case also reports closer scores for other tested ads. Keeping the exception next to the agreement makes the evidence more useful than quoting the favorable part alone.
A different task can expose a different weakness
Our comparison with FCA Financial Lives 2024 estimated 7.91 out of ten for defined-contribution pension providers, against a human average of 6.7. The FCA measure asked holders to rate their own providers on a 0–10 scale and excluded “don’t know” answers. The 1.21-point gap affects which customer experience looks most in need of attention. In that comparison, the simulated averages varied less across the tested product groups.
This is why a good offer-response recreation does not automatically validate a customer-satisfaction ranking. Match the job, population and measured outcome before deciding which evidence supports your project.
The full record stays in its original units
- Read the source population, dates, questions and conditions for the closest case.
- Distinguish observed action, stated preference, model-derived probabilities and qualitative finding matches.
- Inspect the largest consequential miss and the group comparison that could reverse a recommendation.
- Ask how a reported score was calculated and whether the materials needed to reproduce it are available.
The full formulas and inputs behind every reported normalized score are not provided on the case page. The selected paired tables let you inspect practical differences, but they do not justify combining rank agreement, dollar errors, share errors and interview matches into a new universal accuracy percentage.
Open the complete human-comparison evidence for the source links, additional cases and limitations. Our blind-pilot guide explains how to evaluate the work you plan to buy, with the reference answers held back and the rule chosen before scoring.
