Synthetic research reliability: What if the winner changes on a rerun?
A bike-pass comparison shows how to read reversals by segment and decide which offers to carry into human research.
If the winner changes when you repeat a synthetic study under the same conditions, keep both offers in play. Check whether the reversal crosses the margin you agreed would change the decision and which buyer groups account for it. If you still need to pick one offer, resolve a meaningful reversal with evidence from the people who would make that choice.
Decide what would count as a repeat
Fix the audience definition and source data, the offer cards, question wording and order, answer options, response counts by segment, and rules for unusable outputs. Record the study protocol, model or service version, and any generation settings the provider can confirm. Use fresh executions under those conditions. Where a material setting is unavailable, mark the repeat conditions as partly verified.
A deliberate change to wording, audience inputs, or model version answers a sensitivity question. Run it as a separate condition so you can see which change moved the result. If a supposed repeat used a different offer card, correct its label and keep it in the readout. Our silicon sampling guide
Agree on the margin before the first run
Consider a bike-share team that needs to choose one weekday pass for a field trial. Pass A is $18 for three rides. Pass B is $24 for five. Both cover weekday rides of up to 30 minutes, expire after 30 days, and require a fresh purchase to renew. Riders can also keep paying $8 per ride under the existing terms. The protocol, prices, and responses below are illustrative.
The study asks occasional riders, who took one to three weekday rides in the prior month, and regular riders, who took at least four, to choose a pass or keep buying single rides. The team gives both groups equal weight in the combined comparison. Regular riders are the priority group for the trial.
The team plans three runs. To shortlist a pass by itself, it must lead the next most chosen option by at least eight percentage points overall and among regular riders in every run. A smaller lead keeps both offers open. If two runs show leads of at least eight points in opposite directions overall or in the priority group, take both passes into a human comparison. The eight-point margin reflects how large a difference the team would need to drop a pass from the shortlist.
Put every run in the readout
Each run generates 100 attempted choices per segment, all valid in this example. Entries show counts for Pass A, Pass B, and single rides, in that order. The combined lead uses all 200 responses in each run.
Run and rider group | A / B / single-ride choices, n = 100 |
|---|---|
Run 1 · occasional | 48 / 41 / 11 |
Run 1 · regular | 51 / 39 / 10 |
Run 2 · occasional | 46 / 43 / 11 |
Run 2 · regular | 34 / 56 / 10 |
Run 3 · occasional | 47 / 42 / 11 |
Run 3 · regular | 43 / 47 / 10 |
In run 1, A gets 99 of 200 choices and B gets 80. The gap is 19 divided by 200, or 9.5 percentage points. Among regular riders, A leads by 12 points. In run 2, B gets 99 choices to A’s 80 overall and leads by 22 points among regular riders. Both runs clear the margin in opposite directions. In run 3, A leads by 0.5 points overall, which is too close to shortlist a pass. Keep both offers in play and include regular riders in the human check.
Use the same readout for your own comparison. Keep attempted and usable counts, exclusions, and exact conditions with each row. Report segment results even if the combined ranking looks steady. A change in the group that drives the decision can matter more than a stable result overall.
Use human evidence to resolve the choice
Repeatability asks whether the decision holds under the same protocol. Sensitivity asks how it changes when an input changes. Accuracy requires an independent comparison suited to the decision. The same audience inputs can produce the same error in every run. NIST AI Risk Management Framework
Show eligible riders the same pass terms and the single-ride alternative. Agree on a meaningful lead and how you will handle an inconclusive result before reviewing answers. If paid uptake is the decision, measure actual purchases in the field. Keep the human outcome separate from any adjustments to the synthetic setup so it can serve as an independent check. Our guide to synthetic research validation
