Silicon sampling: inspect the evidence behind the recommendation
Read silicon sampling evidence within its model, task and paper version. Reproduce fictional prompt sensitivity and request decision-relevant validation.
Silicon sampling uses language models to generate responses intended to represent a human population. If you are choosing which concept to develop, ask whether the model reproduces the human patterns that matter to that choice. Inspect task-matched validation, subgroup coverage, a named baseline and sensitivity, meaning whether justified changes to the inputs change the recommendation. Plausible language or many generated rows cannot establish that the choice transfers to real buyers.
What the cited studies establish and what you should still ask
Research | Details |
|---|---|
Argyle and colleagues, Out of One, Many, published 2023 | Construction and evaluation described by the authors: GPT-3 conditioned on sociodemographic backstories from participants in US surveys; silicon samples compared with human response patterns Scope to preserve: Evidence about that conditioning, model and social-science context, rather than every topic or model |
Construction and evaluation described by the authors: Interview-grounded agents representing 1,052 people, evaluated on survey responses and other tasks Scope to preserve: Individual interview grounding and the stated evaluation tasks; the reported 85% GSS figure is normalized to human retest performance |
Argyle and colleagues studied GPT-3 conditioned on sociodemographic backstories from U.S. survey participants and compared human response patterns, including subgroup patterns. The result concerns that model, construction and social-science context. It does not establish that a demographic persona alone can answer any product question in any market.
Park and colleagues’ linked November 2024 preprint used qualitative interviews to ground agents representing 1,052 people. Its reported 85% GSS figure is normalized to each person’s own two-week human retest performance. It should not be relabeled “85% correct” for arbitrary marketing questions. Later revisions exist, including June 2026; the historical claim here is bound to the explicitly linked v1.
The constructions differ in the evidence supplied about people and the tasks used for evaluation. They are design references for asking better validation questions, rather than one transferable accuracy score. Neither study evaluated Sapien. Ask which construction, audience and task resemble the particular comparison you would commission.
Chen, Zhu and Zheng’s 2026 preprint compares demographic prompting with non-LLM baselines on GSS and WVS data. Its baselines use a respondent-separated fit/evaluation split and include a question-marginal rule, demographic lookup, logistic regression and random forest. Aggregate agreement can coexist with errors about individuals and subgroup differences. Use this to request separate validation for distributions and targeting decisions. The tested protocols do not evaluate Sapien.
Chidichimo and colleagues’ 2026 preprint organizes eleven tests around response processes, relationships among measures, and agreement with human distributions, groups and experimental effects. It compares nine models across twenty human sources and retains unavailable estimates. For segmentation, ask whether validation preserves the group differences and relationships your decision needs, rather than accepting a matched overall average. This framework concerns its tested configurations and sources; it does not evaluate Sapien.
Write a validation brief around the decision
Imagine your team must choose which of two complete product offers to develop for a defined eligible audience. Define the eligible audience, market, exact cards, exposure and primary response before evaluating a model. “Sounds realistic” is insufficient; the relevant question is whether its recommendation matches suitable human evidence and whether its errors are large enough to change development.
Use a held-out human reference appropriate to the task. Save how the humans were recruited, what they saw and which answers were valid. Preserve the difference between a representative population claim and a comparison in a specified recruited sample. A reference with thin coverage of the decisive segment cannot validate that segment simply because the overall result agrees.
Keep evaluation items out of construction where the design requires a genuine test. If the same answers supply the grounding and the scored reference, reproducing them can reward access to the solution. Name the information allowed in construction, the evaluation withheld, and any tuning performed after seeing a result.
Specify a simple named baseline as well. For a binary concept task, a reference-informed majority rule may be a useful comparator when constructed from permitted training information. For another task, a simpler specified statistical model may be more relevant. Choose and freeze the baseline before inspecting held-out results; it must obey the same information boundary.
Select metrics that express the decision. Inspect the complete response distribution and proportion errors for a share task, ranking and recommendation reversals for concept selection, and important or contradictory themes for open responses. A favorable overall average can conceal the particular error that redirects development.
A buyer’s validation record to request
Record field | What should be inspectable |
|---|---|
Task and decision | Audience, stimuli, instrument, exposure, primary measure and action the result would support. |
Human reference | Recruitment, field date, valid bases, important subgroup coverage and known limitations. |
Model construction | Version, grounding inputs, prompts, configuration, permitted information and preparation date. |
Evaluation | Held-out boundary, named baseline, prespecified metrics, invalid-output handling and all tested conditions. |
Recommendation | Where errors change the action; which audience/tasks are supported; unresolved evidence and failure response. |
Reproduce the complete five-condition exercise
The example CSV and standard-library Python summary reproduce the five conditions described below. The entire dataset, including its “human reference,” is illustrative. Every row has 100 valid answers; no human recruitment or model run took place.
The reference gives A 52 answers and B 48. Prompt 1/profile 1 gives A 60. Changing only the prompt gives A 45, so B becomes preferred. Changing the profile under prompt 1 ties the concepts at 50–50. Repeating the baseline gives A 58. Preserve all five conditions; selecting the attractive first run hides the choice reversal.
To adapt the exercise, hold the instrument and stimuli fixed. Use two justified neutral prompt formulations, two justified profile constructions and a repeated baseline. One candidate formulation is to answer the instrument using the supplied profile; another explicitly asks the model to use only the profile and stimulus while preserving unknown information. These are proposed sensitivity contrasts, not an established validation protocol.
Changing a prompt should not quietly change the offer, add a favorable fact or tell the model which concept to prefer. A justified contrast probes a defensible way of representing the same task. Similarly, compare profile constructions supported by the permitted grounding rather than adding convenient stereotypes. Save the exact text and inputs so a reversal can be diagnosed.
Read the contrasts without averaging away the reversal
The reference favors A by four percentage points on A-versus-B share: 52−48. For A’s share alone, the distance to the 50% choice boundary is two points. The prompt contrast moves A from 60% to 45%, a fifteen-point swing that crosses that boundary. An average over those two conditions would conceal the fact that the recommendation depends on construction.
Relative to the reference, A’s share errors are +8 points for the first construction, −7 for the changed prompt, −2 for the changed profile and +6 for the repeated baseline. These are direct arithmetic contrasts from the five-condition dataset. They are not confidence intervals, estimates of population error or evidence that one prompt is correct.
The repeated baseline changes A from 60% to 58%. That single repeat gives one observed contrast within the chosen construction. It cannot characterize the complete distribution of run variability or validate the audience profile. The profile change addresses a different dependency from the repeat; keep them separate.
For a real concept decision, interpret variation against the action at stake. A small ranking error may be inconsequential when both concepts clearly meet the next-stage gate; a small shift can be decisive near a boundary. Prespecify the useful difference and failure response using commercial context. Set the tolerance for your decision rather than borrowing it from this example.
Make failure informative before looking at results
Agree what would prevent progression: a justified construction reverses the leading concept, a decision-critical subgroup diverges from the appropriate human reference, or too many outputs cannot answer the task. Define whether that outcome triggers better grounding, a narrower scope, revised construction or direct human evidence.
Preserve refused, malformed, contradictory and unanswerable outputs with their disposition. A clean percentage among surviving responses can hide poor coverage. Report attempted outputs and valid outputs alongside the planned measure, without treating generated counts as independent human sampling precision.
Do not choose the run nearest the held-out reference after seeing every result and report it as untouched validation. If construction is tuned using that evaluation, record the tuning and use a suitably fresh evaluation for the revised claim. A buyer need not build the model, but can require this boundary in the report.
Inspect important missing coverage before buying more generated rows. More rows may reduce some within-construction random variation while systematic prompt, grounding or measurement error remains. If the simulated population lacks evidence about a niche purchase situation, volume alone cannot supply that experience.
Use the sensitivity result to decide whether development should proceed
In this exercise, do not progress A solely because the first modeled condition gives it 60%. Investigate why the justified prompt contrast changes the choice, and use the appropriate human evidence to resolve the remaining comparison. The example reference illustrates the desired validation structure; it cannot settle a real product decision.
For a commissioned study, ask the provider to show the task-matched validation and sensitivity record alongside the recommendation. Our managed research service is a scoped, data-grounded modeled-population comparison; the proposal should identify which grounding and validation support the intended inference. Keep any unsupported segment or physical-use claim outside the completed conclusion.
A successful later evaluation supports the audiences, tasks and construction changes actually examined. Use it to decide what the modeled comparison can responsibly settle, then name the next evidence required for launch, behavior or an uncovered buyer situation. That is a useful research handoff even when the immediate recommendation is to gather more targeted human evidence.
Copy the five-condition data and summary
Save the complete CSV as N080-silicon-sensitivity.csv and the Python code as N080-summarize-sensitivity.py in the same folder. Run python3 N080-summarize-sensitivity.py to print A’s valid share, its percentage-point difference from the illustrative reference and the invalid count for every condition. You can also pass a compatible CSV path as the first argument.
All five condition records
status,condition,a_chosen,b_chosen,invalid
fictional teaching data,human-reference,52,48,0
fictional teaching data,prompt1-profile1,60,40,0
fictional teaching data,prompt2-profile1,45,55,0
fictional teaching data,prompt1-profile2,50,50,0
fictional teaching data,prompt1-profile1-repeat,58,42,0
Complete sensitivity summary
"""Summarize the adjacent fictional teaching CSV, or pass a compatible CSV path."""
import csv
import sys
from pathlib import Path
path = Path(sys.argv[1]) if len(sys.argv) > 1 else Path(__file__).with_name('N080-silicon-sensitivity.csv')
with path.open(newline='') as handle:
rows = list(csv.DictReader(handle))
shares = {}
for row in rows:
a, b, invalid = (int(row[key]) for key in ('a_chosen', 'b_chosen', 'invalid'))
if min(a, b, invalid) < 0 or a + b == 0:
raise ValueError('Counts must be nonnegative and include at least one valid response')
shares[row['condition']] = 100 * a / (a + b)
reference = shares['human-reference']
for row in rows:
share = shares[row['condition']]
print(f"{row['condition']}: A={share:.2f}%; difference={share-reference:+.2f} percentage points; invalid={row['invalid']}")
Further reading
Bring the decision, audience and evidence needed to a scoped Sapien study discussion.
