Types of research bias: spot the distortion before choosing the action
Locate selection, wording, processing and interpretation bias, then repair the audience, measure or claim that could change your next decision.
Imagine you are choosing a feature for a subscription product. In this illustrative study, eight of ten active-user responses prefer the displayed feature. Calling that “80% customer demand” changes both who the result describes and what the answer measured. Locate the distortion so you can repair the decision, rather than simply buy a larger study.
Distinguish the bias that enters each stage
Selection and coverage bias concern who can enter the study and who is included. An active-only survey misses low-use and canceled experiences. Nonresponse bias arises when people who answer differ in decision-relevant ways from invited people who do not.
A truthful count can produce a distorted headline
● ● ● ● ● ● ● ● ○ ○
8 favorable / 10 active-user responses
Frame: selected active users. Absent: relevant low-use/canceled experiences. Measure: preference under a displayed feature, not demand.
Filled circles denote eight favorable responses; outlined circles are the two remaining responses, not two verified opponents.
Wording and order bias concern how questions or preceding information shape answers. Processing bias concerns how coding, exclusions or missing values alter the result. Interpretation or confirmation bias concerns which findings are promoted, ignored or generalized. Locate each problem in the workflow before changing the study.
Use the stage of the problem to choose a repair:
Risk | Repair to make and uncertainty to retain |
|---|---|
Selection bias | Instead of asking active subscribers why people cancel, include canceled and retained experiences. Check whether unreachable cancelers differ. |
Coverage bias | An online-only route can omit people rarely online. Assess coverage, choose suitable contact routes and identify who remains outside the frame. |
Leading wording | Replace “How much do you love the easy new refill?” with “What is your first reaction to this refill?” Check whether the stimulus still promises ease. |
Double question | Replace “How clear and convincing is this message?” with separate clarity and credibility questions. Check how respondents interpret each label. |
Order | Collect unaided appeal before listing benefits. Earlier context can still influence later answers. |
Recall bias | Replace exact-use questions about an unspecified year with a recent period and room for uncertainty. Memory still differs from recorded behavior. |
Confirmation bias | Preserve objections and contrary cases; predeclare coding rather than keep only positives. Analyst interpretation remains open to challenge. |
Model dependence | Repeated synthetic agreement is not independent confirmation. Examine grounding and validate with another source; shared training or prompting assumptions may remain. |
Also check nonresponse bias: invited cancelers may respond less often than retained customers. Compare known invitation-frame characteristics and disclose what is unknown; more reminders do not prove the missing experiences match. Check processing bias: coding every blank as dissatisfaction or dropping inconvenient valid responses changes the result. Preserve missing values, original responses and an exclusion log. Leading wording and double questions are measurement risks even when recruitment is sound.
For questionnaire wording and order, consult Pew Research Center’s survey design guidance. Pretesting should investigate what people believe the question asks, not merely whether they can click an answer.
The types matter because the remedies differ. Recruitment can add missing experiences; wording can remove a cue; processing can preserve valid disagreement; interpretation can narrow a headline. No single remedy supplies all four.
Repair the active-user study at three points
Who enters: define the customer population the feature is intended to serve. Include relevant low-use or canceled experiences rather than recruiting only enthusiasts. Examine known invitation-frame differences and retain what is missing.
What is measured: preference under the description is not use or payment. Keep that preference result if it helps select a candidate, then offer a prototype task or buying opportunity for the consequential outcome.
What is claimed: the valid exercise statement is eight of ten active-user responses preferred the displayed feature. A narrower headline is a practical repair when the audience cannot yet support a broader one. Weighting observed attributes cannot automatically reconstruct absent reasons.
The finished brief joins those repairs: a relevant customer frame, neutral task/offer exposure, a declared preference measure and a prototype evaluation. It can change the feature decision without pretending the existing count was meaningless.
Use ambiguous records to test the recommendation
A blank answer, contrary comment or ambiguous code can change the action. Preserve the original and compare defensible handling rules. If the recommendation reverses, report that fragility and investigate the uncertainty instead of choosing the favorable coding.
For modeled responses, vary a meaningful grounding or scenario assumption and record it. Stability under those choices is useful, but repeated agreement is not evidence independent of the model. Match the next check to the distortion that matters for the decision.
What to put beside the marketing headline
Show audience, exact question or stimulus, valid base and the reason the result changes an action. Then ask which relevant experience is absent and whether the original answer supports the proposed conclusion.
A bias review is useful when it chooses a repair. Do not make the report a list of hypothetical problems disconnected from the feature, message or audience choice. The active-user example ends with broader relevant access, truthful preference reporting and a real task for the next claim.
Bring the audience frame, questions and disputed conclusion to an evidence and validation discussion with Sapien.
