Leading and double-barreled questions: repair what the survey measures
Repair leading and double-barreled questions with twelve fictional examples, balanced options and a fee-comprehension task you can inspect.
A leading question supplies the judgment you hope to hear. A double-barreled question asks about two things but allows one answer. Repair either by deciding what you need to learn: splitting “affordable and easy to use” gives price and experience separate decisions, while removing “love our convenient refill” lets the buyer supply their own impression. The twelve illustrative examples below show how to make that repair.
A loaded question assumes something the respondent has not established. “Why do you buy sustainable products?” assumes that purchase behavior before asking for a reason. Check the behavior first, then ask what influenced it. A question can be loaded and leading at the same time; removing praise does not repair an unverified assumption.
Compare each original question with its repair
Each revision includes a reason you can use in your own instrument. For the fee task, imagine you sell refills to household buyers and need to check whether they understand the complete order price.
Replace a moral agreement item with an order task
Original question: Do you agree that nobody should have to pay hidden fees?
Order receipt
2 refills × USD 8 = USD 16 before tax
1 delivery charge per order = USD 3 before tax
Total before tax: USD 19
What total would you expect to pay before tax for this order?
Open amount, explanation and Cannot judge route.
The receipt shows the calculation for you. In fieldwork, show the unit price, quantity and per-order fee, then collect the total before supplying it.
1. “How much do you love our convenient new refill?”
Ask “What is your overall impression of the refill described?” Use very unfavorable / somewhat unfavorable / neither / somewhat favorable / very favorable / cannot judge. The original supplies both a favorable emotion and a benefit before collecting the response.
2. “Is the service affordable and easy to use?”
Split it: “How affordable or unaffordable is the price shown for your circumstances?” and “How easy or difficult was it to complete the task?” A person may find the price manageable while struggling with the interface.
3. “Wouldn’t you agree the premium package is better?”
Show comparable packages and ask which the participant would investigate further, including neither and no preference. Follow with “What led you to that answer?” The original asks for agreement and leaves “better” undefined.
4. “Why do you buy sustainable products?”
First ask about a recent category purchase and what influenced it. Only ask a sustainability follow-up if relevant. The original assumes both the behavior and the reason.
5. “How satisfied are you with delivery and customer support?”
Ask separately about the most recent delivery and, for those who used support, the support interaction. Add a clear not-applicable route. Someone may have received a late order but excellent help resolving it.
6. “Do you agree that nobody should have to pay hidden fees?”
If testing a fee disclosure, show the actual offer and ask what total cost the participant expects, then what information is unclear. General agreement with a moral statement does not establish comprehension of your price.
7. “How often do you regularly use the app?”
Ask “On how many days in the past seven days did you use the app?” Use 0 through 7 and do not know. The defined period and count replace an ambiguous qualifier.
8. “Which useful feature should we add next?”
Ask which, if any, of the described features would be useful for the defined task, with none and cannot judge. If prioritization is required, use a specified ranking or choice design. The original assumes every feature merits development.
9. “Are you happy with the speed and accuracy of recommendations?”
Measure response time or perceived speed separately from relevance or correctness. Define the task and evidence available. One favorable response cannot reveal whether fast but irrelevant recommendations are acceptable.
10. “Why did our ad persuade you to buy?”
Establish whether the person saw the ad and bought, then ask what influenced the decision without assigning causation. A reported influence is different from an experimentally measured effect.
11. “Do you oppose not offering a discount?”
Show the actual price alternatives and ask for consideration or choice under the specified offer. If asking a policy opinion, use a direct favor/oppose question. The double negative makes response direction harder to interpret.
12. “Compared with unreliable competitors, how trustworthy is our brand?”
Remove the unsupported competitor description. Ask about trust in each defined offer with equivalent information, or ask what creates or reduces trust. A comparative insult changes the stimulus before measurement.
The answer labels are part of the measure
A neutral question can still have leading response options. “Good / very good / excellent” excludes negative and neutral views. Overlapping frequency categories make it unclear where some respondents belong. “Other,” “none,” “do not know” and “not applicable” serve different purposes and should be added when the question needs them.
Keep ordered scales in order. Randomize unordered alternatives when the design calls for it, and preserve the display order in the data. Pew’s question-writing guidance explains why wording and question sequence affect interpretation.
Unknown, none and not applicable are different answers. A person without the relevant experience needs a route that does not force a fabricated judgment; a person with an unfavorable view needs an actual unfavorable option. Pretest those labels as well as the sentence.
A complete offer makes the neutral question useful
Replace “Do you agree that nobody should have to pay hidden fees?” with a shown offer: two refills at USD 8 each and one USD 3 per-order delivery charge, both before tax. Ask: “What total would you expect to pay before tax for this order?” Record the amount and explanation, with Cannot judge available.
The expected total is USD 19 before tax. If a response counts one refill, inspect quantity; if it applies delivery per refill, inspect fee scope; if the amount is right for the wrong reason, preserve that contradiction. Complete terms and reasoning make the answer useful. This is a hypothetical diagnostic, not a measured improvement.
The expected amount and the calculation answer different diagnostic questions. A correct USD 19 before tax reached by adding the wrong quantity or fee rule is still an interpretation problem. Preserve the reasoning so the team can repair quantity, unit or per-order scope.
The moral-agreement original cannot supply any of those facts. The replacement succeeds as an instrument when the offer is complete and respondents can show what they understood, not when internal reviewers think it sounds more neutral.
Pretest the two routes a split question creates
Splitting “affordable and easy to use” creates two measures. The affordability question needs a complete price offer; task ease needs the relevant attempted task or an explicitly labeled expectation. An unexposed person cannot report an experience they never had.
Ask respondents to restate each question, explain which information they used and describe the chosen answer. Repair wording, stimulus, routing or response labels at the stage causing the mismatch, then save the change in the reviewer record. Save the original and replacement so the next review checks the actual revision.
Finish the review with the intended decision
Does each question measure one defined thing? Is its wording neutral? Does the respondent have the offer or experience needed to answer? Do the options allow plausible unfavorable, unknown and inapplicable answers? Will the result tell the team what to change?
Apply the same neutral wording to modeled responses and disclose profile/context assumptions. Use our Likert scales guide for balanced labels and our silicon sampling guide for prompt sensitivity. Bring the instrument and decision to Sapien to scope a modeled survey or comparison.
Use the question review records for the material change, not as a paperwork exercise. In a tracker, bridge or identify a break when wording changes the measure. The final review should say which business answer the revised item can supply and whether the response route supports it.
Copy the question-review records
The two supplied records retain the original question, intended measure, failure, replacement, response options and rationale. Pilot notes are blank in both; add actual pretest observations when you have them.
Review record 1
Field | Recorded value |
|---|---|
| How much do you love our convenient new refill? |
| Overall impression |
| Leading benefit and emotion |
| What is your overall impression of the refill described? |
| Very unfavorable to very favorable plus cannot judge |
| Permits unfavorable and neutral reactions |
|
Review record 2
Field | Recorded value |
|---|---|
| Is the service affordable and easy to use? |
| Price fit and task ease |
| Double barrel |
| Ask affordability and task ease separately |
| Separate balanced labels for each question |
| Price and ease can differ |
|
Copy the complete question-review CSV
original,intended_measure,failure,replacement,response_options,rationale,pilot_notes
How much do you love our convenient new refill?,Overall impression,Leading benefit and emotion,What is your overall impression of the refill described?,Very unfavorable to very favorable plus cannot judge,Permits unfavorable and neutral reactions,
Is the service affordable and easy to use?,Price fit and task ease,Double barrel,Ask affordability and task ease separately,Separate balanced labels for each question,Price and ease can differ,
