MaxDiff: decide which benefits deserve development first

MaxDiff asks for the most and least valuable item within repeated subsets to reveal relative priorities in a defined list. Keep wording comparable, use a connected task design and state the score method. Net best-minus-worst counts are descriptive proportions, not fitted utilities or buying probabilities. Use the result to focus development or messaging work; price and product performance require separate evidence.

Choose one decision and comparable items

Imagine you are choosing which meal-planner functions to develop for households arranging dinner. The illustrative task design and counts below show how to prioritize within a fixed list.

Your meal-planner team has more useful functions than development capacity. The list is about value in dinner planning, rather than a mixture of importance, attractiveness and purchase intent. A result measured with one question should be interpreted using that question.

Each statement describes one function at a similar level. “Stress-free family life” would be a whole product promise beside the narrow preparation-time filter and could dominate without explaining what to build. Split compound items and make their mechanism clear before testing the list.

Pilot whether people understand “least valuable.” It means low value among the shown alternatives, rather than harmful or unwanted in all circumstances. A household can consider sharing useful but still select it as least when shopping-list consolidation is more urgent.

Check what every item gets compared with

Use these four subsets: ABC, ABD, ACD and BCD. Each item appears three times; every pair appears together twice. Randomize position within tasks and manage task order. This four-item illustration is small enough to inspect; a real design should consider the full item set, exposure, connectivity and respondent burden.

One best-worst task from the balanced design

Which is most and least valuable when planning your dinners?

For an ABC task, show:

• A: combine ingredients into one shopping list;
• B: save a weekly dinner plan;
• C: filter recipes by preparation time.

Choose one most valuable and a different least valuable item from that displayed set. No response is selected in this example.

The item counts and analysis script let you reconstruct the arithmetic. Sawtooth's official introduction describes best-worst item prioritization.

Copy the task design separately from the counts:

The table specifies sets; randomize presentation positions and manage task order in the instrument.

Four tasks with three items create twelve item appearances per person. Across 100 complete respondents, there are 1,200 item exposures: 300 for each of A, B, C and D. There are 400 best selections and 400 worst selections, one of each per completed task.

The exposure balance makes count comparison easy in this small example. A longer design should also maintain meaningful connected comparisons and manageable burden. Merely showing each item the same number of times does not guarantee that all relevant items are well compared.

Do not always put the same item first within its subset. Position and task order should be managed separately from which items appear together. The saved design specifies sets; the actual instrument and records should preserve their presentation.

Reconstruct the net score and its limits

Every item has 300 exposures under the complete four-task design.

These aggregate counts support the declared net score. For A, (140−40)/300=0.333; B gives 0.233; C −0.200; D −0.367 after rounding. Positive means more best than worst selections within the available exposures. Negative means the reverse. It does not mean negative utility in an independently estimated choice model.

The best counts sum to 400 and the worst counts sum to 400, matching 100 respondents and four tasks. That reconciliation checks arithmetic; it cannot reconstruct individual consistency, task validity or subgroup priorities. An aggregate file does not contain those records.

Sawtooth’s score documentation distinguishes count summaries from modeled scores. A fitted model can account for the task structure and investigate individual differences. Its output needs its own scale interpretation; do not compare a count percentage directly with a model score normalized to 100.

The list defines the competition. A and B lead here among four supplied functions. Adding a crucial missing benefit can change priorities. A high relative score does not establish absolute need, a purchase probability or the price premium for including the function.

Preserve the task-level evidence

An ABC task needs one best and a different worst, both among A, B and C. An unshown code, blank or identical pair fails that complete-task rule. Store participant ID, task ID, shown items, positions and both selections so those checks are possible.

Use actual valid item exposures under the scoring rule. If a participant stops after two tasks, the planned balance no longer guarantees equal observed bases. Do not divide every item by 300 merely because that was the intended schedule.

Declare the treatment of partial completion and invalid tasks before analysis. A complete-case rule and a valid-task rule answer different coverage questions. Show the number of affected people and tasks, especially if exclusions change the roadmap priority.

Retain explanations from a follow-up as a separate layer. They can diagnose why an item mattered, while the best-worst task supplies the comparative choice. A persuasive quote cannot replace the count base, and a count alone cannot tell the team which part of a function to implement.

Turn the top two into product work

A’s lead points to shopping-list quality: duplicate ingredients, units, recipe portions and substitutions must work in an actual shopping journey. B asks whether saving the weekly plan lets people return to it when dinner changes. Those are focused prototype tasks with observable failure points.

Put implementation effort beside the priority. A difficult high-priority function and an inexpensive moderate-priority function can deserve a different release order. Explain the decision in the roadmap without pretending that cost has changed the measured MaxDiff score.

D’s low aggregate result calls for a household check before deletion. Sharing could be important when two adults coordinate meals and unimportant for a solo planner. The aggregates contain no subgroup records, so they cannot establish that distinction. Design a targeted question or inspect appropriate collected records.

If the team needs a subscription tier, translate selected benefits into clear inclusion levels and displayed prices for a separate offer-choice study. A relative rank cannot be converted directly into dollars. A high-priority function may already be expected in every acceptable plan.

Make a bounded roadmap decision

For this decision, concentrate the next prototype work on shopping-list consolidation and weekly-plan usability. Investigate sharing among relevant households before removing it. Defer equal investment in all four until feasibility and use are understood.

The result should travel with the exact list, task plan, score definition and bases. Show why the selected work addresses the leading needs and which uncertainty remains. If a different cost or audience would change the roadmap, keep that tradeoff in the recommendation.

Use ranking when a short list can be meaningfully ordered, conjoint when feasible paid configurations are the question, and observed tasks when delivery of the benefit is uncertain. Confirm design and scoring with the research provider; a modeled study needs task-relevant grounding and validation in addition to a clear instrument.

When the decision is a combination of benefits

TURF stands for total unduplicated reach and frequency. It asks which combination of items reaches the most respondents under a stated rule and can use MaxDiff data. Choosing a set requires knowing which preferences overlap within people. The aggregate counts here cannot show whether A and B serve the same households or whether D adds coverage. Declare what counts as reached; that calculation does not establish observed use or sales.

Copy the counts and analysis

Save the complete CSV as N017-maxdiff.csv and the Python code as N017-maxdiff.py in the same folder, then run python3 N017-maxdiff.py. The standard-library script checks the four-task design, reconciles 400 best and 400 worst selections and prints each net score. The aggregate records do not add respondent-level evidence.

Complete item-count records

item,description,exposures,best_count,worst_count,evidence_type
A,Combined shopping list,300,140,40,fictional_teaching_data
B,Saved weekly dinner plan,300,130,60,fictional_teaching_data
C,Preparation-time filters,300,80,140,fictional_teaching_data
D,Household sharing,300,50,160,fictional_teaching_data

Complete design check and net-score calculation

"""Verify small balanced design and compute fictional net count scores."""
import csv,itertools,collections
from pathlib import Path
tasks=['ABC','ABD','ACD','BCD']
items=collections.Counter(i for t in tasks for i in t)
pairs=collections.Counter(p for t in tasks for p in itertools.combinations(t,2))
assert set(items.values())=={3} and set(pairs.values())=={2}
print('Teaching design:',tasks,'item exposures per respondent:',dict(items),'pair co-occurrences:',dict(pairs))
rows=list(csv.DictReader(Path(__file__).with_name('N017-maxdiff.csv').open()))
assert sum(int(r['best_count']) for r in rows)==400
assert sum(int(r['worst_count']) for r in rows)==400
print('Fictional teaching counts; scores are not estimated utility or probability.')
for r in rows:
    n=int(r['exposures']);b=int(r['best_count']);w=int(r['worst_count'])
    assert b+w<=n
    print(r['item'],round((b-w)/n,6))

For related method choices, consult our MaxDiff-versus-conjoint comparison and our ranking-versus-MaxDiff guide.

Bring the decision, audience and evidence needed to a scoped Sapien study discussion.