A common best-practice claim in conjoint analysis is that asking too many choice tasks leads to satisficing and degraded data quality. However, a 2018 peer-reviewed study in Political Analysis experimentally varied task count (including conditions with ~30 tasks) and found that aggregate attribute effects remained stable within the tested designs. This result does not establish an optimal task count and does not evaluate individual-level reliability or design power.
What the 2018 study tested
What was studied
- The paper experimentally varied the number of choice tasks per respondent, including conditions with dozens of tasks (up to ~30).
- The primary outcome was stability of aggregate attribute effects (AMCEs), not individual-level reliability or prediction accuracy.
- The data came from standard online samples (e.g., MTurk / SSI).
What they found
- Aggregate attribute effects remained remarkably stable as task count increased.
- Increases in satisficing indicators were detectable but limited, and did not materially distort core aggregate effects within the tested range.
What the paper does not show
This study does not establish:
- an “optimal” number of tasks,
- that satisficing never occurs,
- that individual-level utilities remain reliable at high task counts,
- or that high-dimensional designs are sufficiently powered by default.
Its conclusion is narrower: Within the tested designs, increasing task count did not cause collapse of aggregate attribute estimates.
How this contrasts with common best-practice advice
In contrast, much practitioner guidance states or implies that:
- pre-evidence claim: task counts above ~10–15 are generally unsafe,
- pre-evidence claim: satisficing becomes a dominant threat beyond that range,
- pre-evidence claim: fewer tasks are preferable unless respondents are “highly motivated.”
Notably, this guidance often:
- cites the 2018 paper (or related work),
- but presents it as an exception rather than as evidence that the fear of task-count-driven collapse may be overstated—at least for aggregate effects.
This contrast suggests that the disagreement is not really about whether satisficing exists, but about what constraint should govern conjoint design decisions. Much of the confusion arises when respondent burden is treated as the primary limiting factor, rather than the strength of evidence required for the decision at hand.
Why this mismatch persists
The discrepancy appears to stem from three factors:
- Different estimands
The academic result concerns aggregate effects; best-practice advice often targets individual-level reliability and respondent experience. - Heuristics standing in for power
Task caps are frequently justified by fatigue narratives rather than by explicit analysis of evidentiary sufficiency. - Modeling practices that mask insufficiency
Hierarchical Bayesian models tend to produce stable estimates even when information is weak, which can make underpowered designs appear well-behaved.
The correct takeaway
The appropriate conclusion is not that “more tasks are better,” nor that task limits are unnecessary.
It is simply this: Empirical evidence does not support the claim that increasing the number of choice tasks automatically degrades aggregate conjoint estimates due to satisficing, within the ranges tested.
Design decisions still depend on:
- the number of parameters,
- the target estimand,
- and the level of inference required.
Discussions of task count are more productive when framed around evidentiary sufficiency and estimands, rather than assuming fatigue-driven failure as a default.
This tension between empirical robustness and practitioner caution points to a deeper design principle. Conjoint design quality depends on aligning design complexity with the strength of evidence required for the decision context, rather than adhering to fixed task or attribute limits.
Bansak K, Hainmueller J, Hopkins DJ, Yamamoto T. The Number of Choice Tasks and Survey Satisficing in Conjoint Experiments. Political Analysis. 2018;26(1):112-119. doi:10.1017/pan.2017.40