Why Conjoint Studies Fail
Conjoint studies can fail because of poor attribute selection, inappropriate respondents, unrealistic choice tasks, or flawed experimental designs. These are real problems, but avoiding them does not guarantee useful results.
The more consequential failures can go undetected. A conjoint study can follow accepted best practices, estimate successfully, produce plausible part-worths, and still be incapable of supporting the business decision. The fundamental distinction is between a study that can be executed and one that produces defensible evidence.
The following six failure modes explain how conjoint studies fail, often despite competent execution.
Failure Mode #0: Allowing the Method to Define the Decision
Zero comes first because it supersedes all the others. A study can be flawless in its experimental design, estimation, and validation yet still fail if it does not support the business decision.
This happens when the research objective quietly replaces the business decision. During scoping, questions are narrowed, trade-offs softened, or outcomes redefined to fit what the method can conveniently measure. The study may accomplish its research objectives while leaving the original business uncertainty unresolved.
The remedy begins before attributes, levels, or experimental designs are specified. Establish what decision the business needs to make, what alternatives it is considering, and what evidence would distinguish among them. Only then should the research objective and methodology be finalized.
This is the distinction between analytic sufficiency and decision sufficiency. Meeting the standards of an analytic method does not necessarily mean meeting the evidentiary requirements of a business decision.
Failure Mode #1: Silent Design Corruption by Automation
Automated design tools can generate experiments that appear well-constructed while failing to provide sufficient information to estimate the effects that matter. Balance is often emphasized even when improving balance comes at the expense of information gain. Automated feasibility restrictions and other adjustments can introduce further problems.
In my consulting practice, I was asked to investigate a conjoint study whose simulator recommended ever-higher prices, with no meaningful price sensitivity. I re-estimated the model, but the problem persisted. Examining the resulting part-worths revealed implausibly large estimates, which I eventually traced to an experimental design that had prioritized balance at the expense of information gain. The problem was not the estimation procedure. It was the experiment.
The practical safeguard is to insist on information-efficient experimental designs and independently examine their statistical properties. A design should be evaluated on whether it provides the information required for the intended analysis, not merely whether it satisfies the software’s design-generation criteria.
Failure Mode #2: Structural Confounds in Experimental Inputs
Identification requires the experiment provide the variation needed to distinguish the effects being estimated. When attributes or levels systematically move together, the model may be unable to separate their contributions.
These confounds are particularly dangerous because estimation can still produce stable-looking part-worths. Hierarchical pooling and regularization may provide numerical estimates even when the underlying experiment does not independently identify the effects being interpreted. The solution is to inspect the experimental design for confounding before collecting data. Important trade-offs must be independently identifiable, especially when the resulting estimates will inform pricing, product configuration, or portfolio decisions.
Without that information, the model may produce answers to questions the experiment never actually asked.
Failure Mode #3: Executability Mistaken for Validity
Successful estimation is not necessarily a valid model. Convergence, clean part-worths, and successful estimation with simulated responses establish that the estimation procedure can execute. They do not establish that the resulting model adequately represents observed choices or generalizes beyond the estimation data.
Hierarchical Bayesian methods can produce stable estimates through pooling and regularization even when important weaknesses remain in the evidence. A technically successful estimation procedure should never be treated as a substitute for model checking. Researchers should report appropriate model diagnostics and measures of fit, including in-sample and out-of-sample performance. When the model is intended to generalize to new consumers, predictive validation should exclude respondents from estimation rather than merely holding out tasks from respondents already used to fit the model.
The essential distinction is simple: a model that can be estimated has not necessarily demonstrated that it can predict.
Failure Mode #4: Power Miscalibration in Hierarchical Choice Designs
Researchers routinely justify the number of respondents without adequately explaining the number of choice tasks each respondent completes.
This matters because the information contributed by additional respondents is not interchangeable with the information obtained from each individual’s choices. Hierarchical Bayesian estimation can borrow strength across respondents, but pooling cannot automatically replace insufficient individual-level information about preferences and trade-offs.
A study may have a seemingly adequate sample size and stable aggregate results while providing inadequate information to understand how individuals make decisions.
Experimental design should ensure that each respondent encounters enough informative choices to support the intended individual-level inference. The number of tasks should be justified by the complexity of the choice problem and the information required, not merely by conventional survey-length targets or whether the model successfully estimates.
More respondents do not automatically fix too little information within respondents.
Failure Mode #5: Substitution of “Importance” for Predictive Evidence
Importance scores summarize aspects of an estimated preference structure. They do not, by themselves, tell a business what will happen when it changes prices, introduces products, modifies features, or restructures a portfolio.
Importance measures depend on how attributes and their levels are specified. An attribute may appear highly important without providing a useful answer to the strategic question facing the business.
The appropriate next step is to use the estimated choice model to simulate the consequences of alternative business decisions. Instead of simply selecting features with the highest importance scores, evaluate competing product configurations, pricing strategies, and market scenarios. Examine the resulting choice behavior and, where relevant, the economic coherence of the predictions.
The goal is not merely to understand which attributes matter. It is to understand what happens when the business acts.
Why These Failures Persist
These problems persist because they often do not look like errors. Studies can follow accepted best practices, produce clean diagnostics, generate stable part-worths, and tell convincing stories. The weaknesses become apparent only when the evidence is asked to support a consequential decision.
The central question is not whether the analysis was performed correctly. It is whether the experiment identified the necessary effects, whether the model demonstrated adequate fit and predictive generalization, whether the proposed strategic alternatives were evaluated appropriately, and whether the evidence was sufficient for the actual business decision.
When a study cannot meet those requirements, the issue may not be execution at all. It may be that the study was never capable of supporting the decision.
That is why apparently successful conjoint studies can fail.
For a broader discussion, see Why Many Conjoint Studies Don’t Lead to Action — And Why That’s Predictable and Conjoint Analysis Rescue.
