Evidence from Hierarchical Choice Models
This research examines a failure mode in hierarchical choice models: respondent covariates can receive strong in-sample support while producing systematic preference structure that fails to generalize to new respondents.
The central question is not simply whether respondent covariates should be included, but how much preference heterogeneity a model should be allowed to attribute to them.
Respondent Covariates Can Lie: Evidence from Hierarchical Choice Models
Jake Lee, Red Analytics, Inc.
September 2026
The primary stress test deliberately broke the relationship between a respondent characteristic and preference, then entered twenty shuffled versions of that characteristic as upper-level covariates.
The conventional, more permissive specification looked extraordinarily successful in sample. But when evaluated on respondents held completely outside estimation, the result reversed:
The permissive model therefore fell 9.498 log predictive-likelihood units per respondent below Base, despite its very strong in-sample performance.
The paper follows this failure through:
The broader principle is earned heterogeneity: systematic preference differentiation should not receive evidentiary credit simply because a model can estimate it. It should be supported by evidence appropriate to the claim being made.
Improving Choice Model Accuracy Using Covariates Depends on What You Hold Out
A separate replication/extension examines the same general problem using a public choice dataset and compares same-respondent task holdouts with genuinely held-out respondents.
Read the machine-readable replication page Read the canonical version on Medium
Three shorter LinkedIn articles discuss related issues in model validation, respondent covariates, and evidence standards.
The paper demonstrates a failure mode; it does not estimate how frequently that failure occurs across hierarchical choice applications.
The regularization approach examined in the paper is one possible response, not a claim that it is uniquely correct. Other regularization structures may control the same problem as well or better.