Root Likelihood Research Articles

Measurement and evaluation of participant response consistency in Case 1 Best-Worst-Scaling (BWS) in food consumer science

Best-Worst Scaling (BWS) is well suited to survey research, and since this way of collecting data is gaining popularity, it is likely that applications of BWS, especially online, will increase. Despite the many advantages of online surveys, there is a growing awareness that particular attention needs to be given to data quality including the identification and elimination of ’bad respondents’. Post hoc data cleaning is the focus of the present research, which in the context of Case 1 BWS (object case) was directed to two indices of participant response consistency – root likelihood (RLH) and normalised error variance (ErrVarNorm), where the latter was newly developed. Across 18 studies in food consumer science, the two indices are applied, compared and evaluated. It is possible to apply both indices to the data, but if only a single index is to be used, the ErrVarNorm index is recommended because it is easy to calculate, directly measures response consistency and is logically coherent. ErrVarNorm ranges from 0 to 1, where higher values indicate greater response consistency. When excluding participants based on ErrVarNorm < 0.3, between 0 % and 12.4 % of participants were excluded (mean = 5.75 %), while ErrVarNorm < 0.5 led to between 0 % and 23.8 % of participants being excluded (mean = 13.1 %). Excluded participants were more likely to be men and below the age of 45 years old. The impact on study conclusions when excluding participants based on ErrVarNorm < 0.5 were illustrated for three studies in support of the importance of data cleaning.

The Sensitivity and Specificity of Repeated and Dominant Choice Tasks in Discrete Choice Experiments

ObjectivesThis study aimed to identify the most commonly used internal validity tests in the discrete choice experiment (DCE) literature and establish their sensitivity and specificity. MethodsA structured literature review of recent DCE articles (2018-2020Q1) published in the health, marketing, transport economics, and environmental science literature was used to identify commonly used internal validity tests. The 2 most frequently used internal validity tests were incorporated in 4 new data collections. Respondent preferences in each application were summarized using a mixed logit model, which served as the benchmark for the subsequent sensitivity and specificity calculations. The performance of the internal validity tests was also compared with that of the root likelihood (RLH) test, which is a likelihood-based statistical validity test that is commonly used in marketing applications. ResultsDominant and repeated choice tasks were most commonly included in health-related DCE designs. Based on 4 applications, their specificity and sensitivity depend on the type of incorrect response pattern to be detected and on design characteristics such as the number of choice options per choice task and the number of internal validity tests as included in the experimental design. In all but one scenario, the performance of the dominant and repeated choice tasks was considerably worse than that of the RLH test. ConclusionsDominant and repeated choice tasks are unreliable screening tests and costly in terms of statistical power. The RLH test, which is a statistical test that does not require additional choice tasks to be included in the DCE design, provides a more reliable alternative.

PNS99 Using an Online Survey to Elicit Preferences in China: Comparison of Different Methods Based on SF-6DV2

To compare the performance of 3 different elicitation methods in an online survey environment in China. An online panel of general population in China participated in a web survey. Each participant was randomized to complete two out of three preference elicitation technique tasks, which were 8 random Discrete Choice Experiment (DCE) tasks, 6 random Time Trade Off (TTO) tasks, and 10 random Best Worst Scaling (BWS) tasks. Ordinal method results were self-anchored on the death values to estimate the utilities. Models assuming homogeneous preference (Conditional Logit and linear regressions (OLS)) and preference heterogeneity (Hierarchical Bayes and Mixed Effects Linear Hierarchical Bayes) were used. The number of significant levels, percent certainty and Root Likelihood (RLH) were employed for the comparison. A total of 961 participants (40.6% <40 years, 49 % female, 76.4% healthy) who passed the quality checks were utilized in the analysis. Based on the number of significant levels, percent certainty and RLH, the BWS homogenous preference model was the best (0, 30.5%, 0.39) compared to DCE (6, 20%, 0.38) and TTO (17, 26%, 0.27). The worst state (PITS) was valued at 0.17, -0.69 and -0.033 respectively. The TTO heterogenous preference model had a whole dimension insignificant. Between BWS and DCE, the preferred model specifications changed for the DCE and both had none insignificant levels. The Percent certainty was 70 & 55.2, RLH at 0.58 & 0.61 and PITS valued at 0.197 and 0.359 for BWS and DCE models respectively. Using online DCE or BWS to elicit preference in china is feasible while TTO requires further optimization. Data from ordinal methods modeled by heterogenous preference were more consistent than the one from the cardinal method (TTO) or modeled assuming homogenous preference. Cardinal methods still perform worse than ordinal methods in online surveys and generates less reliable results.

PNS248 A Hierarchical Bayes Approach in Exploring Individual Heterogeneity: An Application on Health State Valuation Using a Discrete Choice Experiment

Hierarchical Bayes (HB) is widely used in market research to estimate choice models and generate respondents’ utilities. HB is known for the advantage of including individual heterogeneity when evaluating preferences. Herein we demonstrate the usability, advantage and performance of HB in estimating health preference utilities for the Short Form 6 Dimension version 2 (SF-6Dv2) from an online Discrete Choice Experiments (DCE). An online panel of general population in China participated in a web survey. Each participant was randomized to complete 8 DCE tasks, in which a pairwise choice was firstly made between two health states based on SF-6Dv2, followed by a comparison between the worse of them and death. This second task further allows the direct anchoring of the latent utilities onto the 0-1 death-full health scale. Socio-demographic characteristics and self-assessed health, quality of life were also collected. Both conditional logit model (CL, which assumes a homogeneous preference) and an HB model that allows for preference heterogeneity were used. Monotonicity, the number of significant levels and other model fit statistics were employed for the comparison. A total of 447 participants (41.6% <40 years old, 51.5% female, 69% healthy) who passed the quality threshold were utilized in the analysis. The CL model weighted by trading life had the best fit with 3 inconsistencies, 6 insignificances, and a Root Likelihood (RLH) statistics of 0.57. The HB model also had 3 inconsistencies, one insignificant level, and an RLH of 0.73. The ‘Pits’ state utility was estimated to be 0.44, 0.24 and 0.06 for respondents preferring painful life over peaceful death, neutrals and first group opponents, respectively, based on the HB model. The HB demonstrated a better fit and a more realistic model for estimating utilities than CL. It had the power to elicit the differences preferences between subgroups based on different choice behaviours.

Is what you choose what you want?—outlier detection in choice-based conjoint analysis

Choice-based conjoint (CBC) analysis has long been a popular technique in market research. Because CBC is dependent upon respondents’ stated preferences, respondent variability should be taken into account in part-worth estimation. In the spirit of Bayesian residuals within the probit framework, this paper proposes a novel respondent variability measure for CBC, called the “utility deviation” (UD), to detect outliers who have unusually high respondent variability. UD is constructed based on the standardized deviation between a respondent’s true and representative utilities on the made choices. We compare UD with the largest absolute realized deviation (LARD) statistic and the typically used metric, root likelihood (RLH), in the performance of outlier detection using simulated and empirical data. The results show that UD performs slightly better than LARD and significantly outperforms RLH. Finally, we show that performing outlier detection to exclude misleading data can significantly improve the quality of estimation and resultant applications.