Statistical Analysis of List Experiments

The item count technique is a survey methodology that is designed to elicit respondents’ truthful answers to sensitive questions such as racial prejudice and drug use. The method is also known as the list experiment or the unmatched count technique and is an alternative to the commonly used randomized response method. In this article, I propose new nonlinear least squares and maximum likelihood estimators for efficient multivariate regression analysis with the item count technique. The two-step estimation procedure and the Expectation Maximization algorithm are developed to facilitate the computation. Enabling multivariate regression analysis is essential because researchers are typically interested in knowing how the probability of answering the sensitive question affirmatively varies as a function of respondents’ characteristics. As an empirical illustration, the proposed methodology is applied to the 1991 National Race and Politics survey where the investigators used the item count technique to measure the degree of racial hatred in the United States. Small-scale simulation studies suggest that the maximum likelihood estimator can be substantially more efficient than alternative estimators. Statistical efficiency is an important concern for the item count technique because indirect questioning means loss of information. The open-source software is made available to implement the proposed methodology.

Due to the inherent sensitivity of many survey questions, a number of researchers have adopted an indirect questioning technique known as the list experiment (or the item count technique) in order to minimize bias due to dishonest or evasive responses. However, standard practice with the list experiment requires a large sample size, is not readily adaptable to regression or multivariate modeling, and provides only limited diagnostics. This paper addresses all three of these issues. First, the paper presents design principles for the standard list experiment (and the double list experiment) to minimize bias and reduce variance as well as providing sample size formulas for the planning of studies. Additionally, this paper investigates the properties of a number of estimators and introduces an easy-to-use piecewise estimator that reduces necessary sample sizes in many cases. Second, this paper proves that standard-procedure list experiment data can be used to estimate the probability that an individual holds the socially undesirable opinion/behavior. This allows multivariate modeling. Third, this paper demonstrates that some violations of the behavioral assumptions implicit in the technique can be diagnosed with the list experiment data. The techniques in this paper are illustrated with examples from American politics.

BackgroundAccurately measuring abortion incidence poses many challenges. The list experiment is a method designed to increase the reporting of sensitive or stigmatized behaviors in surveys, but has only recently been applied to the measurement of abortion. To further test the utility of the list experiment for measuring abortion incidence, we conducted list experiments in two countries, over two time periods.Materials and methodsThe list experiment is an indirect method of measuring sensitive experiences that protects respondent confidentiality by hiding individual responses to a binary sensitive item (i.e., abortion) by combining this response with answers to other non-sensitive binary control items. Respondents report the number of list items that apply to them, not which ones. We conducted a list experiment to measure cumulative lifetime incidence of abortion in Malawi, and separately to measure cumulative five-year incidence of abortion in Senegal, among cisgender women of reproductive age.ResultsAmong 810 eligible respondents in Malawi, list experiment results estimated a cumulative lifetime incidence of abortion of 0.9% (95%CI: 0.0, 7.6). Among 1016 eligible respondents in Senegal, list experiment estimates indicated a cumulative five-year incidence of abortion of 2.8% (95%CI: 0.0, 10.4) which, while lower than anticipated, is seven times the proportion estimated from a direct question on abortion (0.4%).ConclusionsTwo test applications of the list experiment to measure abortion experiences in Malawi and Senegal likely underestimated abortion incidence. Future efforts should include context-specific formative qualitative research for the development and selection of list items, enumerator training, and method delivery to assess if and how these changes can improve method performance.

Studies of representative bureaucracy (RB) argue public organizations reflective of the public they serve exhibit better outcomes, especially when serving underrepresented groups. RB theory attributes improved outcomes either to the actions representative bureaucrats take (active representation), or a greater perception of trust and legitimacy toward them by service recipients (symbolic representation), largely treating active and symbolic representation as separate phenomena. We explore the intricate relationship between bureaucracies and the populations they serve by observing the cross-influence between active and symbolic representation, as revealed by self-reported outcomes in discrimination complaints (N = 1,372) referred for voluntary mediation in the United States Postal Service, the REDRESS© program, a context in which mediators are highly limited in representing a claimant's interests given the requirement of impartiality. In exit surveys measuring employee perceptions of organizational justice, we observed the impact of race and gender representation by gauging changes in reported satisfaction when a mediator's race or gender matched the nature of the complaint in cases of race or sex discrimination and sexual harassment, via multivariate regression estimation. These analyses support RB theory regarding sexual harassment complaints, where complainants rated outcomes significantly more favorably for female mediators. We found a negative correlation between female mediators and sex discrimination complaints, as well as African American mediators and race discrimination complainants. To explain this discrepancy, we argue that interactions between symbolic and active representation determine the expectations and perceptions placed on bureaucrats. When a bureaucrat does not meet those expectations, service recipients tend to have a more negative view of organizational justice outcomes.

The goal of this paper is to investigate the repeated substitution method (seeSrivastava, 1967) estimating population variance in finite population sample surveys. We propose an almost unbiased multivariate ratio estimator that has a smaller mean squared error than the conventional biased multivariate ratio estimator (established byIsaki (1983)) and with the same precision as the multivariate regression estimator. Furthermore, it is a computationally much more interesting estimator since to compute it we only need to have knowledge of correlation among available variables, which it is common to have in several practical situations. A comparison of the multivariate ratio estimator proposed and the multivariate regression estimator is given.

The analysis of list experiments depends on two assumptions, known as “no design effect” and “no liars”. The no liars assumption is strong and may fail in many list experiments. I relax the no liars assumption in this paper, and develop a method to provide bounds for the prevalence of sensitive behaviors or attitudes under a weaker behavioral assumption about respondents’ truthfulness toward the sensitive item. I apply the method to a list experiment on the anti-immigration attitudes of California residents and on a broad set of existing list experiment datasets. The prevalence of different items and the correlation structure among items on the list jointly determine the width of the bound estimates. In particular, the bounds tend to be narrower when the list consists of items of the same category, such as multiple groups or organizations, different corporate activities, and various considerations for politician decision-making. My paper illustrates when the full power of the no liars assumption is most needed to pin down the prevalence of the sensitive behavior or attitude, and facilitates estimation of the prevalence robust to violations of the no liars assumption for many list experiment applications.

The list experiment, also known as the item count technique, is becoming increasingly popular as a survey methodology for eliciting truthful responses to sensitive questions. Recently, multivariate regression techniques have been developed to predict the unobserved response to sensitive questions using respondent characteristics. Nevertheless, no method exists for using this predicted response as an explanatory variable in another regression model. We address this gap by first improving the performance of a naive two-step estimator. Despite its simplicity, this improved two-step estimator can only be applied to linear models and is statistically inefficient. We therefore develop a maximum likelihood estimator that is fully efficient and applicable to a wide range of models. We use a simulation study to evaluate the empirical performance of the proposed methods. We also apply them to the Mexico 2012 Panel Study and examine whether vote-buying is associated with increased turnout and candidate approval. The proposed methods are implemented in open-source software.

Limitations of List Experiments in Post-Conflict Zones

Public opinion polls consistently show that a growing majority of Americans support same-sex marriage. Critics, however, raise the possibility that these polls are plagued by social desirability bias, and thereby may overstate public support for gay and lesbian rights. We test this proposition using a list experiment embedded in the 2013 Cooperative Congressional Election Study. List experiments afford respondents an anonymity that allows them to provide more truthful answers to potentially sensitive survey items. Our experiment finds no evidence that social desirability is affecting overall survey results. If there is social desirability in polling on same-sex marriage, it pushes in both directions. Indeed, our efforts provide new evidence that a national opinion majority favors same-sex marriage. To evaluate the robustness of our findings, we analyze a second list experiment, this one focusing on the inclusion of sexual orientation in employment nondiscrimination laws. Again, we find no overall evidence of bias.

Direct measurement of sensitive health events is often limited by high levels of under-reporting due to stigma and concerns about privacy. Abortion in particular is notoriously difficult to measure. This study implements a novel method to estimate the cumulative lifetime incidence of induced abortion in Liberia. In a randomly selected sample of 3219 women ages 15–49 years in June 2013 in Liberia, we implemented the ‘Double List Experiment’. To measure abortion incidence, each woman was read two lists: (A) a list of non-sensitive items and (B) a list of correlated non-sensitive items with abortion added. The sensitive item, abortion, was randomly added to either List A or List B for each respondent. The respondent reported a simple count of the options on each list that she had experienced, without indicating which options. Difference in means calculations between the average counts for each list were then averaged to provide an estimate of the population proportion that has had an abortion. The list experiment estimates that 32% [95% confidence interval (CI): 0.29-0.34) of respondents surveyed had ever had an abortion (26% of women in urban areas, and 36% of women in rural areas, P-value for difference < 0.001), with a 95% response rate. The list experiment generated an estimate five times greater than the only previous representative estimate of abortion in Liberia, indicating the potential utility of this method to reduce under-reporting in the measurement of abortion. The method could be widely applied to measure other stigmatized health topics, including sexual behaviours, sexual assault or domestic violence.

Following the COVID‐19 outbreak, anti‐Asian racism increased around the world, as exhibited through greater instances of abuse and hate crimes. To better understand the scale of anti‐Asian racism and the characteristics of people who may be expressing racial prejudice, we sampled respondents in Australia and the United States over 31 August–9 September 2020 (1375 Australians and 1060 Americans aged 18 or above; source YouGov). To address potential social desirability bias, we use both direct and indirect (list experiment) questions to measure anti‐Asian sentiment and link these variables to key socioeconomic factors. We find that, instead of being universal among general populations, anti‐Asian sentiment is patterned differently across both country contexts and socioeconomic groups. In the United States, the most significant predictor of anti‐Asian bias is political affiliation. By contrast, in Australia, anti‐Asian bias is closely linked to a wide range of socioeconomic factors including political affiliation, age, gender, employment status and income.

Measurement error threatens the validity of survey research, especially when studying sensitive questions. Although list experiments can help discourage deliberate misreporting, they may also suffer from nonstrategic measurement error due to flawed implementation and respondents’ inattention. Such error runs against the assumptions of the standard maximum likelihood regression (MLreg) estimator for list experiments and can result in misleading inferences, especially when the underlying sensitive trait is rare. We address this problem by providing new tools for diagnosing and mitigating measurement error in list experiments. First, we demonstrate that the nonlinear least squares regression (NLSreg) estimator proposed in Imai (2011) is robust to nonstrategic measurement error. Second, we offer a general model misspecification test to gauge the divergence of theMLregandNLSregestimates. Third, we show how to model measurement error directly, proposing new estimators that preserve the statistical efficiency ofMLregwhile improving robustness. Last, we revisit empirical studies shown to exhibit nonstrategic measurement error, and demonstrate that our tools readily diagnose and mitigate the bias. We conclude this article with a number of practical recommendations for applied researchers. The proposed methods are implemented through an open-source software package.

BackgroundThe list experiment is a promising measurement tool for eliciting truthful responses to stigmatized or sensitive health behaviors. However, investigators may be hesitant to adopt the method due to previously untestable assumptions and the perceived inability to conduct multivariable analysis. With a recently developed statistical test that can detect the presence of a design effect – the absence of which is a central assumption of the list experiment method – we sought to test the validity of a list experiment conducted on self-reported abortion in Liberia. We also aim to introduce recently developed multivariable regression estimators for the analysis of list experiment data, to explore relationships between respondent characteristics and having had an abortion – an important component of understanding the experiences of women who have abortions.MethodsTo test the null hypothesis of no design effect in the Liberian list experiment data, we calculated the percentage of each respondent “type,” characterized by response to the control items, and compared these percentages across treatment and control groups with a Bonferroni-adjusted alpha criterion. We then implemented two least squares and two maximum likelihood models (four total), each representing different bias-variance trade-offs, to estimate the association between respondent characteristics and abortion.ResultsWe find no clear evidence of a design effect in list experiment data from Liberia (p = 0.18), affirming the first key assumption of the method. Multivariable analyses suggest a negative association between education and history of abortion. The retrospective nature of measuring lifetime experience of abortion, however, complicates interpretation of results, as the timing and safety of a respondent’s abortion may have influenced her ability to pursue an education.ConclusionOur work demonstrates that multivariable analyses, as well as statistical testing of a key design assumption, are possible with list experiment data, although with important limitations when considering lifetime measures. We outline how to implement this methodology with list experiment data in future research.

RationaleSocial desirability bias, which is the tendency to under-report socially, undesirable health behaviours, significantly distorts information on sensitive behaviours, gained from self-reports and prevents accurate estimation of the prevalence of those, behaviours. We contribute to a growing body of literature that seeks to assess the performance of the list experiment method to improve estimation of these sensitive health behaviours.MethodWe use a double-list experiment design in which respondents serve as the treatment group for one list and as the control group for the other list to estimate the prevalence of two sensitive health behaviours in different settings: condom use among 500 female sex workers in urban Senegal and physical intimate partner violence among 1700 partnered women in rural Burkina Faso. First, to assess whether the list experiment improves the accuracy of estimations of the prevalence of sensitive behaviours, we compare the prevalence rates estimated from self-reports with those elicited through the list experiment. Second, we test whether the prevalence rates of the sensitive behaviours obtained using the double-list design are similar, and we estimate the reduction in the standard errors obtained with this design. Finally, we compare the results obtained through another indirect elicitation method, the polling vote method.ResultsWe show that the list experiment method reduces misreporting by 17 percentage points for condom use and 16–20 percentage points for intimate partner violence. Exploiting the double-list experiment design, we also demonstrate that the prevalence estimates obtained through the use of the two lists are identical in the full sample and across sub-groups and that the double-list design reduces the standard errors by approximately 40% compared to the standard errors in the simple list design. Finally, we show that the list experiment method leads to a higher estimation of the prevalence of sensitive behaviours than the polling vote method.ConclusionThe study suggests that list experiments are an effective method to improve estimation of the prevalence of sensitive health behaviours.

Despite the growing acknowledgment of the importance of loneliness among older individuals, questionnaire length constraints may hinder the inclusion of common multi-item loneliness scales in surveys. Direct, single-item loneliness measures are a practical alternative, but scholars have expressed concerns that such measures may lead to underreporting. Our aim was to test whether such reservations are justified. We conducted a preregistered list experiment among 2,553 people aged 50 + who participated in the Dutch Longitudinal Internet studies for the Social Sciences (LISS) panel. The list experiment method has been developed to unobtrusively gather sensitive information. We compared the list experiment estimate of the prevalence of frequent loneliness with the corresponding direct question estimate to assess downward bias in the latter. Next to pooled models, we estimated models stratified by gender to assess whether loneliness underreporting differed between women and men. Relying on the direct question, we estimated that 5.9% of respondents frequently felt lonely. Our list experiment indicated that the prevalence of frequent loneliness was 13.1%. Although substantial in magnitude, the difference between both estimates was only marginally significant (Δb: 0.072, 95% CI: − 0.003;0.148, p = .06). No evidence of gender differences was found. Although we cannot be conclusive that loneliness estimates are biased downward when a direct question is used, our results call for caution with direct, single-item measures of loneliness if researchers want to avoid underreporting. Replications are needed to gain more precise insights into the extent to which direct, single-item loneliness measures are prone to downward reporting bias.