Improving studies of sensitive topics using prior evidence: an informative Bayesian approach for list experiments

Due to the inherent sensitivity of many survey questions, a number of researchers have adopted an indirect questioning technique known as the list experiment (or the item count technique) in order to minimize bias due to dishonest or evasive responses. However, standard practice with the list experiment requires a large sample size, is not readily adaptable to regression or multivariate modeling, and provides only limited diagnostics. This paper addresses all three of these issues. First, the paper presents design principles for the standard list experiment (and the double list experiment) to minimize bias and reduce variance as well as providing sample size formulas for the planning of studies. Additionally, this paper investigates the properties of a number of estimators and introduces an easy-to-use piecewise estimator that reduces necessary sample sizes in many cases. Second, this paper proves that standard-procedure list experiment data can be used to estimate the probability that an individual holds the socially undesirable opinion/behavior. This allows multivariate modeling. Third, this paper demonstrates that some violations of the behavioral assumptions implicit in the technique can be diagnosed with the list experiment data. The techniques in this paper are illustrated with examples from American politics.

Abortion is highly stigmatized in the United States which prevents its accurate measurement in surveys. The list experiment aims to improve the reporting of abortion history. We evaluated whether a list experiment resulted in higher reporting of abortion experiences than did two direct questions. Utilizing data from a representative survey of adult women of reproductive age in Ohio, we examined abortion history using two direct questions and a double list experiment. Through the double list experiment, we asked respondents to report how many of two lists of health items they had experienced; one list included abortion. We compared weighted history of abortion between these measures and by respondent demographic characteristics (age and socioeconomic status). Estimates of abortion history were similar between direct and list experiment questions. When measured with the two different direct question of abortion history, 8.4% and 8.0% of all respondents indicated ever having an abortion and with the list experiment, 8.5% indicated ever having an abortion. In a Midwestern state-level survey, the list experiment did not lead to increases in abortion reporting as compared to the direct questions. Subgroup analyses require larger samples, and future iterations should incorporate related but non-stigmatized control items to reduce misclassification and under-powering of such subgroup analyses.

Abortion is highly stigmatized in the United States which prevents its accurate measurement in surveys. The list experiment aims to improve the reporting of abortion history. We evaluated whether a list experiment resulted in higher reporting of abortion experiences than did two direct questions. Utilizing data from a representative survey of adult women of reproductive age in Ohio, we examined abortion history using two direct questions and a double list experiment. Through the double list experiment, we asked respondents to report how many of two lists of health items they had experienced; one list included abortion. We compared weighted history of abortion between these measures and by respondent demographic characteristics (age and socioeconomic status). Estimates of abortion history were similar between direct and list experiment questions. When measured with the two different direct question of abortion history, 8.4% and 8.0% of all respondents indicated ever having an abortion and with the list experiment, 8.5% indicated ever having an abortion. In a Midwestern state-level survey, the list experiment did not lead to increases in abortion reporting as compared to the direct questions. Subgroup analyses require larger samples, and future iterations should incorporate related but non-stigmatized control items to reduce misclassification and under-powering of such subgroup analyses.

Direct measurement of sensitive health events is often limited by high levels of under-reporting due to stigma and concerns about privacy. Abortion in particular is notoriously difficult to measure. This study implements a novel method to estimate the cumulative lifetime incidence of induced abortion in Liberia. In a randomly selected sample of 3219 women ages 15–49 years in June 2013 in Liberia, we implemented the ‘Double List Experiment’. To measure abortion incidence, each woman was read two lists: (A) a list of non-sensitive items and (B) a list of correlated non-sensitive items with abortion added. The sensitive item, abortion, was randomly added to either List A or List B for each respondent. The respondent reported a simple count of the options on each list that she had experienced, without indicating which options. Difference in means calculations between the average counts for each list were then averaged to provide an estimate of the population proportion that has had an abortion. The list experiment estimates that 32% [95% confidence interval (CI): 0.29-0.34) of respondents surveyed had ever had an abortion (26% of women in urban areas, and 36% of women in rural areas, P-value for difference < 0.001), with a 95% response rate. The list experiment generated an estimate five times greater than the only previous representative estimate of abortion in Liberia, indicating the potential utility of this method to reduce under-reporting in the measurement of abortion. The method could be widely applied to measure other stigmatized health topics, including sexual behaviours, sexual assault or domestic violence.

Estimating Prevalence of Abortion Using List Experiments: Findings from a Survey of Women in Delaware and Maryland.

Objective The increasing rate of alcohol related traffic accidents, especially among adolescents, has become a major safety concern. While numerous studies have examined drunk driving among young people using direct questioning methods, such behaviors are often viewed as sensitive, increasing the risk of underreporting. This work is to examine the actual prevalence of misreporting regarding drunk driving and riding with a drunk driver, while accounting for potential misreporting bias. Method The study employed a list experiment, an indirect questioning technique, on a sample of 615 undergraduates in Vietnam. Results Our findings reveal that the prevalence of drunk driving among male students is nearly three times higher when measured using the list experiment compared to direct questioning. Furthermore, students from single-parent or parentless households exhibit considerably greater misreporting bias and a higher prevalence of drunk driving than those from families with both parents Conclusion This study suggests that list experiments are an appropriate and effective method for estimating the prevalence of sensitive alcohol consumption-related travel behaviors.

Estimating the sex buying behavior of adult males in the United States: List experiment and direct question estimates

Objective: To explore the performance of the list experiment as a tool to reduce underreporting in estimation of self-managed abortion attempts in the United States. Study Design: We analyzed data from a 2017 nationally representative survey of self-identified females aged 18 to 49 in the United States recruited via Ipsos’ KnowledgePanel (n = 6897). We estimated the proportion of respondents who had ever attempted a self-managed abortion using a direct question as compared to a double list experiment. Results: We estimated that 8.2% (95% CI, 6.5, 9.9) had ever attempted to self-manage an abortion based on responses to the double list experiment—nearly three times the 2.7% (95% CI, 2.2, 3.4) estimated via the direct question. However, one of the two list sets performed inconsistently overall, and particularly among respondents who took the survey in Spanish. Conclusion: Future research that aims to measure self-managed abortion attempts in a hostile political context can consider the list experiment as a tool to reduce under-reporting, with additional qualitative and cognitive work needed to improve list performance.

Scholars increasingly rely on indirect questioning techniques to reduce social desirability bias and item nonresponse for sensitive survey questions. The major drawback of these approaches, however, is their inefficiency relative to direct questioning. We show how to improve the statistical analysis of the list experiment, randomized response technique, and endorsement experiment by exploiting auxiliary information on the sensitive trait. We apply the proposed methodology to survey experiments conducted among voters in a controversial antiabortion referendum held during the 2011 Mississippi General Election. By incorporating the official county-level election results, we obtain precinct- and individual-level estimates that are more accurate than standard indirect questioning estimates and occasionally even more efficient than direct questioning. Our simulation studies shed light on the conditions under which our approach can improve the efficiency and robustness of estimates based on indirect questioning techniques. Open-source software is available for implementing the proposed methodology.

The proliferation of artificial intelligence (AI) technology has brought both innovative opportunities and unprecedented challenges to the education sector. Although AI makes education more accessible and efficient, the intentional misuse of AI chatbots in facilitating academic cheating has become a growing concern. By using the indirect questioning technique via a list experiment to minimize social desirability bias, this research contributes to the ongoing dialog on academic integrity in the era of AI. Our findings reveal that students conceal AI-powered academic cheating behaviors when directly questioned, as the prevalence of cheaters observed via list experiments is almost threefold the prevalence of cheaters observed via the basic direct questioning approach. Interestingly, our subsample analysis shows that AI-powered academic cheating behaviors differ significantly across genders and grades, as higher-grade female students are more likely to cheat than newly enrolled female students. Conversely, male students consistently engage in academic cheating throughout all grades. Furthermore, we discuss potential reasons for the heterogeneous effects in academic cheating behavior among students such as gender disparity, academic-related pressure, and peer effects. Implications are also suggested for educational institutions to promote innovative approaches that harness the benefits of AI technologies while safeguarding academic integrity.

Although induced abortion is common, measurement issues have long made this area of research challenging. The current analysis applies an indirect method known as the list experiment to try to improve survey‐based measurement of induced abortion. We added a double list experiment to a population‐based survey of reproductive age women in Rajasthan, India and compared resulting abortion estimates to those we obtained via direct questioning in the same sample. We then evaluated list experiment assumptions. The final sample completing the survey consisted of 6,035 women. Overall, 1.8 percent of the women reported a past abortion via the list experiment questions, whereas 3.5 percent reported an abortion via the direct questions, and this difference was statistically significant. As such, the list experiment failed to produce more valid estimates of this sensitive behavior on a population‐based survey of reproductive age women in Rajasthan, India. One explanation for the poor list experiment performance is our finding that key assumptions of the methodology were violated. Women may have mentally enumerated the treatment list items differently from the way they enumerated control list items. Further research is required to determine whether researchers can learn enough about how the list experiment performs in different contexts to effectively and consistently leverage its potential benefits to improve measurement of induced abortion.

BackgroundAccurately measuring stigmatized experiences is a challenge across reproductive health research. In this study, we tested a novel method – the list experiment – that aims to reduce underreporting of sensitive events by asking participants to report how many of a list of experiences they have had, not which ones. We applied the list experiment to measure “self-managed abortion” - any attempt by a person to end a pregnancy on one’s own, outside of a clinical setting – a phenomenon that may be underreported in surveys due to a desire to avoid judgement.MethodsWe administered a double list experiment on self-managed abortion to a Texas-wide representative sample of 790 women of reproductive age in 2015. Participants were asked how many of a list of health experiences they had experienced; self-managed abortion was randomly added as an item to half of the lists. A difference in the average number of items reported by participants between lists with and without self-managed abortion provided a population level estimate of self-managed abortion. In 2017, we conducted cognitive interviews with women of reproductive age in four states to understand how women (1) interpreted the list experiment question format, and (2) interpreted the list item on prior experiences attempting to self-manage an abortion.ResultsResults from this list experiment estimated that 8% of women of reproductive age in Texas have ever self-managed an abortion. This number was higher than expected, thus, the researchers conducted cognitive interviews to better understand how people interpreted the list experiment on self-managed abortion. Some women interpreted “on your own” to mean “without the knowledge of friends or family”, as opposed to “without medical assistance”, as intended.ConclusionThe list experiment may have reduced under-reporting of self-managed abortion; however, the specific phrasing of the list item may also have unintentionally increased reporting of abortion experiences not considered “self-managed.” High participation in and comprehension of the list experiment, however, suggests that this method is worthy of further exploration as tool for measuring stigmatized experiences.

Abortion is particularly difficult to measure, especially in legally restrictive settings such as Pakistan. The List Experiment-a technique for measuring sensitive health behaviors indirectly-may minimize respondents' underreporting of abortion due to stigma or legal restrictions, but has not been previously applied to estimate abortion prevalence in Pakistan. A sample of 4,159 married women of reproductive age were recruited from two communities of Karachi in 2018. Participants completed a survey that included a double list experiment to measure lifetime abortion prevalence, as well as direct questions about abortion and other background characteristics. Data were used to calculate direct and indirect estimates of abortion prevalence for the overall sample and by sociodemographic characteristics, as well as to test for a design effect. Regression analyses were conducted to examine associations between characteristics and abortion reporting from direct questioning and the list experiment. The estimate of abortion prevalence from the list experiment was 16%; the estimate from the direct question was 8%. No evidence of a design effect was found. Abortion reporting was associated with most selected characteristics in the regression model for direct questioning, but with few in the list experiment models. That the estimate of abortion prevalence in Karachi generated from the list experiment was twice that generated from direct questioning suggests that the indirect method reduced underreporting, and may have utility to estimate abortion in similar settings and to improve the accuracy of data collecting for other sensitive health topics.

To reduce strategic misreporting on sensitive topics, survey researchers increasingly use list experiments rather than direct questions. However, the complexity of list experiments may increase nonstrategic misreporting. We provide the first empirical assessment of this trade-off between strategic and nonstrategic misreporting. We field list experiments on election turnout in two different countries, collecting measures of respondents’ true turnout. We detail and apply a partition validation method which uses true scores to distinguish true and false positives and negatives for list experiments, thus allowing detection of nonstrategic reporting errors. For both list experiments, partition validation reveals nonstrategic misreporting that is: undetected by standard diagnostics or validation; greater than assumed in extant simulation studies; and severe enough that direct turnout questions subject to strategic misreporting exhibit lower overall reporting error. We discuss how our results can inform the choice between list experiment and direct question for other topics and survey contexts.

This article explores experimental methods in sociology that are used to address sensitive questions. Sensitive questions are those that involve socially stigmatized or taboo subjects, such as illegal activities, personal beliefs, and sexual behavior. The use of traditional survey methods to investigate sensitive questions may lead to response bias or social desirability bias. To overcome these limitations, scientists have developed innovative experimental methods, such as the list experiment. The list experiment is a survey-based technique that provides respondents with a list of items and asks them how many items on the list they have experienced or endorse. Unlike direct questioning, this method allows respondents to express their opinions anonymously and reduces social desirability bias. Moreover, the list experiment can estimate the prevalence of sensitive behaviors or attitudes without revealing individual responses, which makes it particularly useful in studying politically or socially sensitive topics. This article describes the theoretical foundations, design, and implementation of the list experiment. With the help of the VOSviewer program, a bibliographic analysis was carried out based on Scopus database. At the same time, it discusses the advantages and limitations of this method compared to other experimental techniques and highlights some of its practical applications in sociological research. Overall, this article provides a comprehensive overview of the list experiment and its contribution to the advancement of sociological research.