Response Propensity Research Articles

Estimating response propensities in nonprobability surveys using machine learning weighted models

Propensity Score Adjustment (PSA) is a widely accepted method to reduce selection bias in nonprobability samples. In this approach, the (unknown) response probability of each individual is estimated in a nonprobability sample, using a reference probability sample. This, the researcher obtains a representation of the target population, reflecting the differences (for a set of auxiliary variables) between the population and the nonprobability sample, from which response probabilities can be estimated.Auxiliary probability samples are usually produced by surveys with complex sampling designs, meaning that the use of design weights is crucial to accurately calculate response probabilities. When a linear model is used for this task, maximising a pseudo log-likelihood function which involves design weights provides consistent estimates for the inverse probability weighting estimator. However, little is known about how design weights may benefit the estimates when techniques such as machine learning classifiers are used.This study aims to investigate the behaviour of Propensity Score Adjustment with machine learning classifiers, subject to the use of weights in the modelling step. A theoretical approximation to the problem is presented, together with a simulation study highlighting the properties of estimators using different types of weights in the propensity modelling step.

Address matching using machine learning methods: An application to register-based census

Today, most activities of the statistical offices need to be adapted to the modernization policies of the national statistical system. Therefore, the application of machine learning techniques is mandatory for the main activities of statistical centers. These include important issues such as coding business activities, address matching, prediction of response propensities, and many others. One of the common applications of machine learning methods in official statistics is to match a statistical address to a postal address, in order to establish a link between register-based census and traditional censuses with the aim of providing time series census information. Since there is no unique identifier to directly map the records from different databases, text-based approaches can be applied. In this paper, a novel application of machine learning will be investigated to integrate data sources of governmental records and census, employing text-based learning. Additionally, three new methods of machine learning classification algorithms are proposed. A simulation study has been performed to evaluate the robustness of methods in terms of the degree of duplication and purity of the texts. Due to the limitation of the R programming environment on big data sets, all programming has been successfully implemented on SAS (Statistical analysis system) software.

Improving the Efficiency of Outbound CATI As a Nonresponse Follow-Up Mode in Address-Based Samples: A Quasi-Experimental Evaluation of a Dynamic Adaptive Design

Abstract This article evaluates the use of dynamic adaptive design methods to target outbound computer-assisted telephone interviewing (CATI) in the California Health Interview Survey (CHIS). CHIS is a large-scale, annual study that uses an address-based sample (ABS) with push-to-Web mailings, followed by outbound CATI follow-up for addresses with appended phone numbers. CHIS 2022 implemented a dynamic adaptive design in which predictive models were used to end dialing early for some cases. For addresses that received outbound CATI follow-up, dialing was paused after three calls. A response propensity (RP) model was applied to predict the probability that the address would respond to continued dialing, based on the outcomes of the first three calls. Low-RP addresses were permanently retired with no additional dialing, while the rest continued through six or more attempts. We use a difference-in-difference design to evaluate the effect of the adaptive design on calling effort, completion rates, and the demographic composition of respondents. We find that the adaptive design reduced the mean number of calls per sampled unit by about 14 percent (relative to a modeled no-adaptive-design counterfactual) with a minimal reduction in the completion rate and no strong evidence of changes in the prevalence of target demographics. This suggests that RP modeling can meaningfully distinguish between ABS sample units for which additional dialing is and is not productive, helping to control outbound dialing costs without compromising sample representativeness.

Responsive and Adaptive Designs in Repeated Cross-National Surveys: A Simulation Study

AbstractCross-national surveys run the risk of differential survey errors, where data collected vary in quality from country to country. Responsive and adaptive survey designs (RASDs) have been proposed as a way to reduce survey errors, by leveraging auxiliary variables to inform fieldwork efforts, but have rarely been considered in the context of cross-national surveys. Using data from the European Social Survey, we simulate fieldwork in a repeated cross-national survey using RASD where fieldwork efforts are ended early for selected units in the final stage of data collection. Demographic variables, paradata (interviewer observations), and contact data are used to inform fieldwork efforts. Eight combinations of response propensity models and selection mechanisms are evaluated in terms of sample composition (as measured by the coefficient of variation of response propensities), response rates, number of contact attempts saved, and effects on estimates of target variables in the survey. We find that sample balance can be improved in many country-round combinations. Response rates can be increased marginally and targeting high propensity respondents could lead to significant cost savings associated with making fewer contact attempts. Estimates of target variables are not changed by the case prioritizations used in the simulations, indicating that they do not impact nonresponse bias. We conclude that RASDs should be considered in cross-national surveys, but that more work is needed to identify suitable covariates to inform fieldwork efforts.

Predicting Days to Respondent Contact in Cross-Sectional Surveys Using a Bayesian Approach

Abstract Surveys estimate and monitor a variety of data collection parameters, including response propensity, number of contacts, and data collection costs. These parameters can be used as inputs to a responsive/adaptive design or to monitor the progression of a data collection period against predefined expectations. Recently, Bayesian methods have emerged as a method for combining historical information or external data with data from the in-progress data collection period to improve prediction. We develop a Bayesian method for predicting a measure of case-level progress or productivity, the estimated time lag, in days, between first contact attempt and first respondent contact. We compare the quality of predictions from the Bayesian method to predictions generated from more commonly-used predictive methods that leverage data from only historical data collection periods or the in-progress round of data collection. Using prediction error and misclassification as short- or long- day lags, we demonstrate that the Bayesian method results in improved predictions close to the day of the first contact attempt, when these predictions may be most informative for interventions or interviewer feedback. This application adds to evidence that combining historical and current information about data collection, in a Bayesian framework, can improve predictions of data collection parameters.

Total Bias in Income Surveys when Nonresponse and Measurement Errors are Correlated

Abstract Household surveys on income might suffer from quality limitations mainly due to the difficulty of enrolling households (unit nonresponse) and retrieving correct information during the interview (measurement error [ME]). These errors are likely to be correlated because of latent factors, such as the threat of disclosing personal information, the perceived sensitivity of the topic, or social desirability. For survey organizations, assessing the interplay of these errors and their impact on the accuracy and precision of inferences derived from their data is crucial. In this article, we propose to use a standard sample selection model within a total survey error framework to deal with the case of correlated nonresponse error (NR) and ME in estimating average household income. We use it to study the correlation between the two errors, quantify the ME component due to this correlation, and evaluate ME among nonrespondents. Using the Italian Survey on Income and Wealth linked with administrative income data from tax returns, we find a positive correlation between the two errors and that households at the extremes of the income distribution mainly cause this association. Our results show that ME contributes more to the total error than unit nonresponse and that it would be larger in absence of the correlation between the two errors. Finally, efforts to reduce nonresponse rates are worthwhile only for nonrespondents in the lowest estimated response propensity group. If these households participate, the bias decreases because of the reduction in NR that offsets the increase in ME.

Not Random and Not Ignorable. An Examination of Nonresponse to Income Question in the European Social Survey, 2008–2018

This study analyzes the consequences of item nonresponse to the question about a household’s total net income in the European Social Survey (2008–2018). We recognize two mechanisms in avoiding answering the income question: task complexity and question sensitivity, and apply multilevel logistic regressions to predict the probability of refusals or “Do not know” across respondents of different income levels. We find that the refusal to answer the income question is the highest for respondents with lower incomes, while the probability of selecting “Do not know” answers or refusal to answer is the same among respondents with higher incomes. The bias resulting from the correlation between response propensities and household income affects the accuracy of estimates for several attitudinal measures when income is included as an explanatory variable. We recommend reducing the risk of bias by limiting the complexity and sensitivity of the income question and accounting for nonresponse bias.

Case Prioritization in the SIPP: A Five-Year Review

Case prioritization is a concerted effort to achieve high data quality during data collection by reallocating scarce resources to the sample cases that need them the most. Our research, conducted as part of the Survey of Income and Program Participation (SIPP), examines the effect of case prioritization over the changing landscape of this survey. Our findings suggest that since instituting case prioritization, field interviewers exert more effort on higher priority cases, and continual centralized monitoring and intervention have led to improvements in data quality measures, in particular R-indicator and coefficient of variation (CV) of response propensities.

The Effects of Framing the Survey Request and Using Targeted Appeals on Participation in Cross-sectional Surveys

The presentation of survey requests represents an easily modifiable feature of survey communications that can in some contexts affect response propensities. Here, we examine how two features: the framing of the participation request (informed by prospect theory) and the inclusion of targeted appeals based on demographic background (age or electoral district), affect participation rates in the 2021 Icelandic National Election Study, a mixed-mode cross-sectional survey. We find that these features generally fail to affect response propensities and sample composition. However, framed requests produced marginally higher response rates for a subset of the sample that was invited to complete the survey online. This suggests that modifying the presentation of the survey request may be less effective in cross-sectional surveys than in longitudinal ones, where more information is available to inform the content of targeted messages.

Interviewer Involvement in Respondent Selection Moderates the Relationship Between Response Rates and Sample Bias in Cross-National Survey Projects in Europe

Abstract Survey researchers and practitioners often assume that higher response rates are associated with a higher quality of survey data. However, the evidence for this claim in face-to-face surveys is mixed. To explain these mixed results, recent studies have proposed that interviewers’ involvement in respondent selection moderates the effect of response rates on data quality. Previous analyses based on data from the European Social Survey found that response rates are positively associated with data quality when interviewer involvement in respondent selection is minimal. However, the association between response rates and data quality is negative when interviewers are more involved in respondent selection through household frame creation or within-household selection of target persons. These studies have hypothesized that some interviewers deviate from prescribed selection procedures to select individuals with higher response propensities, which increase response rates while reducing data quality. We replicate these results with an extended dataset, including more recent European Social Survey rounds and three other European survey projects: the European Quality of Life Survey, European Values Study, and International Social Survey Programme. Based on our results, we recommend that surveys include procedures to verify respondent-selection practices into their fieldwork control procedures.

Nonignorable Consequences of (Partially) Ignoring Missing Item Responses: Students Omit (Constructed Response) Items Due to a Lack of Knowledge

In recent literature, alternative models for handling missing item responses in large-scale assessments have been proposed. Based on simulations and arguments based on psychometric test theory, it is argued in this literature that missing item responses should never be scored as incorrect in scaling models but rather treated as ignorable or handled based on a model. The present article shows that these arguments have limited validity and illustrates the consequences in a country comparison using the PIRLS 2011 study. It is argued that students omit (constructed response) items because they do not know the correct item answer. A different treatment of missing item responses than scoring them as incorrect leads to significant changes in country rankings, which induces nonignorable consequences regarding the validity of the results. Additionally, two alternative item response models are proposed based on different assumptions for missing item responses. In the first pseudo-likelihood approach, missing item responses for a particular student are replaced by a score that ranges between zero and a model-implied probability computed based on the non-missing items. In the second approach, the probability of a missing item response is predicted by a latent response propensity variable and the item response itself. The models were applied to the PIRLS 2011 study, demonstrating that country comparisons change under different modeling assumptions for missing item responses.

Optimizing Data Collection Interventions to Balance Cost and Quality in a Sequential Multimode Survey

AbstractHigh-quality survey data collection is getting more expensive to conduct because of decreasing response rates and rising data collection costs. Responsive and adaptive designs have emerged as a framework for targeting and reallocating resources during the data collection period to improve survey data collection efficiency. Here, we report on the implementation and evaluation of a responsive design experiment in the National Survey of College Graduates that optimizes the cost-quality tradeoff by minimizing a function of data collection costs and the root mean squared error of a key survey measure, self-reported salary. We used a Bayesian framework to incorporate prior information and generate predictions of estimated response propensity, self-reported salary, and data collection costs for use in our optimization rule. At three points during the data collection process, we implement the optimization rule and identify cases for which reduced effort would have minimal effect on the mean squared error (RMSE) of mean self-reported salary while allowing us to reduce data collection costs. We find that this optimization process allowed us to reduce data collection costs by nearly 10 percent, without a statistically or practically significant increase in the RMSE of mean salary or a decrease in the unweighted response rate. This experiment demonstrates the potential for these types of designs to more effectively target data collection resources to reach survey quality goals.

Handling Missing Values in Surveys With Complex Study Design: A Simulation Study

Abstract The inverse probability weighting (IPW) method is commonly used to deal with missing-at-random outcome (response) data collected by surveys with complex sampling designs. However, IPW methods generally assume that fully observed predictor variables are available for all sampled units, and it is unclear how to appropriately implement these methods when one or more independent variables are subject to missing values. Multiple imputation (MI) methods are well suited for a variety of missingness patterns but are not as easily adapted to complex sampling designs. In this case study, we consider the National Survey of Morbidity and Risk Factors (EMENO), a multistage probability sample survey. To understand the strengths and limitations of using either missing data treatment method for the EMENO, we present an extensive simulation study modeled on the EMENO health survey, with the target analysis being the estimation of population prevalence of hypertension as well as the association between hypertension and income. Both variables are subject to missingness. We test a variety of IPW and MI methods in simulation and on empirical data from the survey, assessing robustness by varying missingness mechanisms, proportions of missingness, and strengths of fitted response propensity models.

State Mental Health Authority Level of Involvement in Coordinated Specialty Care Clinics and Client Outcomes.

State mental health authorities (SMHAs) in all U.S. states and territories administer the Mental Health Block Grant (MHBG) set-aside funding for first-episode psychosis. Funds support implementation of coordinated specialty care (CSC) programs. The authors investigated the relationship between the level of SMHA involvement with CSC programs and clinical outcomes of clients in these programs. As part of a mixed-methods study of 34 CSC programs, SMHAs from 21 states and one U.S. territory associated with the 34 CSC programs participated in a 1-hour interview (between November 2018 and May 2019) focused on SMHA involvement in administration of MHBG set-aside funds and the SMHA's ongoing relationship with funded CSC programs. SMHA involvement was rated on a scale of 1 to 5, with 5 indicating the highest involvement. Client outcome data were collected at the 34 study sites over an 18-month period. Multilevel random-effect modeling was used, controlling for response propensity (propensity score), client demographic variables, and program-level covariates (i.e., fidelity score, staff turnover rates, service area urbanicity, and number of clients enrolled). Clients in CSC programs with SMHAs that were the most involved (level 5) had significantly improved symptoms, social functioning, and role functioning, compared with clients in programs with which SMHAs were least involved (level 1). The findings suggest that increased SMHA involvement in CSC programs is relevant for positive client outcomes. Levels of first-episode psychosis funding doubled in 2021 and 2022, and it is important to identify how SMHAs affect the success of CSC programs and the individuals served.

Prevalence of heavy episodic drinking and alcohol use disorder diagnosis among US college students: Results from the national Healthy Minds Study

Alcohol use is a common, recognized problem on college campuses. This study examined alcohol use in a national sample of US college students across 78 campuses. Using four waves of data from the Healthy Minds Study (2015–2019), we explored variations by student demographics in prevalence of recent: alcohol consumption, heavy episodic drinking (HED, 4/5 + drinks in one sitting), frequent HED (3 + HED events), and lifetime alcohol use disorder (AUD) diagnosis. Stratified analyses and logistic regression with response propensity weights were used. Two-thirds of students consumed alcohol and roughly-one-third engaged in HED in the past 2 weeks. Alcohol use was more common among students who: identified as cis women/men, bisexual or gay/lesbian/queer, white non-Hispanic, lived in Greek housing or off-campus, were not first generation, or those not rating religion as important. Prevalence of HED among recent drinkers was high (56.7%) but varied by gender identity, race-ethnicity, living situation, and religiosity. In addition, higher HED prevalence was reported among: international, undergraduate, and underage (under 21) students. There was little variation in HED by sexual orientation identity or first generation status among recent drinkers. In a sub-sample of students engaging in frequent HED, AUD diagnosis was uncommon (1.4%) and less likely among students identifying as: cis women/men, heterosexual, racial-ethnic minorities (particularly Asian/Asian American or Pacific Islander), international, religious, or living in Greek housing. Alcohol use continues to be a part of college life, while screening and treatment remains rare. There are opportunities for improved programming and outreach acknowledging college student diversity.

Assessing the potential for nonresponse bias and measurement concordance in the clinical preventive services self-administered questionnaire survey1

Health survey statistics are essential for setting national targets, monitoring population-level trends, and tracking progress in adults’ utilization of preventive health services. However, survey estimates can be subject to nonresponse bias. This paper primarily assesses the potential for nonresponse bias in six utilization estimates from the Clinical Preventive Services Self-Administered Questionnaire (PSAQ) survey. The article highlights several findings. First, the eligible PSAQ sample person’s selective nonresponse associated with influenza vaccinations and blood pressure check-ups reported in the Medical Expenditure Panel Survey provides indirect evidence of overestimating the utilization of those services in the PSAQ survey. Second, the study found a few weighting class variables correlated with PSAQ response propensity and survey variables. Third, surprisingly, base-weighted estimates of preventive service utilization changed little despite multiple post-survey adjustments, suggesting no substantial nonresponse bias. Additionally, the study finds moderate to near-perfect concordance in responses to selected preventive service questions across interview modes.

Incorporating real-time citizen responder information to augment EMS logistics operations: A simulation study

Minimizing response time is the key to live-saving missions of emergency medical services (EMS). As an alternative to professional paramedics, citizen responders (CRs), medically trained volunteers in the community, can use their training to help others in their neighborhood or workplace, by rendering emergency responses. Empowered by connected technologies, CRs can be promptly notified by an EMS request upon its arrival, and they may provide time-sensitive (and often life-saving) response before an ambulance arrives. Currently, many EMS agencies, though employing CRs for certain emergency situations, do not necessarily intelligently leverage real-time information of CRs (e.g., position and response propensity) when making decisions on EMS logistic operations (e.g., ambulance dispatch). As a result, opportunities arise for better coordination between CRs and ambulances. In this paper, we investigate the decision problem of dispatching ambulances for priority-differentiated emergencies. We adapt a locally optimal dispatch procedure with incorporation of real-time CR information, which is intended to balance improved response for the current emergency request via dispatching CRs and response preparedness for future requests. We evaluate the adapted dispatch procedure via a discrete event simulation and compare our procedure against the procedure without incorporating CR information and a commonly used dispatch strategy in practice. We perform a sensitivity analysis with respect to the spatial distribution and response propensity levels of CRs. The results suggest that our procedure could lead to substantial and reliable system improvement.

In an Era of Enhanced Cybersecurity: The Effect of Disclosing a Third Party’s Role in Confidentiality Pledges on Response Propensity

Abstract Federal surveys often pledge that personally identifiable information (PII) will be used for statistical purposes only and that the willful disclosure of PII by employees who have been sworn to confidentiality is subject to penalties. In 2015, the passing of the Cybersecurity Enhancement Act required the Department of Homeland Security (DHS) to monitor federal computers for cyber threats. Since DHS personnel are not bound to the confidentiality pledges used by the statistical agencies, the question became how to inform respondents about these changes without alarming them and harming response rates. This paper reports on an experiment conducted in a survey of public-school principals with a simple random sample (n = 4,122). The experiment examined differences in response rates across three versions of pledges disclosing the cybersecurity monitor: one pledge specified explicitly the role of DHS, one referred to the monitors as “Federal employees and contractors,” and a control version referred implicitly to DHS monitors via a legal citation. Half of the pledges included the penalty language, and the other half did not. There were no statistically significant differences between any of the experimental comparisons. We conclude that, of the pledges studied, the DHS version of the pledge with penalty language provides respondents with the most accurate information for deciding whether to participate in the survey without harming response rates.

The effects of nonresponse and follow-up mailings on a citizen survey of law enforcement services

ABSTRACT Law enforcement agencies in many countries conduct surveys to learn about community opinions and concerns but low response rates can raise doubts about the generalizability of the results. This study examined differences between responders vs. nonresponders, and early vs. late responders, in a mailed survey of citizen satisfaction about law enforcement, and whether survey estimates changed with follow-up mailings. The univariate analysis revealed significant but small race/ethnicity and age differences between responders and nonresponders. Nonresponders tended to be nonwhite rather than white and to be younger. Late responders reported more negative attitudes toward law enforcement than did early responders with respect to perceived procedural justice and police legitimacy, although differences were small. While follow-up mailings increased the number of returns, this study found that the characteristics and attitudes of responders changed very little with additional mailings. The multivariate analysis revealed no evidence of multivariate bias, in particular, late survey responses did not change the positive association between procedural justice and police legitimacy. The findings imply that criminal justice scholars and law enforcement agencies that conduct citizen satisfaction surveys might consider increasing the initial sample size to boost statistical precision and power, or targeting unlikely respondents to reduce nonresponse bias, instead of investing in costly follow-up mailings. For police practice, the findings do not rule out invariance assertions about procedural justice. To the extent late responders resemble nonresponders, the impact of procedural justice may not depend on citizens’ survey response propensity.

Benefits of Adaptive Design Under Suboptimal Scenarios: A Simulation Study

Abstract Theories have suggested that adaptive survey designs may reduce biases and variances of survey estimates more so than post-survey adjustment. However, previous research has not incorporated practical constraints into theorizing the utility of adaptive design. How would design factors and suboptimal auxiliary information influence the effect of adaptive design? The current simulation considers three separate components in adaptive design: (1) Modeling—how sample cases’ response propensities are predicted by a response propensity model; (2) Operationalization—how researchers decide to differentially allocate recruitment effort to sample cases, and (3) Achievement—the sizes of the changes in response propensities that are achieved by the adaptive strategies. Each component influences the effect of adaptive design on the biases and variances of survey estimates. The simulation study presented here suggests that adaptive designs based on adequate auxiliary information improve survey estimates more so than post-survey adjustment. However, adaptive designs may backfire if the strategies are not correctly designed due to a lack of critical auxiliary variables.