Nonsubstantive Responses Research Articles

Survey Mode and Item Non-Substantive Response: Insights from a Replication Study and Other Recent Trials

This is a replication of a Survey Practice article that was published in September 2022 that explored survey administration mode and non-substantive response. It revisits the findings from the first study using a randomized controlled design. In this study, we found that on average phone respondents provided more non-substantive responses than web survey respondents, though the portion of phone and web respondents with one or more non-substantive response was similar. Another paper we published in Survey Practice in 2023 also explored mode effects with random assignment to text-to-web and phone groups. Thus, this is the third article by the authors on this topic released in Survey Practice. As part of this paper, we discuss findings from all three articles and contextualize them in relationship to other research on item non-substantive response and survey mode (specifically web and phone).

Mixed-methods analysis of select issues reported in the 2016 World Health Organization verbal autopsy questionnaire

Use of a standardized verbal autopsy (VA) questionnaire, such as the World Health Organization (WHO) instrument, can improve the consistency and reliability of the data it collects. Systematically revising a questionnaire, however, requires evidence about the performance of its questions. The purpose of this investigation was to use a mixed methods approach to evaluate the performance of questions related to 14 previously reported issues in the 2016 version of the WHO questionnaire, where there were concerns of potential confusion, redundancy, or inability of the respondent to answer the question. The results from this mixed methods analysis are discussed across common themes that may have contributed to the underperformance of questions and have been compiled to inform decisions around the revision of the current VA instrument. Quantitative analysis of 19,150 VAs for neonates, children, and adults from five project teams implementing VAs predominately in Sub-Saharan Africa included frequency distributions and cross-tabulations to evaluate response patterns among related questions. The association of respondent characteristics and response patterns was evaluated using prevalence ratios. Qualitative analysis included results from cognitive interviewing, an approach that provides a detailed understanding of the meanings and processes that respondents use to answer interview questions. Cognitive interviews were conducted among 149 participants in Morocco and Zambia. Findings from the qualitative and quantitative analyses were triangulated to identify common themes. Four broad themes contributing to the underperformance or redundancy within the instrument were identified: question sequence, overlap within the question series, questions outside the frame of reference of the respondent, and questions needing clarification. The series of questions associated with one of the 14 identified issues (the series of questions on injuries) related to question sequence; seven (tobacco use, sores, breast swelling, abdominal problem, vomiting, vaccination, and baby size) demonstrated similar response patterns among questions within each series capturing overlapping information. Respondent characteristics, including relationship to the deceased and whether or not the respondent lived with the deceased, were associated with differing frequencies of non-substantive responses in three question series (female health related issues, tobacco use, and baby size). An inconsistent understanding of related constructs was observed between questions related to sores/ulcers, birth weight/baby size, and diagnosis of dementia/presence of mental confusion. An incorrect association of the intended construct with that which was interpreted by the respondent was observed in the medical diagnosis question series. In this mixed methods analysis, we identified series of questions which could be shortened through elimination of redundancy, series of questions requiring clarification due to unclear constructs, and the impact of respondent characteristics on the quality of responses. These changes can lead to a better understanding of the question constructs by the respondents, increase the acceptance of the tool, and improve the overall accuracy of the VA instrument.

Does survey administration mode relate to non-substantive responses? A comparison of email versus phone administration of a residential utility-sponsored energy efficiency program survey

Program evaluation research often relies heavily on surveys and increasingly self-administered online tools. Often, however, a portion of program participants or customers do not have or do not provide email addresses. This study investigates if the type of contact information that members of a population provide relates to non-substantive responses (choosing "don’t know or “prefer not to answer”). We found evidence that survey administration mode relates to non-substantive responses, even while controlling for demographic factors.

Online listening responses and e-learning performance

PurposeThis research investigates the impact of learners' non-substantive responses in online course forums, referred to as online listening responses, on e-learning performance. A common type of response in online course forums, online listening responses consist of brief, non-substantive replies/comments (e.g. “agree,” “I see,” “thank you,” “me too”) and non-textual inputs (e.g. post-voting, emoticons) in online discussions. Extant literature on online forum participation focuses on learners' active participation with substantive inputs and overlooks online listening responses. This research, by contrast, stresses the value of online listening responses in e-learning and their heterogeneous effects across learner characteristics. It calls for recognition and encouragement from online instructors and online forum designers to support this activity.Design/methodology/approachThe large-scale proprietary dataset comes from a leading MOOC (massive open online courses) platform in China. The dataset includes 68,126 records of learners in five MOOCs during 2014–2018. An ordinary least squares model is used to analyze the data and test the hypotheses.FindingsOnline listening responses in course forums, along with learners' substantive inputs, positively influence learner performance in online courses. The effects are heterogeneous across learner characteristics, being more prominent for early course registrants, learners with full-time jobs and learners with more e-learning experience, but weaker for female learners.Originality/valueThis research distinguishes learners' brief, non-substantive responses (online listening responses) and substantive inputs (online speaking) as two types of active participation in online forums and provides empirical evidence for the importance of online listening responses in e-learning. It contributes to online forum research by advancing the active-passive dichotomy of online forum participation to a nuanced classification of learner behaviors. It also adds to e-learning research by generating insights into the positive and heterogeneous value of learners' online listening responses to e-learning outcomes. Finally, it enriches online listening research by introducing and examining online listening responses, thereby providing a new avenue to probe online discussions and e-learning performance.

A consumer perspective on managing the consequences of chain liability

AbstractConsumers tend to hold a focal firm responsible for its suppliers' unsustainable practices (chain liability), suggesting that firms need effective responses that can mitigate negative consumer reactions. In applying psychological contract theory to investigate recovery efforts related to such chain liability, the current study addresses three broad focal firm responses: Do nothing, choose a nonsubstantive response that verbally clarifies its own and the supplier's roles in the incident, or substantively rectify the supplier's wrongdoing with sustainability‐focused responses, such as termination, monitoring or development. With a vignette‐based experiment, we examine consumer perceptions and behaviors in three stages: before the unsustainable supplier incident (pre‐incident), after the incident (post‐incident) and after the focal firm has responded (post‐response). A nonsubstantive, clarification response decreases consumers' purchase intentions; substantive focal firm activities increase purchase intentions, though not fully back to pre‐incident levels. For consumers, termination, monitoring and development seem like equally adequate responses. Although combining several substantive responses offers even greater effectiveness for recovering purchase intentions, it still falls short of reaching pre‐incident levels. Thus, our findings demonstrate the focal firm's capacity to address suppliers' unsustainable practices substantively and recover, at least partially, its damaged relationship with consumers.

Comparison of Data Quality between PAPI and CAPI Modes of Data Collection in Longitudinal Household Surveys: RLMS–HSE Experiment Results

This article presents the results of an experimental study on how the transition from PAPI to CAPI modes affected data quality in longitudinal household surveys. The study was conducted in 2018–2019 within the Russian Longitudinal Monitoring Survey (RLMS–HSE). In the previous paper, which was based on data from the 26th wave of the RLMS HSE, it was shown that the use of CAPI leads to a significant decrease in the rate of non-substantive responses (“Don’t know”), as well as significant differences in sensitive questions. This paper was aimed at verifying these findings using new data collected during the 27th wave of the RLMS–HSE. The results show that the use of CAPI leads to a decrease in the rate of non-substantive responses, which helps to improve data quality. However, it was shown that the use of CAPI could lead to an increase in social desirability bias.

Evaluating the Data Quality of a National Sample of Young Sexual and Gender Minorities Recruited Using Social Media: The Influence of Different Design Formats

Social media recruitment is no longer an uncharted avenue for survey research. The results thus far provide evidence of an engaging means of recruiting hard-to-reach populations. Questions remain, however, regarding whether the data collected using this method of recruitment produce quality data. This article assesses one aspect that may influence the quality of data gathered through nonprobability sampling using social media advertisements for a hard-to-reach sexual and gender minority youth population: recruitment design formats. The data come from the Survey of Today’s Adolescent Relationships and Transitions, which used a variety of forms of advertisements as survey recruitment tools on Facebook, Instagram, and Snapchat. Results demonstrate that design decisions such as the format of the advertisement (e.g., video or static) and the use of eligibility language on the advertisements impact the quality of the data as measured by break-off rates and the use of nonsubstantive responses. Additionally, the type of device used affected the measures of data quality.

The prevalence of declarative and indirect yes/no Questions when children testify in criminal cases of child sexual abuse in the United States

SummaryThe present study assessed how attorneys questioned children in cases of child sexual abuse in the United States tried between 2005 and 2015. Trial testimonies (N = 134) of 5‐ to 17‐year‐olds (M = 12 years old) were coded for the linguistic form of attorneys' questions and children's subsequent responses. Three fourths of all questions were closed ended. Both declarative (statement question; e.g., “And he hit you?”; 21% of questions) and indirect yes/no questions (beginning with an indirect speech act; e.g., “Do you remember X?”; 11% of questions) were common, and produced potentially problematic responses, in comparison with forced‐choice and yes/no questions. Declarative questions elicited the highest rates of unelaborative responses whereas indirect yes/no questions elicited the highest rate of nonsubstantive responses. The findings highlight the importance for researchers to better assess children's responses to declarative questions and for prosecuting attorneys to cautiously use declarative and indirect yes/no questions when questioning children.

The Dynamics of “Neither Agree Nor Disagree” Answers in Attitudinal Questions

Abstract Attitudinal questions are an integral part of surveys in the social sciences. Previous research based on cross-sectional data has shown that both respondents’ characteristics and questionnaire design can lead to higher use of midpoint responses when questions are operationalized on uneven rating scales. To further current understanding of this phenomenon, in this article we apply hybrid regression models to analyze differences between respondents but also developmental changes within respondents, allowing for a more profound interpretation of the dynamics behind midpoint responses, theoretically explained by satisficing behavior. For our midpoint analyses, we use a set of attitudinal item blocks asked in the British Household Panel Survey (BHPS) from 1991 to 2008. Respondents’ reports and interviewers’ judgments offer satisficing-related indicators regarding ability and motivation, enabling particularly accurate analyses of midpoint response behavior in terms of the “neither agree nor disagree” option. Results show that depending on respondents’ specific characteristics, higher use of the midpoint cannot automatically be equated with nonsubstantive response behavior. Furthermore, we demonstrate that specific indicators related to low ability and motivation do not uniformly increase midpoint responses. Application of the hybrid model reveals that changes in respondents’ characteristics do not affect midpoint response. We speculate that there are unobserved personality attributes that affect the propensity to midpoint response and conclude by reflecting on reasons for the increase in midpoint responses over the years.

Re-Examining the Middle Means Typical and the Left and Top Means First Heuristics Using Eye-Tracking Methodology

Abstract Web surveys are a common self-administered mode of data collection using written language to convey information. This language is usually accompanied by visual design elements, such as numbers, symbols, and graphics. As shown by previous research, such elements of survey questions can affect response behavior because respondents sometimes use interpretive heuristics, such as the “middle means typical” and the “left and top means first” heuristics when answering survey questions. In this study, we adopted the designs and survey questions of two experiments reported in Tourangeau, Couper, and Conrad (2004). One experiment varied the position of nonsubstantive response options in relation to other substantive response options and the second experiment varied the order of the response options. We implemented both experiments in an eye-tracking study. By recording respondents’ eye movements, we are able to observe how they read question stems and response options and we are able to draw conclusions about the survey response process the questions initiate. This enables us to investigate the mechanisms underlying the two interpretive heuristics and to test the assumptions of Tourangeau et al. (2004) about the ways in which interpretive heuristics influence survey responding. The eye-tracking data reveal mixed results for the two interpretive heuristics. For the middle means typical heuristic, it remains somewhat unclear whether respondents seize on the conceptual or visual midpoint of a response scale when answering survey questions. For the left and top means first heuristic, we found that violations of the heuristic increase response effort in terms of eye fixations. These results are discussed in the context of the findings of the original studies.

Public perceptions of energy policies: Predicting support, opposition, and nonsubstantive responses

Public policy preferences can impact policymaking, from influencing what policies are or are not implemented to legitimizing policy decisions. Researchers have examined public support/opposition to a wide variety of energy issues, as well as predictors of these preferences. However, researchers have typically excluded the views of those who say they “don’t know” or do not respond to questions about a particular policy, even when they make up a sizable portion of the population. As a result, we know little about those who provide nonsubstantive responses to energy policy questions, as compared to those who provide support/opposition responses. In this article, I examine the roles of issue awareness, risk perception, and socio-demographic characteristics in predicting support, opposition, and not sure responses to three different energy development issues: nuclear power, the Keystone XL pipeline, and hydraulic fracturing. One of the key findings is that the risk perception predictors of nonsubstantive responses more closely resemble the risk perception predictors of opposition rather than the predictors of support. By excluding the policy preferences of those who “don’t know” or do not answer, researchers are likely providing policymakers with estimates that underrepresent the views of those who are more concerned about environmental risk.

Comparing Grids With Vertical and Horizontal Item-by-Item Formats for PCs and Smartphones

Much research has been done comparing grids and item-by-item formats. However, the results are mixed, and more research is needed especially when a significant proportion of respondents answer using smartphones. In this study, we implemented an experiment with seven groups ( n = 1,476), varying the device used (PC or smartphone), the presentation of the questions (grids, item-by-item vertical, item-by-item horizontal), and, in the case of smartphones only, the visibility of the “next” button (always visible or only visible at the end of the page, after scrolling down). The survey was conducted by the Netquest online fieldwork company in Spain in 2016. We examined several outcomes for three sets of questions, which are related to respondent behavior (completion time, lost focus, answer changes, and screen orientation) and data quality (item missing data, nonsubstantive responses, instructional manipulation check failure, and nondifferentiation). The most striking difference found is for the placement of the next button in the smartphone item-by-item conditions: When the button is always visible, item missing data are substantially higher.

Open narrative questions in PC and smartphones: is the device playing a role?

Most survey questions are closed questions, where respondents have to select an answer from a proposed set of alternatives. However, a lot of surveys also include, at least occasionally, some open questions. Open questions that call for elaborated and developed answers, called “open narrative questions”, are used when the researchers want to go deeper into what the respondents think. This paper compares the answers to open narrative questions when the respondent is participating in a PC survey, in a smartphone-not-optimised survey or in a smartphone-optimised survey. The experiment was carried out in Spain using data collected by the Netquest online access panel. Respondents were assigned randomly to each type of device and survey format, in two successive waves. Because respondents have to type in their answer, we expect differences between devices, linked with the size and the kind of keyboards (i.e. physical versus digital, touch-screen or not). Differences are observed between answers that come from PCs and smartphones for the response time per written character, for the number of total characters and for the use of abbreviations, but not for the non-answer and non-substantive responses. No differences are observed between optimised and not optimised versions for smartphones, except for the response time per character written.

Response order effects in the Youth Tobacco Survey: Results of a split-ballot experiment

Response order effects are changes in answers to close-ended questions that arise by varying the order of the response options (Krosnick and Alwin 1987). Two types of response order effects have been documented in previous studies: primacy effects and recency effects (Bishop and Smith 2001; Holbrook et al. 2007; Knauper 1999). Primacy effects, when response options presented earlier in a list of options are selected more often than ones at the end, have been typically observed in paper and pencil self-administered surveys (Krosnick and Alwin 1987). Recency effects, where response options presented later in a list are selected more often, primarily occur in surveys where questions are presented orally (Krosnick and Alwin 1987). Response order effects have been documented with adult respondents, but few studies have examined these effects with children or adolescents (Fuchs 2005). It is believed that no studies on response order effects have been conducted on a tobacco-specific survey. Questionnaires often include non-substantive response options, such as “no opinion” or “don't know,” to allow respondents who have no true opinions to select these options. People with lower education are more likely to be attracted to the “no opinion” response option (Krosnick et al. 2002). As a result, it is recommended that in children's surveys, researchers should minimize the use of non-substantive response categories (Bell 2007). However, in circumstances where it is not possible to avoid using non-substantive response options, no research has shown if the order of these response options has any effect on responses. We assess whether primacy and/or recency effects occur in a self-administered tobacco-related questionnaire among youth and will determine the effects of reversing the order of response options and specifically changing the position of a non-applicable response category. This study is one of very few studies to examine the effect of response order in children's responses, and it is the first to investigate the impact of moving a non-applicable response category from first to last position. To the best of our knowledge, it is also the first to document the presence of response order effects in a tobacco survey and differentiate the effects for tobacco users verses non-users.

Data Quality in PC and Mobile Web Surveys

The considerable growth in the number of smart mobile devices with a fast Internet connection provides new challenges for survey researchers. In this article, I compare the data quality between two survey modes: self-administered web surveys conducted via personal computer and those conducted via mobile phones. Data quality is compared based on five indicators: (a) completion rates, (b) response order effects, (c) social desirability, (d) non-substantive responses, and (e) length of open answers. I hypothesized that mobile web surveys would result in lower completion rates, stronger response order effects, and less elaborate answers to open-ended questions. No difference was expected in the level of reporting in sensitive items and in the rate of non-substantive responses. To test the assumptions, an experiment with two survey modes was conducted using a volunteer online access panel in Russia. As expected, mobile web was associated with a lower completion rate, shorter length of open answers, and similar level of socially undesirable and non-substantive responses. However, no stronger primacy effects in mobile web survey mode were found.

Effects of Survey Question Comprehensibility on Response Quality

Many studies have shown that vague or ambiguous questions are often interpreted idiosyncratically by respondents and thus can increase measurement error. This article provides some evidence that the cognitive effort required to comprehend survey questions affects data quality in a similar way. A web survey experiment revealed that respondents receiving less comprehensible questions provided lower-quality responses (as indicated by breakoff rates, number of nonsubstantive responses, number of neutral responses, and over-time consistency) than respondents receiving control questions that were easier to comprehend. Moreover, interaction effects of question comprehensibility with respondents’ verbal skills and their motivation to answer surveys were found. These findings indicate that survey designers should minimize the cognitive effort required to comprehend their questions and the article suggests specific ways how to do so.

Don't Know Much about Democracy: Reporting Survey Data with Nonsubstantive Responses

AbstractLarge majorities in nearly every country support democracy, according to studies of cross-national surveys. But many of these reports have treated as missing data persons who did not provide a substantive response when asked to offer an opinion about the suitability of democracy as a regime type for their country, which has led to substantial overestimates of expressed support for democracy in some countries. This article discusses the consequences of excluding such nonsubstantive responses and offers suggestions to improve the study of popular support for democracy.

Understanding the effects of substantive responses on trust following a transgression

Four experiments were conducted to investigate the implications of ‘substantive’ responses for the repair of trust following a violation and the cognitive processes that govern how and when they are effective. These studies examined two forms of substantive responses, penance and regulation, that represent different categories of trust repair attempts. The findings from Studies 1–3 suggest that both can be effective to the extent that they elicit the crucial mediating cognition of perceived repentance. Data from Study 2 revealed that trustors saw signals of repentance as more informative when the transgression was due to a lapse of competence than due to a lapse of integrity. Study 4 compared these substantive responses to apologies (a non-substantive response) and revealed that, despite their surface-level differences, they each repaired trust through ‘perceived repentance.’ The paper offers an integrative framework for understanding the relationships among a range of trustor responses.

Telephone and web‐based pediatric day surgery questionnaires

Patient questionnaires are popular tools for assessing and improving service quality, especially as administrators are increasingly expected to consider the patient's voice in their decision making. Despite web-based questionnaire advantages, they have not been previously compared to telephone questionnaires for assessing quality. The purpose of this paper is to compare telephone questionnaire administration with a web-based version. Day surgery patients from a tertiary pediatric hospital completed a telephone interview and a web-based questionnaire with identical questions. The appropriateness of the web version as a telephone version substitute was ascertained by comparing the number of changes in responses, non-responses, differences in means, the number of non-substantive responses and reliability. The web-based questionnaire tended towards more negative responses. The mean number of missing responses did not differ between versions, although the web-questionnaire had more "not sure" responses. Inter-rater reliability was acceptable. Parents without internet access were unable to participate. The web-based questionnaire is a good substitute for telephone-administered questionnaires. The paper shows that parents were able to rate items more candidly owing to the increase in privacy and lack of interviewer bias, which is crucial for improving health service quality.

Predicting Unsure Responses to a Proposed Moose Hunt in Anchorage, Alaska

We examined correlates of nonsubstantive survey responses (i.e., NSR, unsure) to a voting intention question for a proposed moose (Alces alces) hunt. We compared respondents who gave unsure responses (15% of sample) against those who would vote for (51%) or against (34%) the hunt using demographic characteristics, tendency to use NSRs throughout the survey, psychological indicators, prior negative experiences with moose, and hunting participation. Data were obtained from a mail survey (n = 971) of Anchorage, Alaska residents. Respondents with unsure voting intentions were older, female, more likely to give NSRs throughout the survey, and tended to be non-hunters. NSR (i.e., unsure) respondents also had fewer negative experiences with moose, did not have a strong attitude toward the hunt, and were ambivalent toward wildlife in general compared to those who voiced an opinion either for or against the hunt. We used discriminant analysis to identify variables that best differentiated among the three voting intention groups (unsure, for, against). Results suggested that nonsubstantive responses were not random and should not be ignored. Findings are discussed in terms of implications for management and survey design.