Careless Responses Research Articles

Careless responding in crowdsourced alcohol research: A systematic review and meta-analysis of practices and prevalence.

Crowdsourcing-the process of using the internet to outsource research participation to "workers"-has considerable benefits, enabling research to be conducted quickly, efficiently, and responsively, diversifying participant recruitment, and allowing access to hard-to-reach samples. One of the biggest threats to this method of online data collection however is the prevalence of careless responders who can significantly affect data quality. The aims of this preregistered systematic review and meta-analysis were: (a) to examine the prevalence of screening for careless responding in crowdsourced alcohol-related studies; (b) to examine the pooled prevalence of careless responding; and (c) to identify any potential moderators of careless responding across studies. Our review identified 96 eligible studies (∼126,130 participants), of which 51 utilized at least one measure of careless responding, 53.2%, 95% CI [42.7%-63.3%]; ∼75,334 participants. Of these, 48 reported the number of participants identified by careless responding method(s) and the pooled prevalence rate was ∼11.7%, 95% CI [7.6%-16.5%]. Studies using the MTurk platform identified more careless responders compared to other platforms, and the number of careless response items was positively associated with prevalence rates. The most common measure of careless responding was an attention check question, followed by implausible response times. We suggest that researchers plan for such attrition when crowdsourcing participants and provide practical recommendations for handling and reporting careless responding in alcohol research. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Learning From Mistakes: Impact of Careless Responses on Counseling Research Using Amazon’s Mechanical Turk

We investigated the effect of careless or insufficient effort (C/IE) responses in a study using Amazon’s Mechanical Turk. A factor mixture model was used to identify latent classes based on the pattern of responses with biases and examine the effect of C/IE responses on the fit of the theoretical model.

Item wording effects in self-report measures and reading achievement: Does removing careless respondents help?

This study examined the impact of item wording effects on the factor structure of the Student Confidence in Reading (SCR) scale in the 2016 Progress in International Reading Literacy Study (PIRLS). First, using a large sample of students from seven English-speaking countries, item wording effects in the SCR scale were investigated for the entire sample and the subgroups of students based on their reading achievement levels in PIRLS 2016. Second, the presence of item wording effects was re-examined after identifying and removing students who answered negatively worded items inconsistently. The results showed that the use of negatively worded items distorted the factor structure of the SCR scale, leading to an unintentionally multidimensional factor structure. Removing students with careless responses had a negligible impact on the model fit for students at the lowest achievement level, whereas it considerably improved the model fit for other students with moderate to high reading achievement.

Outlier-response pattern checks to improve measurement with the Montgomery-Asberg depression rating scale (MADRS)

Symptom manifestations in affective disorders can be subtle. Small imprecisions in measurement can lead to incorrect estimation of change. Previously, expert-derived scoring inconsistency flags were developed for MADRS. Currently, we derive empirically based outlier-pattern flags, to further detect imprecisions in ratings. NEWMEDS data repository of almost 25,000 MADRS administrations from 11 registration trials of antidepressants was used to identify outlier response patterns reflecting potentially careless responses. Coverage of these flags was compared to previously published expert derived flags. Both sets of flags were also further tested in Monte Carlo simulated data as a proxy to applying flags under conditions of known inconsistency. The outlier flags derived provide cutting points to identify: (1) under and overuse of values (e.g., Scoring “1″ on 6 or more items), (2) disproportionate use of even or odd response choices (e.g., 8 or more odd values), (3) longest consecutive use of value (e.g., more than 5 items in a row scored with same value), (4) high variability within administration (standard deviation greater than 1.8), (5) outlier responses on multiple items (i.e., multivariate outliers), and (6) outlier scoring (e.g., scoring 4,5 or 6 on item 1). Outlier response flags were raised in 26% of the MADRS administration and in 97% of the Monte Carlo data. Of administrations with no expert flag, 21.7% had an outlier flag and of administrations with at least one expert flag, 27.7% also had an outlier flag. Outlier-pattern flags appear to be a useful adjunct to expert derived flags in the quest to improve measurement in clinical trials.

Careless responses and construct validity of Wong-Law Emotional Intelligence Scale.

Careless response in the investigation of emotional intelligence has not been explicitly addressed. In a sample of 180 Spanish adults responding to the Wong-Law Emotional Intelligence Scale, a small prevalence (7.2%) was detected. The impact was small on the psychometric parameters, but they were less inflated, more realistic, and precise.

Identification of careless responding in ecological momentary assessment research: From posthoc analyses to real-time data monitoring.

With the emerging ubiquity of cell phones, ecological momentary assessment (EMA) as a set of methods enable researchers to study momentary social, psychological, and affective responses to everyday life. Additionally, EMA enables researchers to acquire longitudinal data without the need for multiple lab visits. As the use of EMA in research increases, so too does the necessity of determining what constitutes valid or careless individual EMA responses to ensure validity and replicability of findings. Currently, EMA studies solely consider the response rate of a participant for exclusion. Yet, other features of an assessment can help to determine whether a response is careless or implausible. Here, we examined over 18,000 EMA text message responses of individual affect items to derive a data-driven model of what constitutes a "careless response." Results from this study indicate that an overly fast time to complete items (≤ 1 s), an overly narrow within assessment response variance (SD ≤ 5), and the percentage of items that fall at the mode (≥ 60%) are independent and reliable indicators of a careless response. Excluding careless responses such as these remove implausible positive correlations among psychometric antonyms (e.g., relaxed and anxious). Further, by identifying and removing careless responses, we also identify careless responders, participants who could be removed from group analyses. We use these results to develop and introduce an R package, EMAeval, so EMA researchers may similarly identify careless responses and responders either online during data collection or posthoc, after data collection has completed. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Respondent Burden Effects on Item Non-Response and Careless Response Rates: An Analysis of Two Types of Surveys

The respondent burden refers to the effort required by a respondent to answer a questionnaire. Although this concept was introduced decades ago, few studies have focused on the quantitative detection of such a burden. In this paper, a face-to-face survey and a telephone survey conducted in Valencia (Spain) are analyzed. The presence of burden is studied in terms of both item non-response rates and careless response rates. In particular, two moving-window statistics based on the coefficient of unalikeability and the average longstring index are proposed for characterizing careless responding. Item non-response and careless response rates are modeled for each survey by using mixed-effects models, including respondent-level and question-level covariates and also temporal random effects to assess the existence of respondent burden during the questionnaire. The results suggest that the sociodemographic characteristics of the respondents and the typology of the question impact item non-response and careless response rates. Moreover, the estimates of the temporal random effects indicate that item non-response and careless response rates are time-varying, suggesting the presence of respondent burden. In particular, an increasing trend in item non-response rates in the telephone survey has been found, which supports the hypothesis of the burden. Regarding careless responding, despite the presence of some temporal variation, no clear trend has been identified.

Bots, Fraud, and Careless Responding: Challenges of Bad-faith Responses in Survey Research

This symposium is aimed at bringing together a panel of scholars to discuss the prevalence and challenges associated with bad faith responses as well as opportunities to improve the quality of responses in survey research. Bad faith responding is defined as a) intentionally careless responses (e.g., participants responding without reading items or randomly), b) fraudulent participants (e.g., individuals lying about who they are or completing multiple surveys), and c) bots (computer programs) completing one or more surveys and producing fake data. The panel discussion will focus on the trends of bad faith responding in organizational research, the challenges associated with gathering survey data, and the opportunities for preventing, detecting, and mitigating bad faith responses. The primary goals of this panel symposium are to build awareness of bad faith responding, the costs of convenience sampling, and to inform researchers of ways to resolve issues that can arise. By bringing together researchers who have examined these issues with practitioners working for platforms that help manage the survey research process (e.g. – Qualtrics, MTurk), we provide a unique perspective on the prevalence and impact of bad faith survey responding in management research.

Detecting Careless Responding in Survey Data Using Stochastic Gradient Boosting.

Careless responding is a bias in survey responses that disregards the actual item content, constituting a threat to the factor structure, reliability, and validity of psychological measurements. Different approaches have been proposed to detect aberrant responses such as probing questions that directly assess test-taking behavior (e.g., bogus items), auxiliary or paradata (e.g., response times), or data-driven statistical techniques (e.g., Mahalanobis distance). In the present study, gradient boosted trees, a state-of-the-art machine learning technique, are introduced to identify careless respondents. The performance of the approach was compared with established techniques previously described in the literature (e.g., statistical outlier methods, consistency analyses, and response pattern functions) using simulated data and empirical data from a web-based study, in which diligent versus careless response behavior was experimentally induced. In the simulation study, gradient boosting machines outperformed traditional detection mechanisms in flagging aberrant responses. However, this advantage did not transfer to the empirical study. In terms of precision, the results of both traditional and the novel detection mechanisms were unsatisfactory, although the latter incorporated response times as additional information. The comparison between the results of the simulation and the online study showed that responses in real-world settings seem to be much more erratic than can be expected from the simulation studies. We critically discuss the generalizability of currently available detection methods and provide an outlook on future research on the detection of aberrant response patterns in survey research.

Detection of Careless Responses in Online Surveys Using Answering Behavior on Smartphone

Some respondents make careless responses due to the “satisficing,” which is an attempt to complete a questionnaire as quickly and easily as possible. To obtain results that reflect a fact, detecting satisficing and excluding the responses with satisficing from the analysis targets are required. One of the devised methods detects satisficing by adding questions that check violations of instructions and inconsistencies. However, this approach may cause respondents to lose their motivation and prompt them to satisficing. Additionally, a deep learning model that automatically answers these questions was reported. This threatens the reliability of the conventional method. To detect careless responses without inserting such screening questions, machine learning (ML) detection using data obtained from answer results was attempted in a previous study, with a detection rate of 55.6%, which is not sufficient from the viewpoint of practicality. Therefore, we hypothesized that a supervised ML model with a higher detection rate could be constructed by using on-screen answering behavior as features. However, (1) no existing questionnaire system can record on-screen answering behavior and (2) even if the answering behavior can be recorded, it is unclear which answering behavior features are associated with satisficing. We developed an answering behavior recording plug-in for LimeSurvey, an online questionnaire system used all over the world, and collected a large amount of data (from 5,692 people) in Japan. Then, a variety of features were examined and generated from answering behavior, and we constructed ML models to detect careless responses. We call this detection method the ML-ABS (ML-based answering behavior scale). Evaluation by cross-validation demonstrated that the detection rate for careless responses was 85.9%, which is much higher than the previous ML method. Among the various features we proposed, we found that reselecting the Likert scale and scrolling particularly contributed to the detection of careless responses.

Combatting carelessness: Can placement of quality check items help reduce careless responses?

There is extensive literature on identifying careless responses in survey data and acknowledging their negative impact on accurate data analysis (Goodman, Cryder, & Cheema, 2013; Huang et al., 2015a, b; Meade and Craig, 2012). However, there are very limited findings on how researchers can help prevent participants from responding carelessly in the first place. The current study manipulated the placement of a quality check item (i.e., early placement versus late placement) and showed that participants were less likely to carelessly respond when the quality check items were placed later in the survey. Quality check items were also able to identify careless responses more than other approaches (i.e., LongString index, completion time index, and self-reported indicators). When using quality check items, it appears to be best when used toward the end of the survey. There are plenty of opportunities for future studies to expand on this research to uncover ways that can help deter careless responding.

The Effect of Users’ Motivations and Interactivity on Online Word of Mouth

The purpose of this study is to investigate the impacts of two different kinds of interactivity, such as user-to-user and user-to-media interaction, on the relationship between SNS motivation and online word of mouth (WOM). An online survey was conducted with SNS users in Korea. Using the convenience-sampling method, 300 surveys were collected and 295 were used in the actual analysis after excluding data with careless responses or missing values. Hypotheses were tested using Structure Equation Model (SEM) and path analysis by using AMOS22. The results indicate that four different SNS motivations (self-expression, relational, fun, and browsing motivation) have a partially significant positive effect on perceived user-to-user and user-to-media interaction in SNS. Although both user-to-user interactivity and user-to-media interactivity were found to have a significant effect on online word of mouth, by comparing the standardized regression coefficients in these relationships, it was found that user-to-user interactivity has a greater effect on online WOM than user-to-media interactivity. These results show that the motivated SNS users want to express their desire to communicate with other users in contrast than their desire to learn media functions when motivated SNS users reveal their personalities, knowledge, and abilities. Theoretical and managerial implications are discussed.

New Perspectives in Computing the Point of Subjective Equality Using Rasch Models.

In psychophysics, the point of subject equality (PSE) is any of the points along a stimulus dimension at which a variable stimulus (visual, tactile, auditory, and so on) is judged by an observer to be equal to a standard stimulus. Rasch models have been found to offer a valid solution for computing the PSE when the method of constant stimuli is applied in the version of the method of transitions. The present work provides an overview of the procedures for computing the PSE using Rasch models and proposes some new developments. An adaptive procedure is described that allows for estimating the PSE of an observer without presenting him/her with all stimuli pairs. This procedure can be particularly useful in those situations in which psychophysical conditions of the individuals require that the number of trials is limited. Moreover, it allows for saving time that can be used to scrutinize the results of the experiment or to run other experiments. Also, the possibility of using Rasch-based fit statistics for identifying observers who gave unexpected judgments is explored. They could be individuals who, instead of carefully evaluating the presented stimuli pairs, gave random, inattentive, or careless responses, or gave the same response to many consecutive stimuli pairs. Otherwise, they could be atypical and clinically relevant individuals who deserve further investigation. The aforementioned developments are implemented using procedures and statistics that are well established in the framework of Rasch models. In particular, computerized adaptive testing procedures are used for efficiently estimating the PSE of the observers, whereas infit and outfit mean-squares statistics are used for detecting observers who gave unexpected judgments. Results of the analyses carried out on simulated data sets suggest that the proposed developments can be used in psychophysical experiments.

Further Validation of the Response Inconsistency Scale for the Personality Inventory for DSM-5

Assessment of protocol validity is essential for structured personality tests used in clinical decision making. Measures of inconsistent responding allow researchers and clinicians to identify random or careless response patterns that compromise an accurate interpretation of test results. Keeley and colleagues (2016) developed an Inconsistency scale (INC) for the widely used Personality Inventory for DSM-5 (PID-5; Krueger, Derringer, Markon, Watson, & Skodol. 2012. Initial construction of a maladaptive personality trait model and inventory for DSM-5. Psychological Medicine, 42(9), 1879–1890.). The INC produced highly promising results in initial validation studies, and the current study provides a series of additional tests of the adaptability of the INC item pairs across different populations, translations, and versions of the PID-5. Study 1 examines the diagnostic utility of a shortened version of the original INC scale (INC-S) that can be used with the 100-item version of the PID-5; optimum cut scores are identified for this short form adaptation. Study 2 cross-validates the INC-S and compares diagnostic utility to the INC in a sample that completed the full PID-5. Study 3 examines the diagnostic utility of the INC and INC-S using a German translation of the PID-5 with undergraduates and clinical patients. Overall, these validation studies provide robust support for the INC and INC-S scales to discriminate random-generated versus real PID-5 protocols.

A change-point analysis procedure based on weighted residuals to detect back random responding.

Careless or inattentive responding is frequently observed in questionnaire or survey data, which jeopardizes test validity and the generalizability of research findings. It is therefore very important to detect such response behavior. The most frequently encountered type of careless response behavior is back random responding (BRR). Literature suggests that BRR is challenging to detect, with reported power of detection around .5 or lower. Change point analysis (CPA), which is a widely used statistical process control method, can be applied to item response data to detect if aberrant behavior exists in a response pattern. Existing CPA methods, however, may not be suitable for detecting BRR, because the change may not be directional. In this article we propose a weighted-residual-based CPA procedure to detect BRR behavior. The performance of the proposed method was evaluated in a comprehensive simulation study and compared against 3 existing CPA methods. Results indicated that the proposed residual-based CPA procedure can detect BRR behavior with high power for tests of 20 items or longer, while keeping the Type-I error rate well under control. Compared with the 3 existing CPA methods, it leads to comparable empirical Type-I error but a gain in power of 17%-42%. An empirical study further illustrated the utility of the proposed method to detect BRR with a real dataset. Implications of the proposed method, its limitations, as well as future directions are provided at the end. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Psychometrics Anonymous: Does a Transparent Data Sharing Policy Affect Data Collection?

As researchers, we are advised to share our data to improve transparency and increase the reproducibility of experiments. Simultaneously, making data freely accessible can raise ethical questions regarding the participants’ privacy. We first outline the challenges regarding “open data” for researchers in light of the GDPR. Then, we turn to the impact of an open-access data sharing policy on the participants: could the participants’ knowledge about the future use of the data alter the data itself? Through two pre-registered studies (N = 193, collected on campus and N = 543, online participation), we investigate whether disclosing that anonymized data will be publicly shared vs. not shared influences a potential participants’ intention to take part in the study. Using both frequentist and Bayesian analysis, we conclude towards an absence of effect of a difference in data sharing policy on scores in the Big Five questionnaire and social desirability, careless response behavior, and results in the anchoring paradigm. In the second study, a lexicometric analysis of participants’ reactions to openly sharing data reveals a readiness to share data and support transparency under the condition of preserved anonymity. Hence, if anonymity can be ensured, there seems to be no methodological or ethical drawback in transparent and open data sharing policies for many psychological studies.

Game-like personality testing: An emerging mode of personality assessment

A concurrent, construct validity study was conducted to compare responses from a traditional personality instrument to responses from a game-like instrument designed to assess personality. Personality assessment research demonstrates that misleading and careless responses typically stem from lack of internal motivation or interest in the task. To address the underlying motivation issue, we created a text-based fantasy game where participants can select a variety of choices to reflect corresponding personality characteristics. Results from three studies show consistent moderate-to-strong correlations between choices in the game and scores on a traditional Five-Factor Model personality inventory, indicating that game-like personality assessments can be developed with acceptable construct validity in a mentally stimulating way.

Detection and Treatment of Careless Responses to Improve Item Parameter Estimation

In psychological and survey research, the prevalence and serious consequences of careless responses from unmotivated participants are well known. In this study, we propose to iteratively detect careless responders and cleanse the data by removing their responses. The careless responders are detected using person-fit statistics. In two simulation studies, the iterative procedure leads to nearly perfect power in detecting extremely careless responders and much higher power than the noniterative procedure in detecting moderately careless responders. Meanwhile, the false-positive error rate is close to the nominal level. In addition, item parameter estimation is much improved by iteratively cleansing the calibration sample. The bias in item discrimination and location parameter estimates is substantially reduced. The standard error estimates, which are spuriously small in the presence of careless responses, are corrected by the iterative cleansing procedure. An empirical example is also presented to illustrate the proposed procedure. These results suggest that the proposed procedure is a promising way to improve item parameter estimation for tests of 20 items or longer when data are contaminated by careless responses.

Robust maximum marginal likelihood (RMML) estimation for item response theory models.

Self-report data are common in psychological and survey research. Unfortunately, many of these samples are plagued with careless responses, due to unmotivated participants. The purpose of this study was to propose and evaluate a robust estimation method to detect careless or unmotivated responders, while leveraging item response theory (IRT) person-fit statistics. First, we outlined a general framework for robust estimation specific for IRT models. Subsequently, we conducted a simulation study covering multiple conditions in order to evaluate the performance of the proposed method. Ultimately, we showed that robust maximum marginal likelihood (RMML) estimation significantly improves detection rates for careless responders and reduces bias in item parameters across conditions. Furthermore, we applied our method to a real data set, to illustrate the utility of the proposed method. Our findings suggest that robust estimation coupled with person-fit statistics offers a powerful procedure to identify careless respondents for further review and to provide more accurate item parameter estimates in the presence of careless responses.

Social Interaction and Internet‐Based Surveys: Examining the Effects of Virtual and In‐Person Proctors on Careless Response

A lack of human interaction and environmental control in Internet‐based data collection have been suggested as possible antecedents of careless responding, which occurs when participants respond to survey items without regard for item content. To address these possible antecedents, this study investigated whether survey proctoring deterred careless response in an undergraduate sample by reducing environmental distractions. The study randomly assigned respondents to one of three proctoring conditions: remote online un‐proctored, remote online virtually proctored, and in‐person classroom proctored. Data quality was examined via nine careless response indicators. Analyses indicated that proctor presence had effects on a small number of careless response indicators. Virtually proctored participants performed better than un‐proctored participants on one of nine careless response indicators, and in‐person proctored participants performed better on two careless response indicators compared to un‐proctored participants. Environmental distraction fully mediated the relationship between in‐person proctor presence and self‐reported diligence. Implications for survey administration are discussed.