Careless Responding Research Articles

Identifying Careless Responses in Computer-Adaptive Affective Surveys Using Person Fit Analysis

ABSTRACT Careless responding is a pervasive concern in research using affective surveys. Although researchers have considered various methods for identifying careless responses, studies are limited that consider the utility of these methods in the context of computer adaptive testing (CAT) for affective scales. Using a simulation study informed by recent research, we explored the impact of careless response patterns on three types of response quality indicators: model-based person fit statistics, CAT-specific statistical process control indicators, and standalone indices of carelessness. We focused on over-consistency and random responding as forms of carelessness. All three types of indicators were sensitive to carelessness, with some differences related to CAT termination rules and item bank size. Our findings suggest that person fit analysis techniques may also be useful in affective CAT contexts to evaluating response quality from a measurement model perspective, including identifying carelessness. We discuss the implications of our work for research and practice.

Empowering commitment: unraveling the impact of motivating language and the mediating role of trust

PurposeThe purpose of this study is to examine the direct and indirect effects of motivating language on organizational commitment, as this phenomenon has drawn the attention of researchers.Design/methodology/approachThis study employs social exchange theory and motivating language theory to examine data collected from 217 full-time employees across various US-based companies. The proposed hypotheses were analyzed using the PLS-SEM method.FindingsThis study’s findings demonstrate that motivating language positively affects employees’ organizational commitment and affective trust but not cognitive trust, which mediates the relationship between motivating language and organizational commitment.Research limitations/implicationsThere are some limitations of our study that need to be mentioned. First, there are concerns about survey data collection via M-Turk (Shapiro et al., 2013). We attempted to overcome some of these problems by including questions to identify careless respondents. Also, we eliminated many respondents who completed the surveys in unreasonably short periods of time. Hence, we believe we accounted for response bias with these check points. Also, while we believe our final sample is a representative sample due to the significant amount of data elimination during the data collection, we believe that checking for non-response bias, as Armstrong and Overton (1977) suggest, is imperative. Unfortunately, due to the nature of M-Turk, that is impossible. However, M-Turk recruits respondents based on the parameters provided by the researchers, so we expect the non-respondents to be not significantly different from the respondents. In parallel to that, we acknowledge the limitations of our study sample. Due to that reason, our findings must be considered within the context of our sample parameters. We urge future researchers of this area to further validate our findings in different types of samples.Originality/valueTo the best of the authors’ knowledge, they are the first to analyze the impact of motivating language on organizational commitment and the mediating role of trust (cognitive and affective) in this relationship.

Preprocessing Experience-Sampling-Method Data: A Step-by-Step Framework, Tutorial Website, R Package, and Reporting Templates

Experience-sampling-method (ESM) studies have become a very popular tool to gain insight into the dynamics of psychological processes. Although the statistical modeling of ESM data has been widely studied, the preprocessing steps that precede such modeling have received relatively limited attention despite being a challenging phase. At the same time, adequate preprocessing of ESM data is crucial: It provides valuable information about the quality of the data and, importantly, helps to resolve issues in the data that may compromise the validity of statistical analyses. To support researchers in properly preprocessing ESM data, we have developed a step-by-step framework, a tutorial website that provides a gallery of R code, an R package, and templates to report the preprocessing steps. Particular attention is given to three different aspects in preprocessing: checking adherence to the study design (e.g., whether the momentary questionnaires were delivered according to the sampling scheme), examining participants’ response behaviors (e.g., compliance, careless responding), and describing and visualizing the data (e.g., examining distributions of variables).

Development of an Inconsistent Responding Scale for the Big Five Inventory-2

The Big Five Inventory − 2 (BFI-2) is a commonly used self-report assessment of normal personality trait domains (Extraversion, Agreeableness, Conscientiousness, Negative Emotionality, Open-Mindedness) and facets. To date, however, no direct measures of response distortion have been developed for it to identify potentially invalid responses. Such distortions (e.g., careless or random responding) can adversely impact data quality. The current study developed and provided initial validation data for an inconsistent responding scale within the BFI-2 to identify careless responders using two large undergraduate samples and a community sample. To create the scale, we first identified highly correlated BFI-2 item pairs in one undergraduate sample (N = 1,461) and then computed a total score by summing the absolute differences of these item pairs. This scale, the Detection of Response Inconsistency Procedure (DRIP), differentiated randomly generated and genuine data and generally correlated as expected with personality domains and other inconsistent responding scales across samples. The DRIP also incrementally predicted random data beyond a composite of items with exceptionally high or low base rates of endorsement from the Comprehensive Infrequency/Frequency Item Repository. We provide recommendations for DRIP cut scores that can detect careless responding while balancing sensitivity and specificity.

Evaluating the Construct Validity of Instructional Manipulation Checks as Measures of Careless Responding to Surveys.

Careless responding measures are important for several purposes, whether it's screening for careless responding or for research centered on careless responding as a substantive variable. One such approach for assessing carelessness in surveys is the use of an instructional manipulation check. Despite its apparent popularity, little is known about the construct validity of instructional manipulation checks as measures of careless responding. Initial results are inconclusive, and no study has thoroughly evaluated the validity of the instructional manipulation check as a measure of careless responding. Across 2 samples (N = 762), we evaluated the construct validity of the instructional manipulation check under a nomological network. We found that the instructional manipulation check converged poorly with other measures of careless responding, weakly predicted participant inability to recognize study content, and did not display incremental validity over existing measures of careless responding. Additional analyses revealed that instructional manipulation checks performed poorly compared to single scores of other alternative careless responding measures and that screening data with alternative measures of careless responding produced greater or similar gains in data quality to instructional manipulation checks. Based on the results of our studies, we do not recommend using instructional manipulation checks to assess or screen for careless responding to surveys.

A comparison of conventional and resampled personal reliability in detecting careless responding.

Detecting careless responding in survey data is important to ensure the credibility of study findings. Of several available detection methods, personal reliability (PR) is one of the best-performing indices. Curran, Journal of Experimental Social Psychology, 66, 4-19, (2016) proposed a resampled version of personal reliability (RPR). Compared to the conventional PR or even-odd consistency, in which just one set of scale halves is used, RPR is based on repeated calculation of PR across several randomly rearranged sets of scale halves. RPR should therefore be less affected than PR by random errors that may occur when a specific set of scale half pairings is used for the PR calculation. In theory, RPR should outperform PR, but it remains unclear whether it in fact does, and under what conditions the potential gain in detection accuracy is the most pronounced. We conducted two studies: a simulation study examined the performance of the conventional PR and RPR in detecting simulated careless responding, and a real data example study analyzed their performance when detecting human-generated careless responding. In both studies, RPR turned out to be a significantly better careless response indicator than PR. The results also revealed that using 25 resamples for the RPR computation is sufficient to obtain the expected gain in detection accuracy over the conventional PR. We therefore recommend using RPR instead of the conventional PR when screening questionnaire data for careless responding.

대학생의 정서지능이 대학생활적응에 미치는 영향: 사회적지지와 진로적응력의 직렬매개효과

Objectives The objective of this research is to identify the mediating effects of college students’ emotional intelligence on how well they adjust to college life, with social support and career adaptability acting as media, by focusing on emotional intelligence which promotes adjustment to college life. Methods To this end, a survey was conducted with 502 male and female college students from four-year universities in South Korea. After excluding careless responding, a final sample of 473 was used for model analysis. The data were analyzed using SPSS 27 and Process Macro Model 6. Results First, it was found that emotional intelligence and career adaptability of college students directly influence their adaptation to college life, whereas social support was not found to have a significant impact on college adaptation. Second, emotional intelligence of college students indirectly affects adaptation to college life through career adaptability, and emotional intelligence was found not to influence college adaptation through social support. Furthermore, the sequential mediating effects of college students’ emotional intelligence on adaptation to college life through social support and career adaptability were confirmed. Conclusions This study empirically validated the research question that emotional intelligence not only directly influences social support, career adaptability, and adaptation to college life significantly but also indirectly affects adaptation to college life through sequential mediation of social support and career adaptability. This underscores the importance of considering the role and contribution of emotional intelligence in adaptation to college life.

Careless responding detection revisited: Accuracy of direct and indirect measures.

To screen for careless responding, researchers have a choice between several direct measures (i.e., bogus items, requiring the respondent to choose a specific answer) and indirect measures (i.e., unobtrusive post hoc indices). Given the dearth of research in the area, we examined how well direct and indirect indices perform relative to each other. In five experimental studies, we investigated whether the detection rates of the measures are affected by contextual factors: severity of the careless response pattern, type of item keying, and type of item presentation. We fully controlled the information environment by experimentally inducing careless response sets under a variety of contextual conditions. In Studies 1 and 2, participants rated the personality of an actor that presented himself in a 5-min-long videotaped speech. In Studies 3, 4, and 5, participants had to rate their own personality across two measurements. With the exception of maximum longstring, intra-individual response variability, and individual contribution to model misfit, all examined indirect indices performed better than chance in most of the examined conditions. Moreover, indirect indices had detection rates as good as and, in many cases, better than the detection rates of direct measures. We therefore encourage researchers to use indirect indices, especially within-person consistency indices, instead of direct measures.

A novel CFA + EFA model to detect aberrant respondents

Abstract Aberrant respondents are common but yet extremely detrimental to the quality of social surveys or questionnaires. Recently, factor mixture models (FMMs) have been employed to identify individuals providing deceptive or careless responses. We propose a comprehensive FMM for continuous outcomes that combines confirmatory and exploratory factor models to classify both the nonaberrant and aberrant respondents. The flexibility of the proposed classification model allows for the identification of two of the most common aberrant response styles, namely faking and careless responding. We validated our approach by means of two simulations and two case studies. The results indicate the effectiveness of the proposed model in dealing with aberrant responses in social and behavioural surveys.

Identifying the Careless Responders: A Cross-Country Comparison on PISA 2018 Dataset

Careless responding (CR) is one of the major concerns in self-report measures since it affects the quality of collected data. In this research, it was aimed to examine the dataset in terms of CR, to investigate the effects of CR on the psychometric properties of data, and to examine the effectiveness of CR detection methods in a cross-cultural context. Specifically; response time, outlier and long-string indices were used to flag CR and efficiency of using these were compared on PISA 2018 reading related attitude scales for Singapore and Turkey. As a result, it was indicated that the amount of CR was higher for Singapore than Turkey at long-string and response time methods. Furthermore, after removal of CR from dataset, enjoyment level of reading, perception of competence and perception of difficulty of PISA tests scores increased. Another result was that long-string and response time analyses provided significant improvement in model fit and long-string had the highest improvement level. Moreover, removing respondents according to outlier analyses didn’t provide any improvement on model-data fit. In general, results illustrated that careless responding behavior may have important effects on psychometric properties of self-report and screening the data for CR is strongly suggested in future studies.

On assessing trait rumination using the Ruminative Response Scale.

This study explores the stability of scores on psychometrically validated trait questionnaires over time. We illustrate potential pitfalls through a larger study that used the Ruminative Response Scale (RRS) to categorize participants prior to study inclusion into two groups based on their habitual tendency to ruminate. Surprisingly, when we re-administered the RRS at the start of an experimental session, significant score changes occurred, resulting in participants shifting between the two groups. To address this, we modified our recruitment process, aiming to reduce careless responses, including an online RRS assessment a week before the lab appointment. We analyzed the different samples prior to and after changing the recruitment procedure, as well as the total sample regarding the psychometric properties of the RRS. We also explored various indices to identify and predict score changes due to careless responding; however, only a subgroup of participants was successfully identified. Our findings suggest that Mahalanobis distances are effective for identifying substantial score changes, with baseline state rumination emerging as a marginally significant predictor. We discuss the importance of conducting manipulation checks and offer practical implications for research involving psychometrically validated trait questionnaires.

Identifying Careless Responding in Web-Based Surveys

Abstract: The increasing use of web-based surveys in social sciences research has brought forth the challenge of effectively identifying and managing inattentive/careless responding. The existing detection methods have shown limited success, highlighting the need for improved methodologies. This study introduces a novel approach that utilizes time-stamped action sequence data of mouse movements and employs deep learning models to detect careless responding. It introduces the concept of Approximate Areas of Interest (AAOIs) along with the application of Gated Recurrent Units (GRUs) and Bidirectional Long Short-Term Memory (BiLSTM) models. This research presents a flexible and efficient tool that can be applied across different scales and survey contexts. The results demonstrate the superior performance of the proposed approach in identifying group membership, achieving up to 95% accuracy when tested on experimental data with induced inattentiveness. The presented approach offers a potentially promising tool for overcoming the pervasive challenge of detecting careless responding in computer-based surveys.

Building a Bigger Toolbox: The Construct Validity of Existing and Proposed Measures of Careless Responding to Cognitive Ability Tests

Employers commonly use cognitive ability tests in the personnel selection process. Although ability tests are excellent predictors of job performance, their validity may be compromised when test takers engage in careless responding. It is thus important for researchers to have access to effective careless responding measures, which allow researchers to screen for careless responding and to evaluate efforts to prevent careless responding. Previous research has primarily used two types of measures to assess careless responding to ability tests—response time and self-reported carelessness. In the current paper, we expand the careless responding assessment toolbox by examining the construct validity of four additional measures: (1) infrequency, (2) instructed-response, (3) long-string, and (4) intra-individual response variability (IRV) indices. Expanding the available set of careless responding indices is important because the strengths of new indices may offset the weaknesses of existing indices and would allow researchers to better assess heterogeneous careless response behaviors. Across three datasets ( N = 1,193), we found strong support for the validity of the response-time and infrequency indices, and moderate support for the validity of the instructed-response and IRV indices.

Careless responding in online studies is associated with alcohol use: A mega-analysis.

The prevalence of research conducted online in the addiction field has increased rapidly over the past decade. However, little focus has been given to careless responding in these online studies, despite the issues it may cause for statistical inference and generalizability. Our aim was to examine whether alcohol use is associated with careless responses. Raw data were requested from online studies examining alcohol use and related problems which also addressed careless responding. We obtained 13 data sets of 12,237 participants (Mage = 42.16, SD = 15.65, 50.5% female). The sample had an average Alcohol Use Disorders Identification Test (AUDIT) score of 10.88 (SD = 7.77). Predictors included demographic information (age, gender) and AUDIT total scores. The primary outcome was whether an individual was classed as a careless responder, for example, by failing an explicit attention check question. AUDIT total scores were associated with careless responding (OR = 1.07, 95% CI [1.06, 1.08], p < .001). Hazardous drinking or worse was associated with 2.21 greater odds (OR = 2.21, 95% CI [1.81, 2.71] of careless responding, whereas harmful drinking or worse was associated with 3.43 greater odds (OR = 3.43, 95% CI [2.83, 4.17]) and probable dependence was associated with 3.63 greater odds (OR = 3.63, 95% CI [2.95, 4.48]). Alcohol use and related problems are positively associated with careless responding in online research. Removal of individuals identified as careless responders may lead to issues of generalizability, and more care should be taken to identify and handle careless responder data. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Open science perspectives on machine learning for the identification of careless responding: A new hope or phantom menace?

AbstractPowerful methods for identifying careless respondents in survey data are not just important to ensure the validity of subsequent data analyses, they are also instrumental for studying the psychological processes that drive humans to respond carelessly. Conversely, a deeper understanding of the phenomenon of careless responding enables the development of improved methods for the identification of careless respondents. While machine learning has gained substantial attention and popularity in many scientific fields, it is largely unexplored for the detection of careless responding. On the one hand, machine learning algorithms can be highly powerful tools due to their flexibility. On the other hand, science based on machine learning has been criticized in the literature for a lack of reproducibility. We assess the potential and the pitfalls of machine learning approaches for identifying careless respondents from an open science perspective. In particular, we discuss possible sources of reproducibility issues when applying machine learning in the context of careless responding, and we give practical guidelines on how to avoid them. Furthermore, we illustrate the high potential of an unsupervised machine learning method for the identification of careless respondents in a proof‐of‐concept simulation experiment. Finally, we stress the necessity of building an open data repository with labeled benchmark data sets, which would enable the evaluation of methods in a more realistic setting and make it possible to train supervised learning methods. Without such a data repository, the true potential of machine learning for the identification of careless responding may fail to be unlocked.

The effectiveness of warning statements in reducing careless responding in crowdsourced online surveys

Carelessness or insufficient effort responding is a widespread problem in online research, with estimates ranging from 3% to almost 50% of participants in online surveys being inattentive. While detecting carelessness has been subject to multiple studies, the factors that reduce or prevent carelessness are not as well understood. Initial evidence suggests that warning statements prior to study participation may reduce carelessness, but there is a lack of conclusive high-powered studies. This preregistered randomized controlled experiment aimed to test the effectiveness of a warning statement and an improved implementation of a warning statement in reducing participant inattention. A study with 812 participants recruited on Amazon Mechanical Turk was conducted. Results suggest that presenting a warning statement is not effective in reducing carelessness. However, requiring participants to actively type the warning statement statistically significantly reduced carelessness as measured with self-reported diligence, even-odd consistency, psychometric synonyms and antonyms, and individual response variability. The active warning statements also led to statistically significantly more attrition and potentially deterred those who were likely to be careless from even participating in this study. We show that the current standard practice of implementing warning statements is ineffective and novel methods to prevent and deter carelessness are needed.

Detecting Careless Responding in Multidimensional Forced-Choice Questionnaires.

The multidimensional forced-choice (MFC) format is an alternative to rating scales in which participants rank items according to how well the items describe them. Currently, little is known about how to detect careless responding in MFC data. The aim of this study was to adapt a number of indices used for rating scales to the MFC format and additionally develop several new indices that are unique to the MFC format. We applied these indices to a data set from an online survey (N = 1,169) that included a series of personality questionnaires in the MFC format. The correlations among the careless responding indices were somewhat lower than those published for rating scales. Results from a latent profile analysis suggested that the majority of the sample (about 76-84%) did not respond carelessly, although the ones who did were characterized by different levels of careless responding. In a simulation study, we simulated different careless responding patterns and varied the overall proportion of carelessness in the samples. With one exception, the indices worked as intended conceptually. Taken together, the results suggest that careless responding also plays an important role in the MFC format. Recommendations on how it can be addressed are discussed.

Careless Responding: Why Many Findings Are Spurious or Spuriously Inflated

Contrary to long-standing conventional wisdom, failing to exclude data from carelessly responding participants on questionnaires or behavioral tasks will frequently result in false-positive or spuriously inflated findings. Despite prior publications demonstrating this disturbing statistical confound, it continues to be widely underappreciated by most psychologists, including highly experienced journal editors. In this article, we aim to comprehensively explain and demonstrate the severity and widespread prevalence of careless responding’s (CR) inflationary effects in psychological research. We first describe when and why one can expect to observe the inflationary effect of unremoved CR data in a manner accessible to early graduate or advanced undergraduate students. To this end, we provide an online simulator tool and instructional videos for use in classrooms. We then illustrate realistic magnitudes of the severity of unremoved CR data by presenting novel reanalyses of data sets from three high-profile articles: We found that many of their published effects would have been meaningfully, sometimes dramatically, inflated if they had not rigorously screened out CR data. To demonstrate the frequency with which researchers fail to adequately screen for CR, we then conduct a systematic review of CR screening procedures in studies using paid online samples (e.g., MTurk) published across two prominent psychological-science journals. These findings suggest that most researchers either did not conduct any kind of CR screening or conducted only bare minimal screening. To help researchers avoid publishing spuriously inflated findings, we summarize best practices to help mitigate the threats of CR data.

Derivation of individualised item interrelatedness indices of careless responding

Orientation: Presently, carelessly responding (CR) individuals are omitted in terms of several individual indices, including consistency-type indices (that compare performance on only a limited number of matched item pairs), and subsequently, the effectiveness of such screening is evaluated in terms of, among others, the group mean item interrelatedness (IIR) (based on all J[J – 1] item pairs).Research purpose: This research aims to develop individualised versions of the group IIR measures to render them applicable during the screening phase as substitutes for the presently used consistency-type indices.Motivation for the study: Such individual consistency indices may be used together with other CR indices to jointly determine the eventual evaluation results.Research approach/design and method: To develop the intended CR indices mathematical-statistical principles were applied on the product moment correlation and coefficient alpha formulae.Main findings: Three individual IIR indices have been developed which show individual respondents’ respective contributions to the mean item inter-correlation and to coefficient alpha, as measures of group mean IIR.Practical/managerial implications: These indices may be used during screening in lieu of the existing restrictive consistency indices.Contribution/value-add: Carelessly responding respondents who previously may have survived screening because of less-inclusive consistency-type IIR indices and consequently may have negatively affected the eventual evaluation results, are now screened out.

Understanding professional development challenges of Chinese public health professionals: association and prediction analyses with data validity screening.

Little is known about the public health professionals engaged in educating and training new or future researchers in public health. Research in this direction identifies their issues, concerns, challenges, and needs. This study focused on the professional development challenges of Chinese public health professionals. Snowball sampling was utilized. A total of 265 public health professionals participated. An instrument of 6 dimensions (burnout, sleep issue, mood issue, friends' support, exercise, and challenges) was developed, revised, and administered online. Two different approaches, the conventional and data screening approaches, were applied. The former started with item quality analyses, whereas the latter began with data quality checks. The chi-square tests of associations and logistic regressions were performed on both approaches. 19.25% of the participants were detected and deleted as careless respondents. Using both approaches, six professional development challenges except one ("Multidisciplinary learning") were significantly associated with various demographic features. The two approaches produced different models though they converged sometimes. The latent variables of exercise predicted professional development challenges more frequently than other latent variables. Regarding correct classification rates, results from the data screening approach were comparable to those from the conventional approach. The latent variables of exercise, such as "Exercise effects," "Expectations of exercise," and "Belief in exercise," might be understudied. More research is necessary for professional development challenges using exercise as a multidimensional construct. Based on the current study, screening and deleting careless responses in survey research is necessary.