Careless Responses Research Articles

A comparison of conventional and resampled personal reliability in detecting careless responding.

Detecting careless responding in survey data is important to ensure the credibility of study findings. Of several available detection methods, personal reliability (PR) is one of the best-performing indices. Curran, Journal of Experimental Social Psychology, 66, 4-19, (2016) proposed a resampled version of personal reliability (RPR). Compared to the conventional PR or even-odd consistency, in which just one set of scale halves is used, RPR is based on repeated calculation of PR across several randomly rearranged sets of scale halves. RPR should therefore be less affected than PR by random errors that may occur when a specific set of scale half pairings is used for the PR calculation. In theory, RPR should outperform PR, but it remains unclear whether it in fact does, and under what conditions the potential gain in detection accuracy is the most pronounced. We conducted two studies: a simulation study examined the performance of the conventional PR and RPR in detecting simulated careless responding, and a real data example study analyzed their performance when detecting human-generated careless responding. In both studies, RPR turned out to be a significantly better careless response indicator than PR. The results also revealed that using 25 resamples for the RPR computation is sufficient to obtain the expected gain in detection accuracy over the conventional PR. We therefore recommend using RPR instead of the conventional PR when screening questionnaire data for careless responding.

Careless responding detection revisited: Accuracy of direct and indirect measures.

To screen for careless responding, researchers have a choice between several direct measures (i.e., bogus items, requiring the respondent to choose a specific answer) and indirect measures (i.e., unobtrusive post hoc indices). Given the dearth of research in the area, we examined how well direct and indirect indices perform relative to each other. In five experimental studies, we investigated whether the detection rates of the measures are affected by contextual factors: severity of the careless response pattern, type of item keying, and type of item presentation. We fully controlled the information environment by experimentally inducing careless response sets under a variety of contextual conditions. In Studies 1 and 2, participants rated the personality of an actor that presented himself in a 5-min-long videotaped speech. In Studies 3, 4, and 5, participants had to rate their own personality across two measurements. With the exception of maximum longstring, intra-individual response variability, and individual contribution to model misfit, all examined indirect indices performed better than chance in most of the examined conditions. Moreover, indirect indices had detection rates as good as and, in many cases, better than the detection rates of direct measures. We therefore encourage researchers to use indirect indices, especially within-person consistency indices, instead of direct measures.

A novel CFA + EFA model to detect aberrant respondents

Abstract Aberrant respondents are common but yet extremely detrimental to the quality of social surveys or questionnaires. Recently, factor mixture models (FMMs) have been employed to identify individuals providing deceptive or careless responses. We propose a comprehensive FMM for continuous outcomes that combines confirmatory and exploratory factor models to classify both the nonaberrant and aberrant respondents. The flexibility of the proposed classification model allows for the identification of two of the most common aberrant response styles, namely faking and careless responding. We validated our approach by means of two simulations and two case studies. The results indicate the effectiveness of the proposed model in dealing with aberrant responses in social and behavioural surveys.

Implementing Ecological Momentary Assessment in Audiological Research: Opportunities and Challenges.

Ecological momentary assessment (EMA) is a way to evaluate experiences in everyday life. It is a powerful research tool but can be complex and challenging for beginners. Application of EMA in audiological research brings with it opportunities and challenges that differ from other research disciplines. This tutorial discusses important considerations when conducting EMA studies in hearing care. While more research is needed to develop specific guidelines for the various potential applications of EMA in hearing research, we hope this article can alert hearing researchers new to EMA to pitfalls when using EMA and help strengthen their study design. The current article elaborates study design details, such as choice of participants, representativeness of the study period for participants' lives, and balancing participant burden with data requirements. Mobile devices and sensors to collect objective data on the acoustic situation are reviewed alongside different possibilities for EMA setups ranging from online questionnaires paired with a timer to proprietary apps that also have access to parameters of a hearing device. In addition to considerations for survey design, a list of questionnaire items from previous studies is provided. For each item, an example and a list of references are given. EMA typically provides data sets that are rich but also challenging in that they are noisy, and there is often unequal amount of data between participants. After recommendations on how to check the data for compliance, reactivity, and careless responses, methods for statistical analysis on the individual level and on the group level are discussed including special methods for direct comparison of hearing device programs.

On assessing trait rumination using the Ruminative Response Scale.

This study explores the stability of scores on psychometrically validated trait questionnaires over time. We illustrate potential pitfalls through a larger study that used the Ruminative Response Scale (RRS) to categorize participants prior to study inclusion into two groups based on their habitual tendency to ruminate. Surprisingly, when we re-administered the RRS at the start of an experimental session, significant score changes occurred, resulting in participants shifting between the two groups. To address this, we modified our recruitment process, aiming to reduce careless responses, including an online RRS assessment a week before the lab appointment. We analyzed the different samples prior to and after changing the recruitment procedure, as well as the total sample regarding the psychometric properties of the RRS. We also explored various indices to identify and predict score changes due to careless responding; however, only a subgroup of participants was successfully identified. Our findings suggest that Mahalanobis distances are effective for identifying substantial score changes, with baseline state rumination emerging as a marginally significant predictor. We discuss the importance of conducting manipulation checks and offer practical implications for research involving psychometrically validated trait questionnaires.

Quantifying Careless Responses in Student Evaluation of Teaching and Justifying Removal for Data Validity

Surveys are typical for student evaluation of teaching (SET). Survey research consistently confirms the negative impacts of careless responses on research validity, including low data quality and invalid research inferences. SET literature seldom addresses if careless responses are present and how to improve. To improve evaluation practices and validity, the current study proposed a three-step procedure to screen SET data for quantifying careless responses, delete and justify the removal of careless responses, and assess if removing careless responses improved the internal structure of the SET data. For these purposes, a convenience sample was taken from a Chinese university. A web-based survey was administered using a revised version of the Students’ Evaluation of Education Quality. One hundred ninety-nine students evaluated 11 courses with 295 responses. Longstring and Rasch outlier analyses identified 49% of nonrandom and random careless responses. The careless responses impacted evaluation results substantively and were deleted. The subsequent study demonstrated that removal improved data validity, using reliability, separation, and inter-rater agreement from the multi-facet Rasch model and G- and D-coefficients, signal-noise ratios, and error variance from generalizability theory. Removing careless responses improved the data validity in terms of true score variance and discrimination power of the data. Data screening should be a prerequisite to validating SET data based on the research results. Data removal is necessary to improve the research validity only if there is a noticeable change in the estimated teaching abilities. Suggestions and implications were discussed, including developing sound evaluation practices and formative use of SET.

Abusive supervision in public service organisations: investigating the moderating effect of attribution styles

ABSTRACT This study explores how abusive supervision impacts employee turnover intention and performance and examines how attribution styles moderate the relationship in public service organisations. Data were collected from five types of public service organisations in Indonesia (i.e., municipal offices, hospitals, police offices, social security offices, and schools) using a multi-wave longitudinal survey method with a five-workday interval. After removing incomplete responses and participants with careless responses, 369 participants were included in the analysis. The data were analysed using Hayes’ moderated-mediation regression technique. This study confirmed that abusive supervision indirectly reduced performance by increasing turnover intention. However, three attribution styles (i.e., self-, supervisor-, and organisation-directed) showed different moderating effects on the relationship. Self- and organisation-directed attribution could intensify the positive impact of abusive supervision on turnover intention. The negative impact of abusive supervision on performance was observed when supervisor-directed attribution was low.

The prevalence of careless response behaviour and its consequences on data quality in self-report questionnaires on student learning

IntroductionSelf-report questionnaires are widely used in high schools and universities to gain insights into students’ learning strategies and enhance the quality of their education. However, it is important to acknowledge the possibility of respondents being inattentive when completing these questionnaires. While reliability analyses are typically performed at the group level, when providing individual feedback, it is crucial that each respondent’s results are reliable. This study aimed to evaluate the prevalence of careless response behaviour in a questionnaire concerning student learning.MethodsData analysis encompassed a substantial sample of 12,000+ students in their final two years of secondary education, averaging around 17 years of age. Employing five complementary detection techniques, the study identified instances of careless responding present in the questionnaire data.ResultsOur results underscore a notable prevalence of careless response behaviour among the surveyed students. Application of the five detection techniques revealed a substantial number of instances indicating inattentive responding. Furthermore, the questionnaire’s measurement scales were evaluated for reliability. The study noted the presence of carelessness but observed minimal impact on group-level results.DiscussionThe outcomes of this study hold important implications for using self-report questionnaires in education. The prevalence of careless responding emphasizes the need for scrutinizing individual responses. Despite careless responses, their influence on overall group-level data integrity seems restricted. Nonetheless, the study underscores the importance of cautiously interpreting individual-level outcomes, particularly when using these results for individual feedback.

Detecting Careless Cases in Practice Tests

In this paper, we present a novel method for detecting careless responses in a low-stakes practice exam using machine learning models. Rather than classifying test-taker responses as careless based on model fit statistics or knowledge of truth, we built a model to predict significant changes in test scores between a practice test and an official test based on attributes of practice test items. We extracted features from practice test items using hypotheses about how careless test takers respond to items and cross-validated model performance to optimize out-of-sample predictions and reduce heteroscedasticity when predicting the closest official test. All analyses use data from the practice and official versions of the Duolingo English Test. We discuss the implications of using a machine learning model for predicting careless cases as compared with alternative, popular methods.

Understanding professional development challenges of Chinese public health professionals: association and prediction analyses with data validity screening.

Little is known about the public health professionals engaged in educating and training new or future researchers in public health. Research in this direction identifies their issues, concerns, challenges, and needs. This study focused on the professional development challenges of Chinese public health professionals. Snowball sampling was utilized. A total of 265 public health professionals participated. An instrument of 6 dimensions (burnout, sleep issue, mood issue, friends' support, exercise, and challenges) was developed, revised, and administered online. Two different approaches, the conventional and data screening approaches, were applied. The former started with item quality analyses, whereas the latter began with data quality checks. The chi-square tests of associations and logistic regressions were performed on both approaches. 19.25% of the participants were detected and deleted as careless respondents. Using both approaches, six professional development challenges except one ("Multidisciplinary learning") were significantly associated with various demographic features. The two approaches produced different models though they converged sometimes. The latent variables of exercise predicted professional development challenges more frequently than other latent variables. Regarding correct classification rates, results from the data screening approach were comparable to those from the conventional approach. The latent variables of exercise, such as "Exercise effects," "Expectations of exercise," and "Belief in exercise," might be understudied. More research is necessary for professional development challenges using exercise as a multidimensional construct. Based on the current study, screening and deleting careless responses in survey research is necessary.

A Constrained Factor Mixture Model for Detecting Careless Responses that is Simple to Implement

Using constrained factor mixture models (FMM) for careless response identification is still in its infancy. Existing models have overly restrictive statistical assumptions that do not identify all types of careless respondents. The current paper presents a novel constrained FMM model with more reasonable assumptions that capture both longstring and random careless respondents. We provide a comprehensive comparison of the statistical assumptions between the proposed model and two previous constrained models. The proposed model was evaluated using both real data ( N = 1,455) and statistical simulation. The results showed that the model had a superior fit, stronger convergent validity with other indicators of careless responding, more accurate parameter recovery and more accurate identification of careless respondents when compared to its predecessors. The proposed model does not require additional data collection effort, and thus researchers can routinely use it to control careless responses. We provide user-friendly syntax with detailed explanations online to facilitate its use.

Comparing Person-Fit and Traditional Indices Across Careless Response Patterns in Surveys.

Methods to identify carelessness in survey research can be valuable tools in reducing bias during survey development, validation, and use. Because carelessness may take multiple forms, researchers typically use multiple indices when identifying carelessness. In the current study, we extend the literature on careless response identification by examining the usefulness of three item-response theory-based person-fit indices for both random and overconsistent careless response identification: infit MSE outfit MSE, and the polytomous lz statistic. We compared these statistics with traditional careless response indices using both empirical data and simulated data. The empirical data included 2,049 high school student surveys of teaching effectiveness from the Network for Educator Effectiveness. In the simulated data, we manipulated type of carelessness (random response or overconsistency) and percent of carelessness present (0%, 5%, 10%, 20%). Results suggest that infit and outfit MSE and the lz statistic may provide complementary information to traditional indices such as LongString, Mahalanobis Distance, Validity Items, and Completion Time. Receiver operating characteristic curves suggested that the person-fit indices showed good sensitivity and specificity for classifying both over-consistent and under-consistent careless patterns, thus functioning in a bidirectional manner. Carelessness classifications based on low fit values correlated with carelessness classifications from LongString and completion time, and classifications based on high fit values correlated with classifications from Mahalanobis Distance. We consider implications for research and practice.

Does employees’ gender matter? Investigating the indirect effect of abusive supervision on employee creativity through job insecurity in Indonesia

PurposeThis study aims to investigate the effect of abusive supervision on employees’ creativity through the mediating role of job insecurity and the moderating role of subordinate gender in Indonesia.Design/methodology/approachData were collected from various sources using online recruitment methods. The abusive supervision scale, job insecurity scale and employee creativity scale were the three measures in this study. Participants completed a three-wave data collection procedure using an online survey platform. After removing participants with incomplete and careless responses, the final data set contained 515 usable responses.FindingsThe results suggested that the negative effect of abusive supervision on employees’ creativity was mediated by job insecurity, and employees’ gender moderated this adverse effect. Gender roles shaped how employees respond to their leader’s hostile behaviours. In Indonesia, abusive supervision increased employees’ job insecurity and consequently reduced their creativity. However, the damage was more profound for the male employees than the female employees.Practical implicationsGender role theory and perspectives are essential in explaining leader–employee interactions and must be included in leadership strategies. Also, support and resources must be provided equally for both male and female employees. However, more attention must be given to male employees to ensure their security working in organisations. Finally, some interventions are necessary to mitigate the effect of abusive supervision.Originality/valueThis study offers insight into how male and female employees, in a traditional gender role culture like Indonesia, respond to their leader’s abusive behaviours.

Missing data and careless responses: recommendations for instructional communication

Missing data and careless responses: recommendations for instructional communication

The relationship between depression, anxiety, e-health literacy, and health-promoting behavior in nursing students during COVID-19.

The purpose of this study was to identify the correlation between depression, anxiety, e-health literacy (eHL), and health-promoting behaviors among nursing students during the coronavirus disease 2019 (COVID-19) pandemic, and to identify the factors affecting health-promoting behaviors. A cross-sectional study was conducted online, recruiting 301 nursing students recruited from 4 universities in the Chungcheng Province and Daejeon Metropolitan City in South Korea between May 28 and June 30, 2021. Data were analyzed using SPSS WIN 27.0. The general characteristics of the study participants were analyzed by frequency and percentage, and the degree of depression, anxiety, eHL, and health-promoting behaviors were calculated as averages and standard deviations. Differences in health-promoting behaviors according to general characteristics were analyzed using independent t tests and analysis of variance, and a post hoc Scheffe test was conducted. Correlations between depression, anxiety, eHL, and health-promoting behaviors were measured using Pearson correlation matrices. Stepwise multiple regression was performed to identify factors affecting health-promoting behaviors. During the COVID-19 pandemic, the average level of depression and anxiety was reported to be 3.93 (4.71) and 3.40 (4.30), respectively; 33.2% of nursing students experienced more than mild depression and 29.2% experienced anxiety. The average eHL level was 3.91 (0.56), and the average of health-promoting behaviors was 2.43 (0.45). Depression was positively correlated with anxiety (R = 0.734, P < .001) and negatively correlated with health-promoting behaviors (r = -0.198, P = .001), whereas eHL had a positive correlation with health-promoting behaviors (R = 0.347, P < .001). The factors affecting health-promoting behaviors were religion (β = -0.160, standard error [SE] = 0.048), current health status (β = -0.097, SE = 0.032), frequency of searching the Internet for health-related information in a week (β = -0.070, SE = 0.026), interest in health (β = -0.191, SE = 0.039), and critical eHL (β = 0.243, SE = 0.040); the explanatory power was 27.4%. Results demonstrate that during the COVID-19 pandemic, higher depression and anxiety among nursing students decreased health-promoting behaviors, while higher eHL increased health-promoting behaviors, and eHL was a major factor affecting health-promoting behaviors. These results contribute to the provision of basic data for the development of nursing intervention programs and educational strategies that can establish correct health-promoting behaviors by managing depression and anxiety among nursing students and improving eHL.

An Exponentially Weighted Moving Average Procedure for Detecting Back Random Responding Behavior

AbstractBack random responding (BRR) behavior is one of the commonly observed careless response behaviors. Accurately detecting BRR behavior can improve test validities. Yu and Cheng (2019) showed that the change point analysis (CPA) procedure based on weighted residual (CPA‐WR) performed well in detecting BRR. Compared with the CPA procedure, the exponentially weighted moving average (EWMA) obtains more detailed information. This study equipped the weighted residual statistic with EWMA, and proposed the EWMA‐WR method to detect BRR. To make the critical values adaptive to the ability levels, this study proposed the Monte Carlo simulation with ability stratification (MC‐stratification) method for calculating critical values. Compared to the original Monte Carlo simulation (MC) method, the newly proposed MC‐stratification method generated a larger number of satisfactory results. The performances of CPA‐WR and EWMA‐WR were evaluated under different conditions that varied in the test lengths, abnormal proportions, critical values and smoothing constants used in the EWMA‐WR method. The results showed that EWMA‐WR was more powerful than CPA‐WR in detecting BRR. Moreover, an empirical study was conducted to illustrate the utility of EWMA‐WR for detecting BRR.

Validity and reliability of the Chinese version of the Young Positive Schema Questionnaire.

The Young Positive Schema Questionnaire (YPSQ) examines early adaptive or positive schemas as a counterpart to early maladaptive ones. This study investigated the validity and reliability of the Chinese version of the YPSQ (CYPSQ). A convenient community sample of 634 individuals, most of whom were college students, were recruited through a mobile survey. R Careless was used to screen for careless responses. A final sample of 336 was obtained. Confirmatory factor analyses (CFA) of the CYPSQ were conducted using SPSS AMOS 25. Other statistical analyses were conducted using SPSS 25. An 11-factor CYPSQ was identified with an acceptable factor structure (χ2/df = 2.13, SRMR = 0.04, RMSEA = 0.06, GFI = 0.80, TLI = 0.90, and CFI = 0.91). Convergent and discriminant validity were confirmed in most aspects. Concurrent validity was evident with resilience, life satisfaction, depression, and anxiety. Internal reliability was satisfied as the Cronbach's alphas of the 11 factors of the CYPSQ ranged between 0.70 and 0.88. The findings supported the reliability and validity of the CYPSQ in Mainland China.

Careless Response Processes are Heterogeneous: Comment on Goldammer et al. (2020)

Goldammer et al. (2020) examined the performance of careless response detection indices by experimentally manipulating survey instructions to induce careless responding, then compared the ability of various indices to detect these induced careless responses. Based on these analyses, Goldammer et al. concluded that metrics designed to detect overly consistent response patters (i.e. longstring and IRV) were ineffective. In this comment, we critique this conclusion by highlighting critical problems with the experimental manipulation used. Specifically, Goldammer et al.’s manipulations only encouraged overly inconsistent, or random, responding and thus did not induce the full range of behavior that is present in natural careless responding. As such, it is unsurprising that metrics designed to detect overly consistent responding appeared to be ineffective. Because the full range of careless behavior was not induced, Goldammer et al.’s study cannot address the utility of longstring or other consistency metrics. We offer recommendations for alternative experimental manipulations that may produce more naturalistic and diverse careless responding.

Dealing with Careless Responding in Survey Data: Prevention, Identification, and Recommended Best Practices.

Surveys administered online have several benefits, but they are particularly prone to careless responding, which occurs when respondents fail to read item content or give sufficient attention, resulting in raw data that may not accurately reflect respondents' true levels of the constructs being measured. Careless responding can lead to various psychometric issues, potentially impacting any area of psychology that uses self-reported surveys and assessments. This review synthesizes the careless responding literature to provide a comprehensive understanding of careless responding and ways to prevent, identify, report, and clean careless responding from data sets. Further, we include recommendations for different levels of screening for careless responses. Finally, we highlight some of the most promising areas for future work on careless responding.

Would I Lie to You? Using Markov Chains to Detect Careless Responding in Survey Data

Although careless response behaviors by survey participants bear considerable threats to the quality of the data, scholars often do not sufficiently address this issue in their analyses. This might be partially because existing approaches to detect careless responding are specialized to detect a certain type of careless response patterns, but fail to detect others. We develop a post-hoc measure, the Lazy Respondents (Laz.R) index, that is based on first-order Markov chains and the idea that careless respondents’ last response can be used to predict their next response. Using large, publicly available datasets to compare our Laz.R index to the most common precautionary and post-hoc measures, we find that (1) the Laz.R index correctly identifies patterns of careless respondents that some or all of the other indices overlook and (2) the Laz.R index avoids false positives (i.e., patterns that are flagged by single other indices, but which seem to capture truthful answering behavior). Furthermore, we suggest an approach to find a sample-specific cutoff value for careless respondents via kneedle algorithm. Finally, we introduce an R Shiny application that provides scholars a simple and readily available means to implement the Laz.R index.