Careless Responding May Yield Spurious Factors in Unidimensionality Assessment: A Simulation Study

Because careless responding to questionnaire items can be quite common, and even low rates of careless responding can substantially impact some analyzes, researchers have been advised to remove careless respondents before data analysis. However, identifying these respondents is a non-trivial task. In the context of multi-item, unidimensional scales, it has been suggested that response variance may hold information about careless responding. This notion was tested in the current research with two studies. Data collected from respondents indicates that people responding carelessly display more variance in their responses than people responding with more effort. Given this finding, a procedure which uses measures of response variance and cluster analysis to identify careless respondents was developed. The effectiveness of the procedure with different specifications was tested with simulated data and validated with data from actual respondents. On the basis of the results, we advocate using the procedure with specifications that conservatively identify careless respondents. Such an approach will identify the most extreme careless respondents, while maximizing the retention of careful, honest respondents. We discuss the advantages the developed procedure has over existing procedures for identifying careless respondents.

The purpose of this study was to investigate the application of the parallel analysis (PA) method for choosing the number of factors in component analysis for situations in which data are dichotomous or ordinal. Although polychoric correlations are sometimes used as input for component analyses, the random data matrices generated for use in PA typically consist of Pearson correlations. In this study, the authors matched the type of random data matrix to the type of input matrix. Analyses were conducted on both polychoric and Pearson correlation matrices, and random matrices of the same type (polychoric or Pearson) were generated for the PA procedure. PA based on random Pearson correlations was found to perform at least as well as PA based on random polychoric correlations, for nearly all of the conditions studied.

Polychoric correlation is often an important building block in the analysis of rating data, particularly for structural equation models. However, the commonly employed maximum likelihood (ML) estimator is highly susceptible to misspecification of the polychoric correlation model, for instance, through violations of latent normality assumptions. We propose a novel estimator that is designed to be robust against partial misspecification of the polychoric model, that is, when the model is misspecified for an unknown fraction of observations, such as careless respondents. To this end, the estimator minimizes a robust loss function based on the divergence between observed frequencies and theoretical frequencies implied by the polychoric model. In contrast to existing literature, our estimator makes no assumption on the type or degree of model misspecification. It furthermore generalizes ML estimation, is consistent as well as asymptotically normally distributed, and comes at no additional computational cost. We demonstrate the robustness and practical usefulness of our estimator in simulation studies and an empirical application on a Big Five administration. In the latter, the polychoric correlation estimates of our estimator and ML differ substantially, which, after further inspection, is likely due to the presence of careless respondents that the estimator helps identify.

A method of item factor analysis based on Thur stone's multiple-factor model and implemented by marginal maximum likelihood estimation and the EM algorithm is described. Statistical significance of suc cessive factors added to the model is tested by the likelihood ratio criterion. Provisions for effects of guessing on multiple-choice items, and for omitted and not-reached items, are included. Bayes constraints on the factor loadings are found to be necessary to suppress Heywood cases. Numerous applications to simulated and real data are presented to substantiate the accuracy and practical utility of the method. Index terms: Armed Services Vocational Aptitude Bat tery, Beta prior, EM algorithm, Item factor analysis, TESTFACT, Tetrachoric correlation.

The analysis of polychoric correlations via principal component analysis and exploratory factor analysis are well-known approaches to determine the dimensionality of ordered categorical items. However, the application of these approaches has been considered as critical due to the possible indefiniteness of the polychoric correlation matrix. A possible solution to this problem is the application of smoothing algorithms. This study compared the effects of three smoothing algorithms, based on the Frobenius norm, the adaption of the eigenvalues and eigenvectors, and on minimum-trace factor analysis, on the accuracy of various variations of parallel analysis by the means of a simulation study. We simulated different datasets which varied with respect to the size of the respondent sample, the size of the item set, the underlying factor model, the skewness of the response distributions and the number of response categories in each item. We found that a parallel analysis and principal component analysis of smoothed polychoric and Pearson correlations led to the most accurate results in detecting the number of major factors in simulated datasets when compared to the other methods we investigated. Of the methods used for smoothing polychoric correlation matrices, we recommend the algorithm based on minimum trace factor analysis.

Previous research evaluating the performance of Horn's parallel analysis (PA) factor retention method with ordinal variables has produced unexpected findings. Specifically, PA with Pearson correlations has performed as well as or better than PA with the more theoretically appropriate polychoric correlations. Seeking to clarify these findings, the current study employed a more comprehensive simulation study that included the systematic manipulation of 7 factors related to the data (sample size, factor loading, number of variables per factor, number of factors, factor correlation, number of response categories, and skewness) as well as 3 factors related to the PA method (type of correlation matrix, extraction method, and eigenvalue percentile). The results from the simulation study show that PA with either Pearson or polychoric correlations is particularly sensitive to the sample size, factor loadings, number of variables per factor, and factor correlations. However, whereas PA with polychorics is relatively robust to the skewness of the ordinal variables, PA with Pearson correlations frequently retains difficulty factors and is generally inaccurate with large levels of skewness. In light of these findings, we recommend the use of PA with polychoric correlations for the dimensionality assessment of ordinal-level data.

Detecting careless respondents in web-based questionnaires: Which method to use?

We provide a basic review of the data screening and assumption testing issues relevant to exploratory and confirmatory factor analysis along with practical advice for conducting analyses that are sensitive to these concerns. Historically, factor analysis was developed for explaining the relationships among many continuous test scores, which led to the expression of the common factor model as a multivariate linear regression model with observed, continuous variables serving as dependent variables, and unobserved factors as the independent, explanatory variables. Thus, we begin our paper with a review of the assumptions for the common factor model and data screening issues as they pertain to the factor analysis of continuous observed variables. In particular, we describe how principles from regression diagnostics also apply to factor analysis. Next, because modern applications of factor analysis frequently involve the analysis of the individual items from a single test or questionnaire, an important focus of this paper is the factor analysis of items. Although the traditional linear factor model is well-suited to the analysis of continuously distributed variables, commonly used item types, including Likert-type items, almost always produce dichotomous or ordered categorical variables. We describe how relationships among such items are often not well described by product-moment correlations, which has clear ramifications for the traditional linear factor analysis. An alternative, non-linear factor analysis using polychoric correlations has become more readily available to applied researchers and thus more popular. Consequently, we also review the assumptions and data-screening issues involved in this method. Throughout the paper, we demonstrate these procedures using an historic data set of nine cognitive ability variables.

BackgroundDisaster management, as defined by the United Nations Office for Disaster Risk Reduction (UNDRR) involves foresighted planning to prevent, prepare for, respond to, and recover from disasters. Research proves that earthquake knowledge significantly contributes to preparedness behavior. The aim of this research is to develop a psychometrically valid questionnaire following UNDRR guidelines to assess earthquake awareness.MethodAn exploratory sequential mixed-methods study was conducted between April and July 2024 in Tabriz, Iran. In the initial phase of the study, a comprehensive literature review and qualitative research were conducted to develop a preliminary item pool related to earthquake knowledge. Subsequently, the face validity, content validity, and construct validity of the items were assessed, followed by an evaluation of reliability through internal consistency, McDonald’s omega and test-retest methods. Exploratory Factor Analysis (EFA) using polychoric correlations and parallel analysis was conducted to determine factor structure. A polychoric correlation matrix was estimated from the sample of 350 respondents with 1000 iterations and using the principal factors method.ResultsA polychoric correlation matrix was computed in R software (version 4.4.1) to estimate the non-linear relations between 14 ordinal items of the earthquake knowledge scale, of a sample of 350 participants. Parallel analysis using principal axis factoring determined three factors with adjusted eigenvalues greater than zero (observed eigenvalues: 7.5, 1.8, and 1.2 for the first, second, and third factor, respectively), which were retained as significant. The 14-item earthquake knowledge questionnaire (14-EKQ) was organized into three factors: Geological Knowledge, Mitigation Measures, and Preparedness Knowledge, reflecting various dimensions of earthquake awareness. EFA revealed that these three factors collectively accounted for 83.6% of the total variance. The RMSEA value of (RMSE = 0.070) falls within the acceptable range (≤ 0.08), indicating a reasonable fit. The CFI (CFI = 0.916) value is close to the threshold of 0.95, indicating a relatively good fit. The TLI value (TLI = 0.908) is slightly below the threshold of 0.95 but still suggests an acceptable fit. The internal consistency and internal correlation coefficient of EKQ indicated acceptable reliability.ConclusionThis study successfully developed and validated a 14-item EKQ. The scale was organized into three distinct factors: Geological knowledge, Mitigation measures, and preparedness knowledge, which collectively accounted for 83.6% of the total variance, demonstrating strong explanatory power. The use of polychoric correlation matrices, parallel analysis, and principal axis factoring (PAF) improved the factor extraction process by appropriately accounting for the ordinal nature of the questionnaire data. Model fit indices, including RMSEA and TLI, indicated an acceptable to good fit of the scale to the data. Additionally, the scale demonstrated acceptable reliability, as evidenced by internal consistency measures, McDonald’s omega, and test-retest reliability. The study’s EKQ makes a significant contribution to earthquake education by providing a validated tool to assess public awareness across geological knowledge, mitigation strategies, and preparedness knowledge, aligning with UNDRR guidelines. Further research is recommended to confirm its generalizability across diverse populations and contexts.

Parallel analysis (PA) is regarded as one of the most accurate methods to determine the number of factors underlying a set of variables. Commonly, PA is performed on the basis of the variables' product-moment correlation matrix. To improve dimensionality assessments for dichotomous or ordered categorical variables, it has been proposed to replace product-moment correlations with more appropriate coefficients, such as tetrachoric or polychoric correlations. While similar modifications have proven useful for various factor analytic approaches, PA results were not consistently improved. The present article outlines a main reason for this result. Specifically, it explains the dependency of PA results on differing proportions of category probabilities when using tetrachoric or polychoric correlations and shows how to adjust for it by generating appropriate reference eigenvalues. The accuracy of dimensionality assessments of PA accounting for category probability proportions versus not accounting for them is investigated using simulation studies. The results show that the category probability adjusted approach distinctly improves dimensionality assessments. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Item factor analysis (IFA) refers to the factor models and statistical inference procedures for analyzing multivariate categorical data. IFA techniques are commonly used in social and behavioral sciences for analyzing item-level response data. Such models summarize and interpret the dependence structure among a set of categorical variables by a small number of latent factors. In this chapter, we review the IFA modeling technique and commonly used IFA models. Then we discuss estimation methods for IFA models and their computation, with a focus on the situation where the sample size, the number of items, and the number of factors are all large. Existing statistical softwares for IFA are surveyed. This chapter is concluded with suggestions for practical applications of IFA methods and discussions of future directions.

Item factor analysis (IFA), already well established in educational measurement, is increasingly applied to psychological measurement in research settings. However, high-dimensional confirmatory IFA remains a numerical challenge. The current research extends the Metropolis-Hastings Robbins-Monro (MH-RM) algorithm, initially proposed for exploratory IFA, to the case of maximum likelihood estimation under user-defined linear restrictions for confirmatory IFA. MH-RM naturally integrates concepts such as the missing data formulation, data augmentation, the Metropolis algorithm, and stochastic approximation. In a limited simulation study, the accuracy of the MH-RM algorithm is checked against the standard Bock-Aitkin expectation-maximization (EM) algorithm. To demonstrate the efficiency and flexibility of the MH-RM algorithm, it is applied to the IFA of real data from pediatric quality-of-life (QOL) research in comparison with adaptive quadrature-based EM algorithm. The particular data set required a confirmatory item factor model with eight factors and a variety of equality and fixing constraints to implement the hypothesized factor pattern. MH-RM converged in less than 3 minutes to the maximum likelihood solution while the EM algorithm spent well over 4 hourrs.

BackgroundPrimary negative symptoms of schizophrenia contribute heavily to functional disability. Treatment of these symptoms continues to be a major unmet need, even when positive symptoms are controlled. Recent factor analyses of negative symptoms using the PANSS and other symptom assessments in patients with schizophrenia have identified two factors of negative symptoms: expressive and experiential deficits. These two factors most likely have very different clinical, neurocognitive and neurobiological correlates. This study examines the clinical and cognitive correlates associated with expressive and experiential deficits in a large cohort of patients with psychosis before and after computerized cognitive remediation.MethodsThis is a secondary data analysis of subjects enrolled in a cognitive remediation program for 12 weeks. One hundred fifty-one subjects age 18 - 55 with a DSM IV-TR diagnosis of schizophrenia, schizoaffective disorder or bipolar disorder were enrolled. Assessments of demographic, psychopathology (PANSS), cognition (MCBB), and daily living skills (UPSA-Brief) were conducted at baseline and endpoint. Exploratory (EFA) and confirmatory (CFA) factor analyses of PANSS items as well as Pearson’s correlations between factors, demographics, MCCB, and UPSA-Brief scores were examined at baseline and endpoint.ResultsEFA baseline PANSS data resulted in the five-factor model of the PANSS with seven items attributed to the Negative Symptom Factor (NSF; N1, blunted affect; N2, emotional withdrawal; N3, poor rapport; N4, passive social withdrawal; N6, lack of spontaneity and flow of conversation; G7, motor retardation; and G16, active social avoidance). CFA of the NSF revealed a two-factor model consisting of an Expressive Deficit (N1, N3, N6, G7), and an Experiential Deficit (N2, N4, and G16). Difference tests comparing the one-factor and two-factor models found that the two-factor model exhibited significantly better fit than the one-factor model (χ2 = 67.117, df = 1, p ≤ 0.001; CFI = 0.92; Tucker–Lewis index TLI = 0.91; root mean square error of approximation RMSEA = 0.040; and Goodness of Fit index GFI = 0.93). There were significant correlations between the Expressive Deficit factor score and cognition: TMT- A (r=-0.259, p=0.001), BACS Symbol coding (r=-0.287, p=0.001), Category Fluency (r=-0.342, p=0.001), Hopkins Verbal Learning Test – revised (HTLV-R) (r=-0.236, p=0.05), Letter Number Sequencing (r=-0.256, P=0.001), and NAB Mazes (r=-0.409, p=0.001). The Expressive Deficit factor was also significantly correlated with the neurocognitive domains of Processing Speed (r=-0.352, p=0.001) and Reasoning/Problem Solving (r=-0.338, p=0.001). There were no significant correlations between either factor and UPSA-Brief or the MCCB cognitive composite. There were no significant correlations for change from baseline to endpoint in negative symptoms.DiscussionOur results support the negative symptom two-factor model of Expressive Deficit and Experiential Deficit domains. Only the Expressive Deficit factor was associated with baseline deficits in Working Memory, Processing Speed, Reasoning/Problem Solving and Verbal Learning. The association of the Expressive Deficit factor with significant cognitive impairments supports a more profound neurobiological dysfunction in contrast to the Experiential Deficit factor and may represent an important treatment challenge. The relevance of these findings for the treatment of negative symptoms in schizophrenia will be discussed.

Parallel analysis (PA) is an often-recommended approach for assessment of the dimensionality of a variable set. PA is known in different variants, which may yield different dimensionality indications. In this article, the authors considered the most appropriate PA procedure to assess the number of common factors underlying ordered polytomously scored variables. They proposed minimum rank factor analysis (MRFA) as an extraction method, rather than the currently applied principal component analysis (PCA) and principal axes factoring. A simulation study, based on data with major and minor factors, showed that all procedures consistently point at the number of major common factors. A polychoric-based PA slightly outperformed a Pearson-based PA, but convergence problems may hamper its empirical application. In empirical practice, PA-MRFA with a 95% threshold based on polychoric correlations or, in case of nonconvergence, Pearson correlations with mean thresholds appear to be a good choice for identification of the number of common factors. PA-MRFA is a common-factor-based method and performed best in the simulation experiment. PA based on PCA with a 95% threshold is second best, as this method showed good performances in the empirically relevant conditions of the simulation experiment.

ObjectivesInformed consent is a central ethical and legal practice in medicine, yet communication skills specific to this task are under-assessed in undergraduate medical education. This study aimed to develop and validate the informed consent assessment scale (ICAS), a tool designed to evaluate communication competencies essential for delivering informed consent.MethodsThis psychometric validation study was conducted over three academic years (2021–2023) and included 456 fifth-year medical students who completed a 10-min OSCE station on obtaining consent for right colectomy. The ICAS was developed through expert consensus using a structured focus group and qualitative assessment of content validity. Response process evidence was collected by querying assessors about their decision-making during scoring. Internal structure was examined using the exploratory factor analysis (EFA) with tetrachoric correlations, as well as the item response theory (IRT; Rasch and 2-parameter logistic (2PL) models). Reliability was assessed using Cronbach’s alpha, McDonald’s omega, and IRT-derived conditional reliability. Concurrent validity was evaluated through correlations with faculty and standardized-patient communication scores.ResultsParallel analysis supported a one-factor solution. The scale demonstrated essential unidimensionality (UniCo = 0.903, ECV = 0.807, and MIREAL = 0.235) and good model fit (RMSEA = 0.032, CFI = 0.966, and WRMR = 0.039). Reliability was high (McDonald’s ω = 0.841 and Cronbach’s α = 0.837). Q3 analysis indicated no local item dependence (mean Q3 = −0.037 and SD = 0.100). Item discrimination parameters in the 2PL model varied across items, enabling differentiation of student performance. ICAS scores showed strong correlations with global examiner ratings and moderate correlations with broader communication scales, supporting concurrent validity.Conclusion and practice implicationsThe ICAS is a valid and reliable instrument for assessing communication skills specific to informed consent. Its application in objective structured clinical examinations (OSCEs) provides actionable feedback for learners and supports curriculum efforts to strengthen ethically competent clinical communication.