Person-fit Indices Research Articles

Enhancing the Detection of Social Desirability Bias Using Machine Learning: A Novel Application of Person-Fit Indices

Social desirability bias (SDB) is a common threat to the validity of conclusions from responses to a scale or survey. There is a wide range of person-fit statistics in the literature that can be employed to detect SDB. In addition, machine learning classifiers, such as logistic regression and random forest, have the potential to distinguish between biased and unbiased responses. This study proposes a new application of these classifiers to detect SDB by considering several person-fit indices as features or predictors in the machine learning methods. The results of a Monte Carlo simulation study showed that for a single feature, applying person-fit indices directly and logistic regression led to similar classification results. However, the random forest classifier improved the classification of biased and unbiased responses substantially. Classification was improved in both logistic regression and random forest by considering multiple features simultaneously. Moreover, cross-validation indicated stable area under the curves (AUCs) across machine learning classifiers. A didactical illustration of applying random forest to detect SDB is presented.

Comparing Person-Fit and Traditional Indices Across Careless Response Patterns in Surveys.

Methods to identify carelessness in survey research can be valuable tools in reducing bias during survey development, validation, and use. Because carelessness may take multiple forms, researchers typically use multiple indices when identifying carelessness. In the current study, we extend the literature on careless response identification by examining the usefulness of three item-response theory-based person-fit indices for both random and overconsistent careless response identification: infit MSE outfit MSE, and the polytomous lz statistic. We compared these statistics with traditional careless response indices using both empirical data and simulated data. The empirical data included 2,049 high school student surveys of teaching effectiveness from the Network for Educator Effectiveness. In the simulated data, we manipulated type of carelessness (random response or overconsistency) and percent of carelessness present (0%, 5%, 10%, 20%). Results suggest that infit and outfit MSE and the lz statistic may provide complementary information to traditional indices such as LongString, Mahalanobis Distance, Validity Items, and Completion Time. Receiver operating characteristic curves suggested that the person-fit indices showed good sensitivity and specificity for classifying both over-consistent and under-consistent careless patterns, thus functioning in a bidirectional manner. Carelessness classifications based on low fit values correlated with carelessness classifications from LongString and completion time, and classifications based on high fit values correlated with classifications from Mahalanobis Distance. We consider implications for research and practice.

Functional Data Analysis and Person Response Functions

ABSTRACT The purpose of this study is to illustrate the use of functional data analysis (FDA) as a general methodology for analyzing person response functions (PRFs). Applications of FDA to psychometrics have included the estimation of item response functions and latent distributions, as well as differential item functioning. Although FDA has been suggested for modeling PRFs, there has been relatively little research stressing this application. FDA offers an approach for diagnosing person responses that may be due to guessing and other sources of within-person multidimensionality. PRFs provide graphical displays that can be used to highlight unusual response patterns, and to identify persons that are not responding as expected to a set of test items. In addition to examining individual PRFs, functional clustering techniques can be used to identify subgroups of persons that may be exhibiting categories of misfit such as guessing. A small simulation study is conducted to illustrate how FDA can be used to identify persons exhibiting different levels of guessing behavior (5%, 10%, 15% and 20%). The methodology is also applied to real data from a 3rd grade science assessment used in a southeastern state. FDA offers a promising methodology for evaluating whether or not meaningful scores have been obtained for a person. Typical indices of psychometric quality, such as standard errors of measurement and person fit indices, are not sufficient for representing certain types of aberrance in person response patterns. Nonparametric graphical methods for estimating PRFs that are based FDA provide a rich source of validity evidence regarding the meaning and usefulness of each person’s score.

A Comparison of Person-Fit Indices to Detect Social Desirability Bias.

Social desirability bias (SDB) has been a major concern in educational and psychological assessments when measuring latent variables because it has the potential to introduce measurement error and bias in assessments. Person-fit indices can detect bias in the form of misfitted response vectors. The objective of this study was to compare the performance of 14 person-fit indices to identify SDB in simulated responses. The area under the curve (AUC) of receiver operating characteristic (ROC) curve analysis was computed to evaluate the predictive power of these statistics. The findings showed that the agreement statistic outperformed all other person-fit indices, while the disagreement statistic , dependability statistic , and the number of Guttman errors also demonstrated high AUCs to detect SDB. Recommendations for practitioners to use these fit indices are provided.

The Effect of Person Misfit on Item Parameter Estimation and Classification Accuracy: A Simulation Study

Often, important decisions regarding accountability and placement of students in performance categories are made on the basis of test scores generated from tests, therefore, it is important to evaluate the validity of the inferences derived from test results. One of the threats to the validity of such inferences is aberrant responding. Several person fit indices were developed to detect aberrant responding on educational and psychological tests. The majority of the person fit literature has been focused on creating and evaluating new indices. The aim of this study was to assess the effect of aberrant responding on the accuracy of estimated item parameters and refining estimations by using person fit statistics by means of simulation. Our results showed that the presence of aberrant response patterns created bias in the both b and a parameters at the item level and affected the classification of students, particularly high-performing students, into performance categories regardless of whether aberrant response patterns were present in the data or were removed. The results differed by test length and the percentage of students with aberrant response patterns. Practical and theoretical implications are discussed.

The Detection of Cheating on E-Exams in Higher Education-The Performance of Several Old and Some New Indicators.

In this paper, we compare the performance of 18 indicators of cheating on e-exams in higher education. Basis of the study was a field experiment. The experimental setting was a computer assisted mock exam in an introductory course on psychology conducted at a university. The experimental manipulation consisted in inducing two forms of cheating (pre-knowledge, test collusion) in a subgroup of the examinees. As indicators of cheating, we consider well-established person-fit indices (e.g., the U3 statistic), but also several new ones based on process data (e.g., response times). The indicators were evaluated with respect to their capability to separate the subgroup of the cheaters from the remaining examinees. We additionally employed a classification tree for detecting the induced cheating behavior. With this proceeding, we aimed at investigating the detectability of cheating in the day-to-day educational setting where conditions are suboptimal (e.g., tests with low psychometric quality are used). The indicators based on the number of response revisions and the response times were capable to indicate the examinees who cheated. The classification tree achieved an accuracy of 0.95 (sensitivity: 0.42/specificity: 0.99). In the study, the number of revisions was the most important predictor of cheating. We additionally explored the performance of the indicators to predict the specific form of cheating. The specific form was identified with an accuracy of 0.93.

Identifying and Classifying Aberrant Response Patterns Through Functional Data Analysis

We propose new methods for identifying and classifying aberrant response patterns (ARPs) by means of functional data analysis. These methods take the person response function (PRF) of an individual and compare it with the pattern that would correspond to a generic individual of the same ability according to the item-person response surface. ARPs correspond to atypical difference functions. The ARP classification is done with functional data clustering applied to the PRFs identified as ARP. We apply these methods to two sets of simulated data (the first is used to illustrate the ARP identification methods and the second demonstrates classification of the response patterns flagged as ARP) and a real data set (a Grade 12 science assessment test, SAT, with 32 items answered by 600 examinees). For comparative purposes, ARPs are also identified with three nonparametric person-fit indices (Ht, Modified Caution Index, and ZU3). Our results indicate that the ARP detection ability of one of our proposed methods is comparable to that of person-fit indices. Moreover, the proposed classification methods enable ARP associated with either spuriously low or spuriously high scores to be distinguished.

Effectiveness of Person Fit Indices in Item Response Models with Different Degrees of Item Local Dependence

This study aimed to examine the effectiveness of person fit indices (Wright’s weighted index, Drasgow index and Almehrizi’s weighted index) in item response models with different degrees of item local dependence (0.0, 0.3, 0.6, and 0.9) using simulated item parameters. Item responses for 40 samples each with 10000 subjects (a total of 400000 subjects) were simulated on a test of 60 items. Item discrimination parameters ranged between 0.19 and 1.79 and item difficulty parameters ranged between -2 and +2. 20% of test items were manipulated to show local dependence for each level of local dependence degrees. Student ability was generated to follow a standard normal distribution. Assumptions of item response theory were examined in all data sets using exploratory factor analysis and residual analysis using NOHARM platform for unidimensionality and Q3 index for local independence. Results showed that there was an increase in the percentages of non-conforming persons when increasing the degree of items local dependence for the three person fit indices (Wright’s weighted index, Drasgow index and Almehrizi’s weighted index). Results showed also that the percentages of non-conforming persons were larger with Wright’s weighted index than with Drasgow index and Almehrizi’s weighted index. The distributional properties of the three indices showed relatively consistent in distributional properties. Drasgow index and Almehrizi’s weighted index were very similar distributional properties. Also, there was a larger agreement index between Wright’s weighted index and Drasgow index.

Effectiveness of Person Fit Indices in Item Response Models with Different Degrees of Item Local Dependence

This study aimed to examine the effectiveness of person fit indices (Wright’s weighted index, Drasgow index and Almehrizi’s weighted index) in item response models with different degrees of item local dependence (0.0, 0.3, 0.6, and 0.9) using simulated item parameters. Item responses for 40 samples each with 10000 subjects (a total of 400000 subjects) were simulated on a test of 60 items. Item discrimination parameters ranged between 0.19 and 1.79 and item difficulty parameters ranged between -2 and +2. 20% of test items were manipulated to show local dependence for each level of local dependence degrees. Student ability was generated to follow a standard normal distribution. Assumptions of item response theory were examined in all data sets using exploratory factor analysis and residual analysis using NOHARM platform for unidimensionality and Q3 index for local independence. Results showed that there was an increase in the percentages of non-conforming persons when increasing the degree of items local dependence for the three person fit indices (Wright’s weighted index, Drasgow index and Almehrizi’s weighted index). Results showed also that the percentages of non-conforming persons were larger with Wright’s weighted index than with Drasgow index and Almehrizi’s weighted index. The distributional properties of the three indices showed relatively consistent in distributional properties. Drasgow index and Almehrizi’s weighted index were very similar distributional properties. Also, there was a larger agreement index between Wright’s weighted index and Drasgow index.

Using Rasch Model to Detect Differential Person Functioning and Cheating Behavior in Natural Sciences Learning Achievement Test

The existence of abberant response showed the inaccuracy of measurement which, in turn, threatens the test validity. This study aims at: (1) Discovering the proportion of students who were having Differential Person Functioning (DPF) in final term assessment test of natural sciences course for 8th grade in odd semester of academic year 2016/2017 in Tegal Regency, Indonesia; (2) Identifying the students suspected of cheating during the final term assessment test of natural sciences course for 8th grade in odd semester of academic year 2016/2017 in Tegal Regency, Indonesia. This research involved 1011 student responses to final term assessment test of natural sciences course for 8th grade in odd semester of academic year 2016/2017 in Tegal Regency, Indonesia. The data were taken from four junior high schools; SMPN I Dukuhturi, SMPN I Suradadi, SMPN 2 Slawi and SMPN 2 Dukuhwaru. The scoring was done using Rasch model and the person fit index used Sijtsma’s Ht person fit statistic (Ht). The result showed: (1) 14% of the students attending final term assessment of natural sciences course for 8th grade in the odd semester of academic year 2016/2017 in Tegal Regency, Indonesia were detected of having DPF. From the four junior high schools involved in the research, three junior high schools have a proportion of students having DPF at a range from 9.6% to 23%, and only 1.1% of the other one’s students were having DPF; (2) Nine pairs of students were suspected of cheating during the final term assessment of natural sciences course for 8th grade in the odd semester of academic year 2016/2017 in Tegal Regency, Indonesia.

Rasch Analysis of the Activities-Specific Balance Confidence Scale in Individuals Poststroke

ObjectiveTo examine the psychometric properties of the Activities-specific Balance Confidence (ABC) scale using Rasch analysis for individuals poststroke. DesignRetrospective cohort. SettingData was extracted from the Locomotor Experience Applied Post-Stroke phase 3, multisite, randomized controlled clinical trial. ParticipantsCommunity-dwelling, ambulatory, older adults (N=406) (mean age ± SD, 61.97±12.76y; 45.07% female) approximately 2 months poststroke. InterventionNone. Main Outcome MeasuresWe examined unidimensionality, local dependence, rating-scale structure, item and person fit, person-item match, and separation index of the ABC scale. ResultsConfirmatory and exploratory factor analysis showed the ABC scale was adequately unidimensional and 3-item pairs had local dependence. A collapsed 5-category rating scale was superior to the 101-category scale. The hardest item was “walking outside on an icy sidewalk,” the easiest item was “getting into or out of a car,” and no items misfit. The ABC scale had high person reliability (0.93), despite 10.5% of individuals misfitting the expected response pattern. Mean ability level of the sample was slightly lower (−0.56 logits) than the mean item difficulty indicating that the ABC scale adequately matched our sample’s balance confidence. The ABC scale did not have a floor or ceiling effect and separated individuals into 5 statistically distinct strata (separation index=3.71). ConclusionsThe Rasch model supports the use of the ABC scale to measure balance confidence in individuals poststroke. The consistency of our results with previous Rasch analyses on the ABC scale demonstrates the instrument responds similarly across multiple populations; community-dwelling older-adults, outpatient orthopedic physical therapy, stroke, Parkinson disease, and lower-limb amputation. Recommendations include collapsing the rating scale and developing a computerized-adaptive test version of the scale to enhance clinical utility.

Adjusting Person Fit Index for Skewness in Cognitive Diagnosis Modeling

Because the validity of diagnostic information generated by cognitive diagnosis models (CDMs) depends on the appropriateness of the estimated attribute profiles, it is imperative to ensure the accurate measurement of students’ test performance by conducting person fit (PF) evaluation to avoid flawed remediation measures. The standardized log-likelihood statistic lZ has been extended to the CDM framework. However, its null distribution is found to be negatively skewed. To address this issue, this study applies different methods of adjusting the skewness of lZ that have been proposed in the item response theory context, namely, χ2-approximation, Cornish-Fisher expansion, and Edgeworth expansion to bring its null distribution closer to the standard normal distribution. The skewness-corrected PF statistics are investigated by calculating their type I error and detection rates using a simulation study. Fraction-subtraction data are also used to illustrate the application of these PF statistics.

The Assessment and Impact of Careless Responding in Routine Outcome Monitoring within Mental Health Care

Careless responding by mental health patients on self-report assessments is rarely investigated in routine care despite the potential for serious consequences such as faulty clinical decisions. We investigated validity indices most appropriate for detecting careless responding in routine outcome monitoring (ROM) in mental health-care. First, we reviewed indices proposed in previous research for their suitability in ROM. Next, we evaluated six selected indices using data of the Brief Symptom Inventory and the Mood and Anxiety Symptom Questionnaire from 3,483 outpatients. Simulations showed that for typical ROM scales the index, Mahalanobis distance, and inter-item standard deviation may be too strongly confounded with the latent trait value to compare careless responding across patients with different symptom severity. Application of two different classification methods to the validity indices did not converge in similar prevalence estimates of careless responding. Finally, results suggest that careless responding does not have a substantial biasing effect on scale-score statistics. We recommend the person-fit index to screen for random careless responding in large ROM data sets. However, additional research should further investigate methods for detecting repetitive responding in typical ROM data and assess whether there are specific circumstances in which simpler validity statistics or direct screening methods perform similarly as the index.

Determining at What Age Children Provide Sound Self-Reports: An Illustration of the Validity-Index Approach.

In psychological assessment of children, it is pivotal to establish from what age on self-reports can complement or replace informant reports. We introduce a psychometric approach to estimate the minimum age for a child to produce self-report data that is of similar quality as informant data. The approach makes use of statistical validity indicators such as person-fit and long-string indices, and can be readily applied to data commonly collected in psychometric studies of child measures. We evaluate and illustrate the approach, using self-report and informant-report data of the PedsQL, a pediatric health-related quality of life measure, from 651 child-mother pairs. To evaluate the approach, we tested various hypotheses about the validity of the self-report data, using the person-fit index as the validity indicator and the mother informant-data as a benchmark for validity. Results showed that discriminated between self-reports of younger and older children, between self-reports of children that completed the PedsQL alone or with a parent, and between self-reports and informant reports. We conclude that the validity-index approach has good potential for future applications. Future research should further evaluate the approach for different types of questionnaires (e.g., personality inventories) and using different validity indices (e.g., response-bias indices).

Aberrant Response Patterns on the Jordanian Version of Otis–Lennon Test and their Impact on the Accuracy of Estimating a Person's Ability and Information Function

This study aimed at investigating the aberrant response patterns and their impacts on the Jordanian version Otis- Lennon as well as the accuracy in the estimation of a person's' ability and information function test. To achieve this goal, the Jordanian version of the Ability Test primary II level form K was administrated to 568 first-grade male and female students of Ajloun district public schools during 2016/2017. The Lz person fit index and the three-parameter logistic model were used to analyze students' responses to test items to assess the person ability, information function test, and to detect aberrant response patterns. The results revealed that the response patterns of 56 students were aberrant based on Lz index; also the results showed that the factors responsible for the presence of this aberrant response were: guessing, cheating, laziness, and exponential creatively. The results indicated that when the aberrant patterns were excluded, both the accuracy person's estimating ability and the information function test had significantly increased at different ability levels.

Aberrant Response Patterns on the Jordanian Version of Otis–Lennon Test and their Impact on the Accuracy of Estimating a Person's Ability and Information Function

This study aimed at investigating the aberrant response patterns and their impacts on the Jordanian version Otis- Lennon as well as the accuracy in the estimation of a person's' ability and information function test. To achieve this goal, the Jordanian version of the Ability Test primary II level form K was administrated to 568 first-grade male and female students of Ajloun district public schools during 2016/2017. The Lz person fit index and the three-parameter logistic model were used to analyze students' responses to test items to assess the person ability, information function test, and to detect aberrant response patterns. The results revealed that the response patterns of 56 students were aberrant based on Lz index; also the results showed that the factors responsible for the presence of this aberrant response were: guessing, cheating, laziness, and exponential creatively. The results indicated that when the aberrant patterns were excluded, both the accuracy person's estimating ability and the information function test had significantly increased at different ability levels.

Detecting Random Responses in a Personality Scale Using IRT-Based Person-Fit Indices

Abstract. Random response is a very common aberrant response behavior in personality tests and may negatively affect the reliability, validity, or other analytical aspects of psychological assessment. Typically, researchers use a single person-fit index to identify random responses. This study recommends a three-step person-fit analysis procedure. Unlike the typical single person-fit methods, the three-step procedure identifies both global misfit and local misfit individuals using different person-fit indices. This procedure was able to identify more local misfit individuals than single-index method, and a graphical method was used to visualize those particular items in which random response behaviors appear. This method may be useful to researchers in that it will provide them with more information about response behaviors, allowing better evaluation of scale administration and development of more plausible explanations. Real data were used in this study instead of simulation data. In order to create real random responses, an experimental test administration was designed. Four different random response samples were produced using this experimental system.

Exploring the correspondence between traditional score resolution methods and person fit indices in rater-mediated writing assessments

Abstract Scoring procedures for rater-mediated writing assessments often include checks for agreement between the raters who score students’ essays. When raters assign non-adjacent ratings to the same essay, a third rater is often employed to “resolve” the discrepant ratings. The procedures for flagging essays for score resolution are similar to person fit analyses based on item response theory (IRT). We used data from two writing performance assessments in science and social studies to explore the correspondence between traditional score resolution procedures and IRT person fit statistics. We observed that rater agreement criteria and person fit criteria flag many, but not all, of the same rating profiles for additional investigation. We also observed significantly different values of person fit statistics between students whose essays were and were not flagged for third ratings by the rater agreement criteria. Finally, when we used resolved ratings in place of the original ratings, we observed improvements in person fit for most, but not all, of the students whose essays were flagged for third ratings. These results suggest that person fit analyses may provide a complimentary approach to rater agreement criteria. We discuss these results in terms of their implications for research and practice.

Response Time Based Nonparametric Kullback-Leibler Divergence Measure for Detecting Aberrant Test-Taking Behavior

Many important high-stakes decisions—college admission, academic performance evaluation, and even job promotion—depend on accurate and reliable scores from valid large-scale assessments. However, examinees sometimes cheat by copying answers from other test-takers or practicing with test items ahead of time, which can undermine the effectiveness of such assessments in yielding accurate, precise information of examinees' performances. This study focuses on the utility of a new nonparametric person-fit index using examinees' response times to detect two types of cheating behaviors. The feasibility of this method was investigated vis-à-vis a Monte Carlo simulation as well as through analyzing data from a large-scale assessment. Findings indicate that the proposed index was quite successful in detecting pre-knowledge cheating and extreme one-item cheating.

Case Diagnostics for Factor Analysis of Ordered Categorical Data With Applications to Person-Fit Measurement

Mahalanobis distance (M-distance) case diagnostics are a useful tool for assessing response pattern inconsistency in factor analysis; however, the derivations of these statistics assume continuous variables, which limits their utility in ordinal self- or rater-report data. This research generalizes M-distance diagnostics to categorical factor analysis. We prove that the residual-based M-distance dr is equivalent to the person-fit index lco, which motivates the use of the new categorical M-distance dr* as a person-fit index. dr* is compared and contrasted with zh, a commonly used item response theory person-fit index. A simulation study is used to show that a simple transformation of dr* satisfies established criteria for person-fit measures. A sample of responses to the Rosenberg Self-Esteem Scale is used to determine parameters for a simulation study, and real data are analyzed to contrast the use of dr and dr* as indexes of person-fit in continuous and categorical factor analysis.