Person-fit Analysis Research Articles

Total, Between-, and Within-Item Attribute Person Fit Analysis Using Mokken Scaling Techniques: An Exploratory Nonparametric Approach to Person Fit

The purpose of this study was to consider how researchers can incorporate item attributes into nonparametric person fit analysis for the purpose of providing additional insight into person fit beyond what is available using typical total-scale person fit approaches. Inspired by parametric person fit statistics that consider patterns of residuals at the total-scale, between-item-attribute, and within-item-attribute levels, we proposed simple adaptations to standard Mokken Scale Analysis (MSA) person fit analysis techniques to reflect these three levels of person fit analysis. We demonstrated the technique using a secondary analysis of survey data related to learning motivation and considered how the results may be interpreted in an affective assessment context. We used a simple exploratory simulation study to consider the techniques under a wider range of conditions. Overall, the results suggested that it is possible to incorporate item attributes into MSA person fit analysis at the total-scale, between-item-attribute, and within-item attribute levels, and that each set of analyses provides a unique perspective on person fit.

Adapting the Person Fit Analysis: Ideas on Detecting Person Misfit in Computerized Adaptive Testing

In this editorial chapter, I aim to summarize findings on person fit analysis in computerized adaptive testing (CAT) from prior research and discuss potential avenues for further research. In item response theory (IRT) applications, person fit quantifies fit of a response pattern to the model (Bradlow & Weiss, 2001, p. 86). Person misfit refers to unexpected response patterns by individuals. There are many potential reasons of misfit including special knowledge (Sinharay, 2016), cheating, guessing (Meijer, 1996), fatigue (Swearingen, 1998), warming up (Meijer, 1996), or faking (Ferrando & Anguiano-Carrasco, 2012). Evaluation of misfit is a significant step for addressing discrepancies within the measurement model. When IRT models are used, evidence of model fit which involves person fit analysis results should be reported (Standard 4.10; AERA, APA & NCME, 2014) as validity evidence to enhance score interpretations. Once misfitting items are identified, corrective steps such as item revision or removal can be implemented. For examinees who exhibit misfit, additional exploration can be undertaken to pinpoint behaviors that might necessitate adjustments to the test program or corrective interventions for particular examinees.

Performance of Nonparametric Person-Fit Statistics with Unfolding versus Dominance Response Models

ABSTRACT Person-fit analyses are commonly used to detect aberrant responding in self-report data. Nonparametric person fit statistics do not require fitting a parametric test theory model and have performed well compared to other person-fit statistics. However, detection of aberrant responding has primarily focused on dominance response data, thus the effectiveness of person-fit statistics in detecting different aberrant behaviors in ideal point data is unclear. This study compares the performance of nonparametric person-fit statistics in unfolding and dominance model contexts. Results for dominance data indicate that increases in detection rates depend, among other factors, on type of aberrant responding and person-fit statistic used. The detection of aberrant responses in ideal point data was ineffective using four nonparametric person-fit statistics, with slightly higher type I error and power less than 0.25. Additional research is needed to identify or develop nonparametric or parametric person-fit statistics effective for aberrant behavior exhibited in ideal point data.

Appraising Traditional and Purpose-built Person Fit Statistics’ Power to Detect Cheating

Person-fit statistics (PFSs) have been suggested as a tool to detect cheating in large-scale testing, and this study investigates their potential for this application. Most PFSs are equally sensitive to scores that appear spuriously high or spuriously low. Xia & Zheng introduced four PFSs that are meant to be more sensitive to spuriously high scores and therefore may be more appropriate for detecting cheating. Comparing the power of these weighted PFSs against the power of traditional PFSs to detect cheating shows that there is no single best statistic in all or most scenarios, and in most scenarios, most examinees flagged as cheating by person fit analysis did not cheat. Implications for operational use of PFSs to detect cheating are discussed.

A Psychometric Look at Principal Professional Development

This study evaluates aspects related to P12 principals’ professional development needs in South Carolina regarding the three domains of school leadership: management, instructional leadership, and program administration. A survey to rate principals’ current leadership knowledge, rank order their professional development needs, and provide a confidence rating regarding their abilities was given to over 1,100 principals and 85 superintendents. Through examining relationships with a psychometric model, results derived latent leadership ability scores and self-reported confidence ratings of principals as well as the superintendents’ leadership scores and confidence ratings of their principals. This study found a significant discrepancy between principals’ and superintendents’ confidence ratings and their corresponding leadership ability scores, respectively. A further analysis of the rank-ordered professional development needs highlighted instructional leadership to be the most needed topic for professional development. Finally, atypical response patterns regarding principal’s current leadership knowledge are also identified through person-fit analysis to provide additional information regarding P-12 principals’ professional development needs.

Rasch Analysis of the University Student Depression Inventory (USDI) Using the Polytomous Partial Credit Model

The University Student Depression Inventory (USDI; Khawaja and Bryden 2006) is a 30- item scale that is used to measure depressive symptoms among university students. Its psychometric properties have been widely investigated under the classical test theory (CTT). This study explored the application of the polytomous Rasch partial credit model (PCM) in evaluating the USDI using response data from a sample of Filipino university students (n = 441). Using sequential tests under the Rasch measurement framework, model fitting was performed through item- and person-fit analyses to detect and address possible sources of measurement noise, followed by tests of local independence and differential item functioning (DIF). Results revealed that the original scale contained five misfitting items (6, 7, 10, 12, 20); hence, the deletion of such items was proposed to provide a new but psychometrically sound measure of student depression. Further analysis of the data detected person misfits whose responses were removed in subsequent analyses of local independence and DIF. One pair of locally dependent items (25, 26) and three gender-biased items (1, 3, 8) were detected, which necessitates further item review for possible idiosyncratic meanings. This study showed that Rasch analysis of self-reported questionnaires like the USDI can complement factor analytic approaches, especially in the detection of multiple sources of measurement errors that may undermine the quality of survey data.

A cross-cultural comparison of non-cognitive outputs towards science between Turkish and Dutch students taking into account detected person misfit

Comparing countries according to their PISA results can be considered cross-cultural studies. An important issue in these studies is that the measurement tools must be culturally and linguistically equivalent. Cultural or linguistic differences in measurement tools may threaten validity. Aberrant behavior is another important factor that affects validity. Person-fit analysis is a statistical method of detecting aberrant individual behavior. This study investigated the effects of students who were flagged as aberrant on a comparison of Turkish and Dutch students’ non-cognitive outputs towards science in the PISA 2015. Whether the items contained differential item functioning (DIF) or not was determined by a Poly-SIBTEST for each scale. Misfit students were detected by non-parametric GNp and U3p statistics. It was determined that flagged students were not effective for comparing the mean of non-cognitive outputs towards science, but it was observed that flagged students had an effect on DIF.

A Model‐Data‐Fit‐Informed Approach to Score Resolution in Performance Assessments

AbstractMany large‐scale performance assessments include score resolution procedures for resolving discrepancies in rater judgments. The goal of score resolution is conceptually similar to person fit analyses: To identify students for whom observed scores may not accurately reflect their achievement. Previously, researchers have observed that rater‐agreement methods and person fit analyses result in similar conclusions about which students’ achievement estimates warrant additional investigation, and that score resolution generally improves person fit. We consider the implications of using person fit analysis as an initial step to identify performances for score resolution, and of using fit indices to identify raters to provide additional ratings. We simulated student responses to multiple‐choice items and a writing task. We simulated various types of person misfit in the writing task and identified the persons who needed resolution using a person‐fit index. Results indicate larger improvements in person fit after resolution when the person fit approach was used compared to a rater‐agreement approach. With the fit‐informed approach, person fit improved for of the misfitting students; with the rater‐agreement approach, person fit improved for around one third of these students. We consider the implications of our findings for mixed‐format assessments.

Application of Rasch Analysis in Measuring Teacher Collegial Supervisory Instrument's Reliability and Validity

The concept of collegial supervision (CS) is defined as collaborative work beyond their professional sphere of relationships often offered by educators through feedback and sharing platform. However, there is still lack of studies and instruments that evaluated the CS practice within Malaysian context. In measuring the suitability of CS practice in secondary schools in Malaysia, a questionnaire with 26 self-developed items that represent five sub-dimensions/constructs namely, collegial relations, teacher's province, teacher growth, teacher collaboration, and reflective inquiry was developed from series of interviews with secondary teachers. The major aim of this paper is to validate and examined the psychometric elements through the application of Rasch analysis in measuring items'-person reliability, principal component analysis, items - person distribution, fit and dimensionality analyses. The analysis was performed based on feedback of 357 teachers from secondary teachers. Findings revealed on high values on person-items reliability, and the items' difficulty are significantly aligned or matched with teachers' ability. Also, principal component analysis revealed an acceptable value of raw variance explained, and that most teachers agreed with most of the items through structure measurement on the items' validity. Thus, it is concluding on the internal consistencies of the items within Malaysian CS which later contributed to the CS items for Malaysian secondary schools.

Identifying Misfitting Achievement Estimates in Performance Assessments: An Illustration Using Rasch and Mokken Scale Analyses

Researchers apply individual person fit analyses as a procedure for checking model-data fit for individual test-takers. When a test-taker misfits, it means that the inferences from their test score regarding what they know and can do may not be accurate. One problem in applying individual person fit procedures in practice is the question of how much misfit it takes to make the test score an untrustworthy estimate of achievement. In this paper, we argue that if a person’s responses generally follow a monotonic pattern, the resulting test score is “good enough” to be interpreted and used. We present an approach that applies statistical procedures from the Rasch and Mokken measurement perspectives to examine individual person fit based on this good enough criterion in real data from a performance assessment. We discuss how these perspectives may facilitate thinking about applying individual person fit procedures in practice.

A Short Note on Aberrant Responses Bias in Item Response Theory

Item response models often cannot calculate true individual response probabilities because of the existence of response disturbances (such as guessing and cheating). Many studies on aberrant responses under item response theory (IRT) framework had been conducted. Some of them focused on how to reduce the effect of aberrant responses, and others focused on how to detect aberrant examinees, such as person fit analysis. The purpose of this research was to derive a generalized formula of bias with/without aberrant responses, that showed the effect of both non-aberrant and aberrant response data on the bias of capability estimation mathematically. A new evaluation criterion, named aberrant absolute bias (|ABIAS|), was proposed to detect aberrant examinees. Simulation studies and application to a real dataset were conducted to demonstrate the efficiency and the utility of |ABIAS|.

Detecting Random Responses in a Personality Scale Using IRT-Based Person-Fit Indices

Abstract. Random response is a very common aberrant response behavior in personality tests and may negatively affect the reliability, validity, or other analytical aspects of psychological assessment. Typically, researchers use a single person-fit index to identify random responses. This study recommends a three-step person-fit analysis procedure. Unlike the typical single person-fit methods, the three-step procedure identifies both global misfit and local misfit individuals using different person-fit indices. This procedure was able to identify more local misfit individuals than single-index method, and a graphical method was used to visualize those particular items in which random response behaviors appear. This method may be useful to researchers in that it will provide them with more information about response behaviors, allowing better evaluation of scale administration and development of more plausible explanations. Real data were used in this study instead of simulation data. In order to create real random responses, an experimental test administration was designed. Four different random response samples were produced using this experimental system.

Exploring the correspondence between traditional score resolution methods and person fit indices in rater-mediated writing assessments

Abstract Scoring procedures for rater-mediated writing assessments often include checks for agreement between the raters who score students’ essays. When raters assign non-adjacent ratings to the same essay, a third rater is often employed to “resolve” the discrepant ratings. The procedures for flagging essays for score resolution are similar to person fit analyses based on item response theory (IRT). We used data from two writing performance assessments in science and social studies to explore the correspondence between traditional score resolution procedures and IRT person fit statistics. We observed that rater agreement criteria and person fit criteria flag many, but not all, of the same rating profiles for additional investigation. We also observed significantly different values of person fit statistics between students whose essays were and were not flagged for third ratings by the rater agreement criteria. Finally, when we used resolved ratings in place of the original ratings, we observed improvements in person fit for most, but not all, of the students whose essays were flagged for third ratings. These results suggest that person fit analyses may provide a complimentary approach to rater agreement criteria. We discuss these results in terms of their implications for research and practice.

Methods for Estimating Item-Score Reliability.

Reliability is usually estimated for a test score, but it can also be estimated for item scores. Item-score reliability can be useful to assess the item’s contribution to the test score’s reliability, for identifying unreliable scores in aberrant item-score patterns in person-fit analysis, and for selecting the most reliable item from a test to use as a single-item measure. Four methods were discussed for estimating item-score reliability: the Molenaar–Sijtsma method (method MS), Guttman’s method , the latent class reliability coefficient (method LCRC), and the correction for attenuation (method CA). A simulation study was used to compare the methods with respect to median bias, variability (interquartile range [IQR]), and percentage of outliers. The simulation study consisted of six conditions: standard, polytomous items, unequal parameters, two-dimensional data, long test, and small sample size. Methods MS and CA were the most accurate. Method LCRC showed almost unbiased results, but large variability. Method consistently underestimated item-score reliabilty, but showed a smaller IQR than the other methods.

Using person response functions to investigate areas of person misfit related to item characteristics

ABSTRACTIndividual person fit analyses provide important information regarding the validity of test score inferences for an individual test taker. In this study, we use data from an undergraduate statistics test (N = 1135) to illustrate a two-step method that researchers and practitioners can use to examine individual person fit. First, person fit is examined numerically with several indices based on the Rasch model (i.e., Infit, Outfit, and Between-Subset statistics). Second, person misfit is presented graphically with person response functions, and these person response functions are interpreted using a heuristic. Individual person fit analysis holds promise for improving score interpretation in that it may detect potential threats to validity of score inferences for some test takers. Individual person fit analysis may also highlight particular subsets of items (on which a test taker performs unexpectedly) that can be used to further contextualize her or his test performance.

Why Education Practitioners and Stakeholders Should Care About Person Fit in Educational Assessments

In this article, A. Adrienne Walker introduces the concept of person fit to education stakeholders as a source of evidence to inform the trustworthiness of a test score for interpretation and use (validity). Person fit analyses are used in educational measurement research to explore the degree to which a person's test score can be interpreted as a reasonable estimate of her achievement level. Although much academic literature has been produced on person fit, the findings and implications of this research remain largely unknown to education stakeholders. Walker argues that the practice of using test scores based on misfitting response patterns is problematic, and by including individual person fit in quality procedures, assessment research, practice, and policy can improve the way educational assessments serve instruction.

The Person-Fit Statistic in an Unfolding Model Context.

Although person-fit analysis has a long-standing tradition within item response theory, it has been applied in combination with dominance response models almost exclusively. In this article, a popular log likelihood-based parametric person-fit statistic under the framework of the generalized graded unfolding model is used. Results from a simulation study indicate that the person-fit statistic performed relatively well in detecting midpoint response style patterns and not so well in detecting extreme response style patterns.

Response Inconsistency of Patient-Reported Symptoms as a Predictor of Discrepancy Between Patient and Clinician-Reported Depression Severity.

The aim of this study was to assess the extent to which discrepancy between self-reported and clinician-rated severity of depression are due to inconsistent self-reports. Response inconsistency threatens the validity of the test score. We used data from a large sample of outpatients ( N = 5,959) who completed the self-report Beck Depression Inventory-II (BDI-II) and the clinician-rated Montgomery-Åsberg Depression Rating Scale (MADRS). We used item response theory based person-fit analysis to quantify the inconsistency of the self-report item scores. Inconsistency was weakly positively related to patient-clinician discrepancy (i.e., higher BDI-II scores relative to MADRS scores). The mediating effect of response inconsistency in the relationship between discrepancy and demographic (e.g., ethnic origin) and clinical variables (e.g., cognitive problems) was negligible. The small direct and mediating effects of response inconsistency suggest that inaccurate patient self-reports are not a major cause of patient-clinician discrepancy in outpatient samples. Future research should investigate the role of clinician biases in explaining clinician-patient discrepancy.

Person Fit Analysis in Computerized Adaptive Testing Using Tests for a Change Point

Meijer and van Krimpen-Stoop noted that the number of person-fit statistics (PFSs) that have been designed for computerized adaptive tests (CATs) is relatively modest. This article partially addresses that concern by suggesting three new PFSs for CATs. The statistics are based on tests for a change point and can be used to detect an abrupt change in test performance of examinees during a CAT. The Type I error rate and power of the statistics are computed from a detailed simulation study. The performances of the new statistics are compared with those of four existing PFSs using receiver operating characteristics curves. The new statistics are then computed using data from an operational and high-stakes CAT. The new PFSs appear promising for assessment of person fit for CATs.

PerFit: An R Package for Person-Fit Analysis in IRT

Checking the validity of test scores is important in both educational and psychological measurement. Person-fit analysis provides several statistics that help practitioners assessing whether individual item score vectors conform to a prespecified item response theory model or, alternatively, to a group of test takers. Software enabling easy access to most person-fit statistics was lacking up to now. The PerFit R package was written in order to fill in this void. A theoretical overview of relatively simple person-fit statistics is provided. A practical guide showing how the main functions of PerFit can be used is also given. Both numerical and graphical tools are described and illustrated using examples. The goal is to show how person-fit statistics can be easily applied to testing of questionnaire data.