Item Parameter Drift Research Articles

A many-facet Rasch measurement model approach to investigating objective structured clinical examination item parameter drift.

Objective Structured Clinical Examinations (OSCEs) are widely used for assessing clinical competence, especially in high-stakes environments such as medical licensure. However, the reuse of OSCE cases across multiple administrations raises concerns about parameter stability, known as item parameter drift (IPD). AIMS &OBJECTIVES: This study aims to investigate IPD in reused OSCE cases while accounting for examiner scoring effects using a Many-facet Rasch Measurement (MFRM) model. Data from 12 OSCE cases, reused over seven administrations of the Internationally Educated Nurse Competency Assessment Program (IENCAP), were analyzed using the MFRM model. Each case was treated as an item, and examiner scoring effects were accounted for in the analysis. The results indicated that despite accounting for examiner effects, all cases exhibited some level of IPD, with an average absolute IPD of 0.21 logits. Three cases showed positive directional trends. IPD significantly affected score decisions in 1.19% of estimates, at an invariance violation of 0.58 logits. These findings suggest that while OSCE cases demonstrate sufficient stability for reuse, continuous monitoring is essential to ensure the accuracy of score interpretations and decisions. The study provides an objective threshold for detecting concerning levels of IPD and underscores the importance of addressing examiner scoring effects in OSCE assessments. The MFRM model offers a robust framework for tracking and mitigating IPD, contributing to the validity and reliability of OSCEs in evaluating clinical competence.

Leveraging Item Parameter Drift to Assess Transfer Effects in Vocabulary Learning

ABSTRACT Longitudinal models typically emphasize between-person predictors of change but ignore how growth varies within persons because each person contributes only one data point at each time. In contrast, modeling growth with multi-item assessments allows evaluation of how relative item performance may shift over time. While traditionally viewed as a nuisance under the label of “item parameter drift” (IPD), IPD may be of substantive interest if it reflects how learning manifests on different items or subscales at different rates. In this study, we apply the Explanatory Item Response Model to assess IPD in a causal inference context. Simulation results show that when IPD is not accounted for, both parameter estimates and standard errors can be affected. We illustrate with an empirical application to the persistence of transfer effects from a content literacy intervention , revealing how researchers can leverage IPD to achieve a more fine-grained understanding of how vocabulary learning develops over time.

Detecting Item Parameter Drift in Small Sample Rasch Equating

ABSTRACT Screening items for parameter drift helps protect against serious validity threats and ensure score comparability when equating forms. Although many high-stakes credentialing examinations operate with small sample sizes, few studies have investigated methods to detect drift in small sample equating. This study demonstrates that several newly researched drift detection strategies can improve equating accuracy under certain conditions with small samples where some anchor items display item parameter drift. Results showed that the recently proposed methods mINFIT and mOUTFIT as well as the more conventional Robust-z helped mitigate the adverse effects of drifting anchor items in conditions with higher drift levels or with more than 75 examinees. In contrast, the Logit Difference approach excessively removed invariant anchor items. The discussion provides recommendations on how practitioners working with small samples can use the results to make more informed decisions regarding item parameter drift.

Comparing Drift Detection Methods for Accurate Rasch Equating in Different Sample Sizes

ABSTRACT Rasch common-item equating is often used in high-stakes testing to maintain equivalent passing standards across test administrations. If unaddressed, item parameter drift poses a major threat to the accuracy of Rasch common-item equating. We compared the performance of well-established and newly developed drift detection methods in small and large sample sizes, varying the proportion of test items used as anchor (common) items and the proportion of drifted anchors. In the simulated-data study, the most accurate equating was obtained in large-sample conditions with a small-moderate number of drifted anchors using the mINFIT/mOUTFIT methods. However, when any drift was present in small-sample conditions and when a large number of drifted anchors were present in large-sample conditions, all methods performed ineffectively. In the operational-data study, percent-correct standards and failure rates varied across the methods in the large-sample exam but not in the small-sample exam. Different recommendations for high- and low-volume testing programs are provided.

Item Parameter Drift 분석을 통한 진로성숙도 척도의 종단적 측정불변성 검증

The purpose of this study is to examine longitudinal measurement invariance of a career maturity scale for middle school students through an item parameter drift (IPD) analysis. Data obtained from the middle school panel participating in the Daegu Education Longitudinal Study (DELS) for 3 years were analyzed. Career maturity was measured with polytomous items, and IPD items were detected for the first and the second grade and for the second and the third grade using the likelihood ratio test based on the graded response model. As a result of the IPD analysis, it was found that IPD was detected for 3 items (items 1, 2, and 10) between the first and the second grade and for 4 items (items 1, 2, 6, and 8) between the second and the third grade. Among them, items 1 and 2, which were consecutively flagged as IPD, are self-understanding questions, and the average score of self-understanding items gradually decreased although the step parameters of those items became smaller. As a result, it was found that the IPD items caused a large decrease in the average item scores of career maturity. This suggests that if the IPD items are used without testing longitudinal measurement invariance, the difference may look larger than what it actually is.

The study of the effect of item parameter drift on ability estimation obtained from adaptive testing under different conditions

Item parameter drift (IPD) is the systematic differentiation of parameter values of items over time due to various reasons. If it occurs in computer adaptive tests (CAT), it causes errors in the estimation of item and ability parameters. Identification of the underlying conditions of this situation in CAT is important for estimating item and ability parameters with minimum error. This study examines the measurement precision of IPD and its impacts on the test information function (TIF) in CAT administrations. This simulation study compares sample size (1000, 5000), IPD size (0.00 logit, 0.50 logit, 0.75 logit, 1.00 logit), percentage of items containing IPD (0%, 5%, 10%, 20%), three time points and item bank size (200, 500, 1000) conditions. To examine the impacts of the conditions on ability estimations; measurement precision, and TIF values were calculated, and factorial analysis of variance (ANOVA) for independent samples was carried out to examine whether there were any differences between estimations in terms of these factors. The study found that an increase in the number of measurements using item bank with IPD items results in a decrease in measurement precision and the amount of information the test provides. Factorial ANOVA for independent samples revealed that measurements precision and TIF differences are mostly statistically significant. Although all IPD conditions negatively affect measurement precision and TIF, it has been shown that sample size and item bank size generally do not have an increasing or decreasing effect on these factors.

Effect of Item Parameter Drift in Mixed Format Common Items on Test Equating

The aim of the study was to examine the common items in the mixed format (e.g., multiple-choices and essay items) contain parameter drifts in the test equating processes performed with the common item non-equivalent groups design. In this study, which was carried out using Monte Carlo simulation with a fully crossed design, the factors of test length (30 and 50), sample size (1000 and 3000), common item ratio (30 and 40%), ratio of items with item parameter drift (IPD) in common items (20 and 30%), location of common items in tests (at the beginning, randomly distributed, and at the end) and IPD size in multiple-choice items (low [0.2] and high [1.0]) were studied. Four test forms were created, and two test forms do not contain parameter drifts. After the parameter drift was performed on the first of the other two test forms, the parameter drift was again performed on the second test form. Test equating results were compared using the root mean squared error (RMSE) value. As a result of the research, ratio of items with IPD in common items, IPD size in multiple-choice items, common item ratio, sample size and test length on equating errors were found to be significant.

Impact of Item Parameter Drift on Rasch Scale Stability in Small Samples over Multiple Administrations

ABSTRACTTraditional psychometric guidelines suggest that at least several hundred respondents are needed to obtain accurate parameter estimates under the Rasch model. However, recent research indicates that Rasch equating results in accurate parameter estimates with sample sizes as small as 25. Item parameter drift under the Rasch model has been previously investigated, but not under small sample conditions. The current study simulated data using the Rasch model to investigate the impact of varying proportions, magnitudes, and directions of item drift across multiple administrations with sample sizes of 10, 25 and 50. High proportions of high magnitude drift strongly affected bias, RMSE, and classification accuracy. High proportions of positive drift (items becoming harder across administrations) were particularly problematic for classification accuracy, with accuracy falling to less than 60% in the most extreme conditions. Results indicate that item drift affects Rasch examinee ability estimates similarly under both small and large sample conditions, and even small to moderate degrees of unaddressed item drift can substantially affect the validity of scores and pass-fail decisions.

Item Parameter Drift of the Self-Description Questionnaire I

Abstract. The purpose of this study was to evaluate the psychometric properties of the Self-Description Questionnaire I (SDQI) with regard to children’s mathematics self-concept based on the item response theory (IRT) framework. The researchers adopted the IRT-based techniques for uncovering item-level characteristics including discrimination ( a parameter) and thresholds ( b parameters). The data for this study were drawn from the Early Childhood Longitudinal Study, Kindergarten Class of 1998–99 (ECLS-K) with a sample of 14,631 children. The IRT-based evaluations suggest that the SDQI items have sufficient discrimination, but insufficient item thresholds in assessing third- and fifth-graders’ mathematics self-concept. Thus, it is recommended that the current SDQI be reconsidered for its age appropriateness. In addition, affective items, which involve enjoyment, liking, and positive emotions, are found to be better indicators of self-concept than cognitive items, which involve perceived competence, ability beliefs, and self-efficacy. Results also indicate the presence of item parameter drift (IPD) in all SDQI items, suggesting beta change of the scale over time – children become more sensitive to the response anchors as they grow older. Findings of this study provide insights into the conceptualization and operationalization of academic self-concept, as well as practical implications for future use and development of self-concept measures.

Evaluating CAT-adjusted Approaches for Suspected Item Parameter Drift Detection

ABSTRACTEvents such as curriculum changes or practice effects can lead to item parameter drift (IPD) in computer adaptive testing (CAT). The current investigation introduced a point- and weight-adjusted D2 method for IPD detection for use in a CAT environment when items are suspected of drifting across test administrations. Type I error and power rates of the proposed method were compared to a more traditional, non-adjusted D2 method and two recently suggested methods for use in a CAT environment: pseudo-count robust z and pseudo-count D2 methods. Though all CAT-adjusted IPD detection methods compared resulted in high power, the pseudo-count D2 method was found to have the highest power rates, with the proposed D2 method a close second. All four methods were found to have acceptable type I error rates.

Item Parameter Drift in Context Questionnaires from International Large-Scale Assessments

The purpose of the current study was to examine the impact of item parameter drift (IPD) occurring in context questionnaires from an international large-scale assessment and determine the most appropriate way to address IPD. Focusing on the context of psychometric and educational research where scores from context questionnaires composed of polytomous items were employed for the classification of examinees, the current research investigated the impacts of IPD on the estimation of questionnaire scores and classification accuracy with five manipulated factors: the length of a questionnaire, the proportion of items exhibiting IPD, the direction and magnitude of IPD, and three decisions about IPD. The results indicated that the impact of IPD occurring in a short context questionnaire on the accuracy of score estimation and classification of examinees was substantial. The accuracy in classification considerably decreased especially at the lowest and highest categories of a trait. Unlike the recommendation from literature in educational testing, the current study demonstrated that keeping items exhibiting IPD and removing them only for transformation were appropriate when IPD occurred in relatively short context questionnaires. Using 2011 TIMSS data from Iran, an applied example demonstrated the application of provided guidance in making appropriate decisions about IPD.

Item Parameter Drift in Computer Adaptive Testing Due to Lack of Content Knowledge

This study was conducted to analyze potential item parameter drift (IPD) impact on person ability estimates and classification accuracy when drift affects an examinee subgroup. Using a series of simulations, three factors were manipulated: (a) percentage of IPD items in the CAT exam, (b) percentage of examinees affected by IPD, and (c) item pool targeting. IPD impact on ability estimation was evaluated using bias and root mean square error (RMSE). A multivariate general linear model treating three factors as fixed effects and bias and RMSE as outcome variables are used to investigate both main and interaction effects. The impact on classification accuracy was examined based on number and percentages of misclassifications, their significance, and rank order correlations. The findings revealed that IPD exposed to a sub-group of examinees can affect classification accuracy of those examinees substantially, but IPD impact on average ability estimation was small. The study provides useful information to schools, states, and countries planning to implement or are currently implementing CAT as part of their assessments by emphasizing IPD in subgroups may violate fairness and consistency principles of CAT applications.

Internal Structure and Item Characteristics of the Phonological Awareness Literacy Screening in Spanish for Preschool

We used multidimensional item response theory to test the internal structure of the Phonological Awareness Literacy Screening in Spanish for Preschool and test for item parameter drift. The measure is aligned with the simple view of reading, which defines reading as consisting of two equally important dimensions: decoding and language comprehension. It involves 134 items grouped into nine different tasks. We administered a pilot version of the measure to 677 students in 2014 and a final version to 968 students in 2015. We did not find evidence that code-related skills and oral language are distinct but correlated dimensions among preschoolers. Rather, a single general dimension of early literacy with task-related specific dimensions fit best. A benefit of our modeling approach is that we can examine the internal structure while also evaluating the difficulty and discrimination of individual items. Results suggest a stable ordering of items within tasks and that some Spanish letters are learned more easily than others.

On the Cover

Educational Measurement: Issues and PracticeVolume 37, Issue 1 p. 3-3 Data VisualizationFree Access On the Cover First published: 25 March 2018 https://doi.org/10.1111/emip.12187AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Featured on the cover is the final winning submission from the 2017 EM:IP Cover Graphic/Data Visualization competition. It is entitled Item Parameter Drift Plot for a Computer Adaptive Test and it was submitted by Yu-Feng Chang at the Minnesota Department of Education and her colleagues from Pearson, Changjiang Wang, Kevin J. Cappaert, and Gerald W. Griph. An initial glance at this graphic makes it readily apparent whether a given item is experiencing drift, as evidenced by a gap between the observed and expected item characteristic curves. But this is not what sets this graphic apart and qualified it as a winning submission. Rather, it was the way it streamlines information regarding item parameter drift, demonstrating its potential utility as a tool for psychometricians and educational measurement specialists. Often, when tasked with creating a data visualization, we strive for a visual or graphic where less is more, while at the same time meeting the goal of presenting data in a manner that readily conveys the purpose of the visual to the audience. It is also the case that good visuals also serve as useful tools for psychometricians (and other researchers)—helping to inform their test-related decisions. Many researchers report that they understand the data more fully and are better able to make decisions when all the relevant information is laid out in front of them. This is exactly what this graphic does. For a better understanding of this graphic, Chang and colleagues provided the following description: Item parameter drift (IPD) analysis is a key process to ensure items in a testing program perform equivalently across different administrations. Oftentimes, psychometricians use visual inspection of the drift plots in addition to various statistical criteria to evaluate the amount of drift in a test item. In a computer adaptive test (CAT) items are administered adaptively according to students’ abilities so the items are administered to a more homogeneous group of students of which the number of item administrations vary.The plots for two 3-PL items are presented, one for an item with minimal drift (left) and the other for an item with considerable drift (right). In each plot the observed and expected item characteristic curves (ICC) are presented. The observed ICC, in red, is obtained using the Stone (2000) pseudo-count approach while the expected ICC, in green, is generated using the item parameters. In addition to the two ICCs, circles are plotted along the observed ICC to indicate the number of students who take the test item at each ability level, which allows the psychometrician to evaluate the impact of the IPD on students who are administered a given item.Multiple sources of auxiliary information are provided on the top panel. These include the item type (e.g., multiple choice), the year the item was first field tested (i.e., age of the item), the item parameters, number of students taking the item, and the historical item exposure rates. In the lower right corner of the plots, information from statistical IPD analyses is presented, which includes the robust Z, d-squared, and the chi-square values. The items flagged by statistical criteria are asterisked (see the right plot). The plots, as presented here, are generated for the 3-PL IRT model but similar plots can also be generated for other IRT models. If you are interested in learning more about this informative data visualization, contact the principal author Yu-Feng Chang (yu-feng.chang@state.mn.us). We want to hear your feedback! Let us know what you think by emailing Ally Shay Thomas (allythomas@pitt.edu). Acknowledgment The authors would like to acknowledge George Henly and Johnny Denbleyker for their contribution to this project. Supporting Information Filename Description emip12187-sup-0001-SuppMat.zip1.1 MB Additional Supporting Information, including the R code, may be found in the online version of this article at the publisher's website: http://onlinelibrary.wiley.com/journal/10.1111/(ISSN)1745-3992. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article. Reference Stone, C. A. (2000). Monte Carlo-based null distribution for an alternative goodness-of-fit test statistic in IRT models. Journal of Educational Measurement, 37, 58– 75. Volume37, Issue1Special Issue: Strengthening the Connections between Classroom and Large-Scale AssessmentsSpring 2018Pages 3-3 ReferencesRelatedInformation

Item Parameter Drift in a Time-Varying Predictor

ABSTRACTThe current simulation study examined the effects of Item Parameter Drift (IPD) occurring in a short scale on parameter estimates in multilevel models where scores from a scale were employed as a time-varying predictor to account for outcome scores. Five factors, including three decisions about IPD, were considered for simulation conditions. It was revealed that IPD occurring in a relatively shorter scale led to a substantial increase in the amount of relative bias in parameter estimates. The bias was more prominent in the estimates of level-2 time-varying predictors relative to those of level-1 time-varying predictors. Regarding the decisions about IPD, keeping items exhibiting IPD was more appropriate than removing them based on the results from relative bias of standard errors of estimates. Based on the findings, it can be concluded that removing items exhibiting IPD may lead to an increase of Type II errors due to the underestimation of parameter estimates and overestimation of standard errors. The applied example showed findings consistent with those in the simulation study.

The Consequences of Ignoring Item Parameter Drift in Longitudinal Item Response Models

ABSTRACTUtilizing a longitudinal item response model, this study investigated the effect of item parameter drift (IPD) on item parameters and person scores via a Monte Carlo study. Item parameter recovery was investigated for various IPD patterns in terms of bias and root mean-square error (RMSE), and percentage of time the 95% confidence interval covered the true parameter. The simulation results suggest that item parameters were not recovered well when IPD was ignored, especially if there was a larger number of IPD conditions. In addition, coverage was not accurate in all IPD conditions when IPD is ignored. Also, the results suggest that the accuracy of person scores (measured by bias) is potentially problematic when the larger number of IPD items is ignored. However, the overall accuracy (measured by RMSE) and coverage were unexpectedly acceptable in the presence of IPD as defined in this study.

Standard Errors for National Trends in International Large-Scale Assessments in the Case of Cross-National Differential Item Functioning

ABSTRACTStandard errors computed according to the operational practices of international large-scale assessment studies such as the Programme for International Student Assessment’s (PISA) or the Trends in International Mathematics and Science Study (TIMSS) may be biased when cross-national differential item functioning (DIF) and item parameter drift are present. This bias may be somewhat reduced when cross-national DIF is correlated over study cycles, which is the case in PISA. This article reviews existing methods for calculating standard errors for national trends in international large-scale assessments and proposes a new method that takes into account the dependency of linking errors at different time points. We conducted a simulation study to compare the performance of the standard error estimators. The results showed that the newly suggested estimator outperformed the existing estimators as it estimated standard errors more accurately and efficiently across all simulated conditions. Implications for practical applications are discussed.

Investigating Separate and Concurrent Approaches for Item Parameter Drift in 3PL Item Response Theory Equating

This study examines separate and concurrent approaches to combine the detection of item parameter drift (IPD) and the estimation of scale transformation coefficients in the context of the common item nonequivalent groups design with the three-parameter item response theory equating. The study uses real and synthetic data sets to compare the two approaches based on IPD flagging rates, type I error and power rates, and recovery of scale transformation coefficients. Results indicate that the two approaches render similar outcomes with stable anchor sets. However, they can produce dissimilar results with unstable anchor sets because of differences in the performance of their IPD components. Further, the findings of this study caution about working backward from equated cut scores to motivate the selection of an anchor set.

Psychometric Consequences of Subpopulation Item Parameter Drift.

This study defines subpopulation item parameter drift (SIPD) as a change in item parameters over time that is dependent on subpopulations of examinees, and hypothesizes that the presence of SIPD in anchor items is associated with bias and/or lack of invariance in three psychometric outcomes. Results show that SIPD in anchor items is associated with a lack of invariance in dimensionality structure of an anchor test, a lack of invariance in scaling coefficients across subpopulations, and a lack of invariance in ability estimates. It is demonstrated that these effects go beyond what can be understood from item parameter drift or differential item functioning.

Investigating the Impact of Item Parameter Drift for Item Response Theory Models with Mixture Distributions.

This study investigates the impact of item parameter drift (IPD) on parameter and ability estimation when the underlying measurement model fits a mixture distribution, thereby violating the item invariance property of unidimensional item response theory (IRT) models. An empirical study was conducted to demonstrate the occurrence of both IPD and an underlying mixture distribution using real-world data. Twenty-one trended anchor items from the 1999, 2003, and 2007 administrations of Trends in International Mathematics and Science Study (TIMSS) were analyzed using unidimensional and mixture IRT models. TIMSS treats trended anchor items as invariant over testing administrations and uses pre-calibrated item parameters based on unidimensional IRT. However, empirical results showed evidence of two latent subgroups with IPD. Results also showed changes in the distribution of examinee ability between latent classes over the three administrations. A simulation study was conducted to examine the impact of IPD on the estimation of ability and item parameters, when data have underlying mixture distributions. Simulations used data generated from a mixture IRT model and estimated using unidimensional IRT. Results showed that data reflecting IPD using mixture IRT model led to IPD in the unidimensional IRT model. Changes in the distribution of examinee ability also affected item parameters. Moreover, drift with respect to item discrimination and distribution of examinee ability affected estimates of examinee ability. These findings demonstrate the need to caution and evaluate IPD using a mixture IRT framework to understand its effects on item parameters and examinee ability.