Item Parameter Drift Research Articles

A Stepwise Test Characteristic Curve Method to Detect Item Parameter Drift

An important assumption of item response theory is item parameter invariance. Sometimes, however, item parameters are not invariant across different test administrations due to factors other than sampling error; this phenomenon is termed item parameter drift. Several methods have been developed to detect drifted items. However, most of the existing methods were designed to detect drifts in individual items, which may not be adequate for test characteristic curve–based linking or equating. One example is the item response theory–based true score equating, whose goal is to generate a conversion table to relate number-correct scores on two forms based on their test characteristic curves. This article introduces a stepwise test characteristic curve method to detect item parameter drift iteratively based on test characteristic curves without needing to set any predetermined critical values. Comparisons are made between the proposed method and two existing methods under the three-parameter logistic item response model through simulation and real data analysis. Results show that the proposed method produces a small difference in test characteristic curves between administrations, an accurate conversion table, and a good classification of drifted and nondrifted items and at the same time keeps a large amount of linking items.

Modified Robust Z method for equating and detecting item parameter drift

This study is aimed at: (1) revising the criterion used in Robust Z Method for detecting Item Parameter Drift (IPD), (2) identifying the strengths and weaknesses of the modified Robust Z Method, and (3) investigating the effect of IPD on examinees’ classification consistency using empirical data. This study used two types of data. The simulated data were in the form of responses of 20,000 students on 40 dichotomous items generated by simulating six variables including: (1) ability distribution, (2) differences of groups’ ability between groups, (3) type of drifting, (4) magnitude of drifting, (5) anchor test length, and (6) number of drifting items. The empirical data was 4,187,444 students’ response of UN SD/MI 2011 who administered 41 test forms of Indonesian language, mathematics, and science. Modified Robust Z method was used to detect IPD and the IRT true score equating method was used to analyze the classification consistency. The results of this study show that: (1) the criterion of 0.5 point raw score TCC difference leads to 100% consistency on passing classification, (2) the modified Robust Z is accurate to detect the b and ab- drifting when the minimal length of anchor test is 25%, (3) IPD occurring on empirical data affected the passing status of more than 2,000 students.

An Examination of Two Procedures for Identifying Consequential Item Parameter Drift

The purpose of the present study was to develop and evaluate two procedures flagging consequential item parameter drift (IPD) in an operational testing program. The first procedure was based on flagging items that exhibit a meaningful magnitude of IPD using a critical value that was defined to represent barely tolerable IPD. The second procedure was based on flagging items in which the D2 statistic was more than two standard deviations from the mean. Both procedures were implemented using an iterative purification approach to detect IPD. A simulation study was implemented to evaluate the effectiveness of both detection procedures in flagging non-negligible IPD. Both procedures were able to identify IPD and the iterative purification method provided useful information regarding the consequences of excluding or including a flagged item. The advantages and disadvantages of both procedures as well as possible modifications intended to improve the procedures’ effectiveness are discussed in the article.

Equating the HBSC Family Affluence Scale across survey years: a method to account for item parameter drift using the Rasch model.

To investigate the measurement invariance (MI) of the Family Affluence Scale (FAS) measured in the Health Behavior in School-aged Children (HBSC) survey, and to describe a method for equating the scale when MI is violated across survey years. This study used a sample of 14,076 Norwegian and 17,365 Scottish adolescents from the 2002, 2006 and 2010 HBSC surveys to investigate the MI of the FAS across survey years. Violations of MI in the form of differential item functioning (DIF) due to item parameter drift (IPD) were modeled within the Rasch framework to ensure that the FAS scores from different survey years remain comparable. The results indicate that the FAS is upwardly biased due to IPD in the computer item across survey years in the Norwegian and Scottish samples. Ignoring IPD across survey years resulted in the conclusion that family affluence is increasing quite consistently in Norway and Scotland. However, the results show that a large part of the increase in the FAS scores can be attributed to bias in the FAS because of IPD across time. The increase in the FAS was more modest in Scotland and slightly negative in Norway once the DIF in the computer item was accounted for in this study. When the comparison of family affluence is necessary over different HBSC survey years or when the longitudinal implications of family affluence are of interest, it is necessary to account for IPD in interpretation of changes in family affluence across time.

Effects of Item Parameter Drift on Vertical Scaling With the Nonequivalent Groups With Anchor Test (NEAT) Design

The authors explored the effects of drifting common items on vertical scaling within the higher order framework of item parameter drift (IPD). The results showed that if IPD occurred between a pair of test levels, the scaling performance started to deviate from the ideal state, as indicated by bias of scaling. When there were two items drifting with 0.5 logits, IPD could have a substantial effect on vertical scaling. Although IPD had little impact on the recoveries of the parameters on the whole developmental scale, its effects on the recoveries of some parameters by separate grade and grade pair were manifest. Specifically, the mean achievement estimates became worse conditional on the pair of test levels between which IPD occurred, and the estimations of grade-to-grade growth and effect size were distorted for the grade pair corresponding to the test pair that involved IPD. Neither the estimation of standard deviation of the achievement nor the grade-to-grade variability was influenced by IPD.

EXAMINING THE IMPACT OF DRIFTED POLYTOMOUS ANCHOR ITEMS ON TEST CHARACTERISTIC CURVE (TCC) LINKING AND IRT TRUE SCORE EQUATING

ABSTRACTIn a common‐item (anchor) equating design, the common items should be evaluated for item parameter drift. Drifted items are often removed. For a test that contains mostly dichotomous items and only a small number of polytomous items, removing some drifted polytomous anchor items may result in anchor sets that no longer resemble mini‐versions of the new and old test forms. In this study, the impact of drifted polytomous anchor items on the test characteristic curve (TCC) linking and item response theory (IRT) true score equating for a test containing only a small number of polytomous items was investigated. Simulated tests were constructed to mimic a real large‐scale test. The magnitude of the item parameter drift, anchor length, number of drifted polytomous items in the anchor set, and the ability distributions of the groups taking the old form and new form were manipulated. Results suggest that anchor length and number of drifted polytomous items had a relatively large impact on the linking and equating results. The accuracy of linking and equating results were affected by the magnitude of item parameter drift. The ability distributions of the groups had little effect on the linking and equating results. In general, excluding drifted polytomous anchor items resulted in an improvement in equating results.

The Impact of Multidirectional Item Parameter Drift on IRT Scaling Coefficients and Proficiency Estimates

Item parameter drift (IPD) occurs when item parameter values change from their original value over time. IPD may pose a serious threat to the fairness and validity of test score interpretations, especially when the goal of the assessment is to measure growth or improvement. In this study, we examined the effect of multidirectional IPD (i.e., some items become harder while other items become easier) on the linking procedure and rescaled proficiency estimates. The impact of different combinations of linking items with various multidirectional IPD on the test equating procedure was investigated for three scaling methods (mean-mean, mean-sigma, and TCC method) via a series of simulation studies. It was observed that multidirectional IPD had a substantive effect on examinees' scores and achievement level classifications under some of the studied conditions. Choice of linking method had a direct effect on the results, as did the pattern of IPD.

Expected Equating Error Resulting From Incorrect Handling of Item Parameter Drift Among the Common Items

Incorrect handling of item parameter drift during the equating process can result in equating error. If the item parameter drift is due to construct-irrelevant factors, then inclusion of these items in the estimation of the equating constants can be expected to result in equating error. On the other hand, if the item parameter drift is related to the construct being measured, then removal of those items from the estimation of the equating constants can be expected to result in equating error. The effect of incorrect handling of item parameter drift on equating has been examined via simulation studies and other methods by a number of authors with mixed results. In this article, this effect is assessed by estimating the expected equating error under the assumptions that the three-parameter item response theory model is used for calibration—equating.

Applying Multilevel Item Response Theory to Vision-Related Quality of Life in Dutch Visually Impaired Elderly

Instead of applying the usual longitudinal methods to assess the outcome of low-vision rehabilitation services in terms of vision-related quality of life, a three-level Item Response Theory (IRT) method was proposed. The translated Vision-Related Quality of Life Core Measure (VCM1) and Low Vision Quality Of Life (LVQOL) questionnaires were used in a nonrandomized follow-up study among elderly patients (n = 296) referred to two different low-vision rehabilitation services in the Netherlands. Factor analysis was performed on the matrix of polychoric correlations to investigate (uni-)dimensionality and to prepare both questionnaires for the multilevel IRT analyses. A statistical model, which was characterized by a graded response model for rating scales, was developed. Threshold and item difficulty parameters and group by time-specific mean fixed effects were estimated. Random individual effects were predicted. Measurement invariance across occasions was tested. The VCM1 and the LVQOL "reading and fine work" dimension showed item parameter drift. In the multidisciplinary rehabilitation center patients, deterioration was found on the "mobility" dimension after 1 year and improvement was found on "adjustment" and "visual (motor) skills" after 5 months (p < 0.05). Patients in both low-vision services showed improvement on the "reading small print" subscale at both follow-up time points (p < 0.05). Except for improvement in "reading small print," low-vision rehabilitation services did not seem to contribute substantially to any other dimensions of vision-related quality of life. The results showed a change in only a limited number of individual patients. However, with regard to the field of low-vision rehabilitation, the proposed IRT method seemed to be successful in the follow-up of individuals. IRT specific software was unnecessary. The data did not have to be complete and the use of cumulative logits made the proposed IRT method an economical and efficient approach. Because of item parameter drift, the VCM1 was difficult to interpret. The use of multilevel IRT models with longitudinal data and dependent observations is recommended.

Understanding Parameter Invariance in Unidimensional IRT Models

One theoretical feature that makes item response theory (IRT) models those of choice for many psychometric data analysts is parameter invariance, the equality of item and examinee parameters from different examinee populations or measurement conditions. In this article, using the well-known fact that item and examinee parameters are identical only up to a set of linear transformations specific to the functional form of a given IRT model, violations of these transformations for unidimensional IRT models are investigated using analytical, numerical, and visual tools. Because item parameter drift (IPD) constitutes a lack of invariance (LOI) at the individual item level or item set level, the magnitudes and effects of IPD on examinee response probabilities and true scores are algebraically derived and connected to empirical results from a recent simulation study. Thus, this article facilitates a deeper understanding of the exact statistical formulation of parameter invariance as a fundamental property of latent variable measurement and explicates some practical consequences of LOI for decision making.

Temporal Invariance in Repeated Cross-Sectional ‘Fear of Crime’ Research

‘Fear of crime’ research is gradually outgrowing the tenacious criticism of being caught within a conservative methodological frame, inherent to the era when the research tradition originated. Over time, the idea that ‘fear of crime’ can be measured in a reliable and valid way by using a single indicator has been questioned. However, using more advanced measures confronts the research tradition with ‘new’ issues related to measurement error, and therefore urges methodological novelty and sophistication to a field notoriously starved of both. Previously, we provided evidence that in order for cross-cultural comparisons to be meaningful, the instruments used to measure ‘fear of crime’, need to exhibit adequate cross-cultural equivalence. This contribution argues that comparing ‘fear of crime’ levels across time is bound to this logic as well. Considering the growing attention paid in recent years to crime, insecurity and ‘fear of crime’ within public opinion in Belgium, as in Western Europe and in an international context in general, it is not unlikely that respondents have indeed changed or are changing their understanding of the issue. Using structural equation modelling (LISREL), this issue of potential temporal invariance or item parameter drift in ‘fear of crime’ item sets, will be assessed in two large scale, repeated cross-sectional research projects in Belgium, Le. the Safety Monitor and the APS-survey.

Establishing Measurement Equivalence and Invariance in Longitudinal Data With Item Response Theory

If measurement invariance does not hold over 2 or more measurement occasions, differences in observed scores are not directly interpretable. Golembiewski, Billingsley, and Yeager (1976) identified 2 types of psychometric differences over time as beta change and gamma change. Gamma change is a fundamental change in thinking about the nature of a construct over time. Beta change can be described as respondents' change in calibration of the response scale over time. Recently, researchers have had considerable success establishing measurement invariance using confirmatory factor analytic (CFA) techniques. However, the use of item response theory (IRT) techniques for assessing item parameter drift can provide additional useful information regarding the psychometric equivalence of a measure over time that is not attainable with traditional CFA techniques. This article marries the terminology commonly used in CFA and IRT techniques and illustrates real advantages for identifying beta change over time with IRT methods rather than typical CFA methods, utilizing a longitudinal assessment of job satisfaction as an example.

A Method for Maintaining Scale Stability in the Presence of Test Speededness

Administering tests under time constraints may result in poorly estimated item parameters, particularly for items at the end of the test (Douglas, Kim, Habing, &amp; Gao, 1998; Oshima, 1994). Bolt, Cohen, and Wollack (2002) developed an item response theory mixture model to identify a latent group of examinees for whom a test is overly speeded, and found that item parameter estimates for end‐of‐test items in the nonspeeded group were similar to estimates for those same items when administered earlier in the test. In this study, we used the Bolt et al. (2002) method to study the effect of removing speeded examinees on the stability of a score scale over an II‐year period. Results indicated that using only the nonspeeded examinees for equating and estimating item parameters provided a more unidimensional scale, smaller effects of item parameter drift (including fewer drifting items), and less scale drift (i.e., bias) and variability (i.e., root mean squared errors) when compared to the total group of examinees.

The Effect of Item Parameter Drift on Examinee Ability Estimates

This study investigated the effect of item parameter drift on ability estimates under item response theory. Item response data for two testing occasions were simulated for the two-parameter logistic model under the following crossed conditions: test length, sample size, percentage of drifting items, and type of drift. Results indicated that item parameter drift, under the conditions simulated, had a small effect on ability estimates.

A Comparison of Procedures to Detect Item Parameter Drift

Monte carlo methods were used to compare several measures of item parameter drift. Number of examinees, number of items, and number of drift items in the test were manipulated. Overall, Lord's x2 measure was the most effective in identifying items that exhibited drift. However, the measure was accurate only when the studied item's c parameter was constrained to be equal across the two assessment years. Of the remaining measures, the best methods (a z test based on Raju's exact unsigned integral, the NAEP BILOG/PARSCALE computer program's x2 by subgroup, and Kim and Cohen's closed-interval signed-area measure) required empirical estimates of critical values for the test statistics in order to function well. This requirement detracts from their usefulness.

COMPARING THE EFFECTIVENESS OF PROCEDURES TO DETECT ITEM PARAMETER DRIFT

ABSTRACTThis study used Monte Carlo methods to examine the use of several methods to identify item parameter drift. Number of examinees, number of items, and number of drift items in the test were manipulated. Overall, Lord's χ2 measure was the most effective in identifying items that exhibited drift. However, the measure was only accurate when the c‐parameter was constrained to be equal across calibrations. Of the remaining methods, the best (z‐test based on Raju's exact unsigned integral, the PARSCALE computer program's χ2 by subgroup, and Kim and Cohen's closed interval signed area measure) required empirical estimates of critical values for the test statistics in order to function well. This aspect detracts from their usefulness.

Item Pool Maintenance in the Presence of Item Parameter Drift

Differential linear drift of item location parameters over a 10 ‐year period is demonstrated in data from the College Board Physics Achievement Test. The relative direction of drift is associated with the content of the items and reflects changing emphasis in the physics curricula of American secondary schools. No evidence of drift of discriminating power parameters was found. Statistical procedures for detecting, estimating, and accounting for item parameter drift in item pools for long‐term testing programs are proposed