Anchor Item Research Articles

The Clinician-reported Genetic testing Utility InDEx (C-GUIDE) for Prenatal Care: Initial evidence of content and construct validity

ObjectiveTo develop and assess the face and construct validity of the Clinician-reported Genetic testing Utility InDEX (C-GUIDETM) for genetic testing in prenatal care. MethodsFollowing a literature review and consultation with clinical experts, a preliminary draft of C-GUIDE Prenatal was developed. Its face and content validity were then assessed by 19 prenatal genetics’ providers using interviews and surveys. Feedback informed further revisions. To test construct validity, four geneticist raters completed C-GUIDE on a retrospective sample of cases that received prenatal genetic testing and completed a concurrent global assessment of utility of these cases using an anchor item. A generalized estimating equations model was used to adjust for rater correlation and measure the association between C-GUIDE scores, global item scores, and potential clinical variables. ResultsTo develop C-GUIDE Prenatal, 7 items were removed, 10 items were modified, and 4 items were added. For 101 cases rated for validation, on average, a 1-point increase in the global item score was associated with an increase of 1.1 in the C-GUIDE score (p=0.04). Compared to uninformative results, informative positive and informative negative results were associated with a mean increase of 10.7 (SE=1.05) (p<0.001) and 5.6 (SE=1.85) (p<0.001), respectively. As indications for testing, known/familial variants were associated with a mean increase in the C-GUIDE score of 4.7 (SE=2.21) (p < 0.001) compared to ultrasound findings. C-GUIDE scores increased by a mean of 3.0 (SE=0.23) among cases for whom pregnancies were ongoing compared to those for whom they were not (p<0.01). ConclusionsThe significant positive associations between C-GUIDE total and the global item score and between C-GUIDE total, result type, indication for testing, and pregnancy status in the expected directions provide evidence of construct validity.

Establishing minimal clinically important differences for the quality of life scale QLICD-SLE for patients with systemic lupus erythematosus based on ROC curve

The minimal clinically important difference (MCID) is an important concept with big appeal in a field struggling to interpret quality of life (QOL) and other patient-reported outcomes (PRO), is also a bridge between statistics and clinical medicine. This study uses the ROC curve to formulate the MCID value of the Quality of Life Instruments for Chronic Diseases of Systemic lupus erythematosus (QLICD-SLE V2.0) scale. Using the representative item “In general, would you say your health is” of the MOS item short form health survey(SF-36) as an anchor, the questionnaire of QLICD-SLE V2.0 and the anchor item were used to investigate the patients on the first day of hospitalization, and the day before the patient was discharged. 279 patients with lupus erythematosus were participated in this longitudinal follow-up study. The ROC curve was constructed by using the classification based on the anchor item as the gold standard and the difference score of the scale as the test variable. The cut-off point corresponding to the maximum value of the Youden index in the ROC curve is taken as the minimum clinical importance difference (MCID) value of the QLICD-SLE (V2.0) scale. The Results showed that the MCID of physical domain, psychological domain, social domain, general module, specific module and QLICD-SLE (V2.0) total scale are 8.3, 2.3, 2.5, 2.7, 9.2 and 3.2, respectively. Area under the ROC curve of QLICD-SLE (V2.0) is 0.898, P (Area = 0.5) < 0.001, the sensitivity is 100%, the specificity is 66.9%. It concluded that if the total scores after treatments changes at least 3.2 points positively, the treatment intervention can be considered as clinically significant. It is more convincing to use the corresponding cut-off point as the MCID for ROC curve method can visualize the sensitivity and specificity.

혼합형 가교문항 구성 방법이 혼합형 검사 동등화에 미치는 영향

With the introduction of innovative items in the computer-based NAEA, the item format became diversified, and the current method of selecting anchor items from multiple-choice (MC) items became burdensome for test developers. The purpose of this study is to examine whether a mixed-format anchor items including both MC and constructed-response (CR) items could hold the stability of equating in the NAEA, and give desirable direction to expand the range of anchor items and to make test construct flexible. A simulation study was conducted based on a common-item non-equivalent group design with the test conditions of the Korean subject in the NAEA. The treatment conditions include (1) a type of anchor item set (a single MC or a mixed), (2) a composition type of mixed anchor item (MC:CR = 1:1, 3:2, or 3:4), (3) a length of mixed anchor items within a test (25%, 22%, or 19%), and (4) the presence of multidimensionality according to the item format were assumed. The main result is that there was no significant difference in the equating error between the single MC and the mixed anchor item condition. Specifically the equating error of the single MC was smaller than that of the mixed-format cases where the ratios between MC and CR were 1:1 and 3:2. In the 3:4-ratio mixed condition, the equating error was smaller than that of the single MC condition. A length of mixed anchor items within a test showed somewhat mixed results, probably because there was not much variation between the conditions in the composition ratio, about 20% in test. When there was multidimensionality according to the item format in the test, the equating error increased significantly with mixed anchor items compared to that of unidimensional test condition. In sum, about 20% of the mixed anchor items with slightly higher CR ratio was found to be suitable in the unidimensional test and caution was called for in using mixed anchor items in multidimensional test.

Estimating meaningful thresholds for multi-item questionnaires using item response theory

PurposeMeaningful thresholds are needed to interpret patient-reported outcome measure (PROM) results. This paper introduces a new method, based on item response theory (IRT), to estimate such thresholds. The performance of the method is examined in simulated datasets and two real datasets, and compared with other methods.MethodsThe IRT method involves fitting an IRT model to the PROM items and an anchor item indicating the criterion state of interest. The difficulty parameter of the anchor item represents the meaningful threshold on the latent trait. The latent threshold is then linked to the corresponding expected PROM score. We simulated 4500 item response datasets to a 10-item PROM, and an anchor item. The datasets varied with respect to the mean and standard deviation of the latent trait, and the reliability of the anchor item. The real datasets consisted of a depression scale with a clinical depression diagnosis as anchor variable and a pain scale with a patient acceptable symptom state (PASS) question as anchor variable.ResultsThe new IRT method recovered the true thresholds accurately across the simulated datasets. The other methods, except one, produced biased threshold estimates if the state prevalence was smaller or greater than 0.5. The adjusted predictive modeling method matched the new IRT method (also in the real datasets) but showed some residual bias if the prevalence was smaller than 0.3 or greater than 0.7.ConclusionsThe new IRT method perfectly recovers meaningful (interpretational) thresholds for multi-item questionnaires, provided that the data satisfy the assumptions for IRT analysis.

Outlier Detection Using t-test in Rasch IRT Equating under NEAT Design.

In equating practice, the existence of outliers in the anchor items may deteriorate the equating accuracy and threaten the validity of test scores. Therefore, stability of the anchor item performance should be evaluated before conducting equating. This study used simulation to investigate the performance of the t-test method in detecting outliers and compared its performance with other outlier detection methods, including the logit difference method with 0.5 and 0.3 as the cutoff values and the robust z statistic with 2.7 as the cutoff value. The investigated factors included sample size, proportion of outliers, item difficulty drift direction, and group difference. Across all simulated conditions, the t-test method outperformed the other methods in terms of sensitivity of flagging true outliers, bias of the estimated translation constant, and the root mean square error of examinee ability estimates.

Measurement Invariance: Dealing with the Uncertainty in Anchor Item Choice by Model Averaging

ABSTRACT A core challenge in modeling partial measurement invariance (MI) is choosing reference items as anchors for which MI indeed holds. Many approaches dealing with this issue have been proposed, each making a different assumption about MI and yielding a single set of anchor items. Here, we consider the case where i) partial MI modeling is used for estimating effects, e.g., a group mean difference, and ii) there is no straightforward theoretical reason to choose specific items as anchors. We argue that in this situation the uncertainty of anchor item choice should be considered and propose to use model averaging with a priori defined model weights. The approach allows not only to depict uncertainty in the anchor items choice but also allows to include prior knowledge and beliefs of the researcher. We derive the properties of the approach and illustrate its use with an example on the assessment of obsessive-compulsive disorder.

Detection of Outliers in Anchor Items Using Modified Rasch Fit Statistics

ABSTRACT The existence of outliers in the anchor items can be detrimental to the estimation of examinee ability and undermine the validity of score interpretation across forms. However, in practice, anchor item performance can become distorted due to various reasons. This study compares the performance of modified INFIT and OUTFIT Rasch statistics with the Logit Difference approach with 0.3 and 0.5 as the predetermined cutoff values, and the Robust z statistic with 1.645 and 2.7 as the cutoff values through a simulation study by varying the sample size, proportion of outliers, item difficulty drift direction, and group difference magnitude. The results suggest that both modified INFIT and OUTFIT statistics perform very similarly and outperform the other methods in all aspects, including sensitivity of flagging outliers, specificity of flagging non-outliers, recovery of translation constant, and recovery of examinee ability in all simulated conditions.

Advantages of Spike and Slab Priors for Detecting Differential Item Functioning Relative to Other Bayesian Regularizing Priors and Frequentist Lasso

ABSTRACT An important step in scale development and assessment is to evaluate differential item functioning (DIF) across segments of the population. Recent approaches use lasso regularization to simultaneously detect DIF in all items and avoid incorrect anchor item assumptions that incur inflated error rates for classical DIF evaluation methods. Although promising, lasso methods cause underestimated standard errors and incorrect p-values. An alternative is Bayesian regularization that provides empirical standard errors. However, we point out that using empirical criteria such as credible intervals for selecting DIF parameters has limited validity. We argue that using a spike-and-slab prior with an inclusion probability criterion provides more theoretically coherent DIF selection and inference over Bayesian regularizing priors with empirical selection rules or frequentist lasso. We demonstrate this by simulation studies with Multi-group Item Response Theory and Moderated Nonlinear Factor Analysis models. Practical utility of the spike-and-slab prior selection criterion is discussed.

Differential Item Functioning Analyses of the Patient-Reported Outcomes Measurement Information System (PROMIS®) Measures: Methods, Challenges, Advances, and Future Directions.

Several methods used to examine differential item functioning (DIF) in Patient-Reported Outcomes Measurement Information System (PROMIS®) measures are presented, including effect size estimation. A summary of factors that may affect DIF detection and challenges encountered in PROMIS DIF analyses, e.g., anchor item selection, is provided. An issue in PROMIS was the potential for inadequately modeled multidimensionality to result in false DIF detection. Section 1 is a presentation of the unidimensional models used by most PROMIS investigators for DIF detection, as well as their multidimensional expansions. Section 2 is an illustration that builds on previous unidimensional analyses of depression and anxiety short-forms to examine DIF detection using a multidimensional item response theory (MIRT) model. The Item Response Theory-Log-likelihood Ratio Test (IRT-LRT) method was used for a real data illustration with gender as the grouping variable. The IRT-LRT DIF detection method is a flexible approach to handle group differences in trait distributions, known as impact in the DIF literature, and was studied with both real data and in simulations to compare the performance of the IRT-LRT method within the unidimensional IRT (UIRT) and MIRT contexts. Additionally, different effect size measures were compared for the data presented in Section 2. A finding from the real data illustration was that using the IRT-LRT method within a MIRT context resulted in more flagged items as compared to using the IRT-LRT method within a UIRT context. The simulations provided some evidence that while unidimensional and multidimensional approaches were similar in terms of Type I error rates, power for DIF detection was greater for the multidimensional approach. Effect size measures presented in Section1 and applied in Section2 varied in terms of estimation methods, choice of density function, methods of equating, and anchor item selection. Despite these differences, there was considerable consistency in results, especially for the items showing the largest values. Future work is needed to examine DIF detection in the context of polytomous, multidimensional data. PROMIS standards included incorporation of effect size measures in determining salient DIF. Integrated methods for examining effect size measures in the context of IRT-based DIF detection procedures are still in early stages of development.

Linking of Rasch-Scaled Tests: Consequences of Limited Item Pools and Model Misfit

In the context of item response theory (IRT), linking the scales of two measurement points is a prerequisite to examine a change in competence over time. In educational large-scale assessments, non-identical test forms sharing a number of anchor-items are frequently scaled and linked using two− or three-parametric item response models. However, if item pools are limited and/or sample sizes are small to medium, the sparser Rasch model is a suitable alternative regarding the precision of parameter estimation. As the Rasch model implies stricter assumptions about the response process, a violation of these assumptions may manifest as model misfit in form of item discrimination parameters empirically deviating from their fixed value of one. The present simulation study investigated the performance of four IRT linking methods—fixed parameter calibration, mean/mean linking, weighted mean/mean linking, and concurrent calibration—applied to Rasch-scaled data with a small item pool. Moreover, the number of anchor items required in the absence/presence of moderate model misfit was investigated in small to medium sample sizes. Effects on the link outcome were operationalized as bias, relative bias, and root mean square error of the estimated sample mean and variance of the latent variable. In the light of this limited context, concurrent calibration had substantial convergence issues, while the other methods resulted in an overall satisfying and similar parameter recovery—even in the presence of moderate model misfit. Our findings suggest that in case of model misfit, the share of anchor items should exceed 20% as is currently proposed in the literature. Future studies should further investigate the effects of anchor item composition regarding unbalanced model misfit.

Anchors Aweigh: How the Choice of Anchor Items Affects the Vertical Scaling of 3PL Data with the Rasch Model

ABSTRACT Vertical scaling is used to put tests of different difficulty onto a common metric. The Rasch model is often used to perform vertical scaling, despite its strict functional form. Few, if any, studies have examined anchor item choice when using the Rasch model to vertically scale data that do not fit the model. The purpose of this study was to investigate the implications of anchor item choice on bias in growth estimates when data do not fit the Rasch model. Data were generated with varying levels of true difference between grades and levels of the lower asymptote. When true growth or the lower asymptote were zero, estimates were unbiased and anchor item choice was not consequential. As true growth and the lower asymptote both increased, growth was underestimated and choice of anchor items had an impact. Easy anchor items led to less biased estimates of growth than hard anchor items.

Detecting Differential Item Functioning Using Multiple-Group Cognitive Diagnosis Models.

This study proposes a multiple-group cognitive diagnosis model to account for the fact that students in different groups may use distinct attributes or use the same attributes but in different manners (e.g., conjunctive, disjunctive, and compensatory) to solve problems. Based on the proposed model, this study systematically investigates the performance of the likelihood ratio (LR) test and Wald test in detecting differential item functioning (DIF). A forward anchor item search procedure was also proposed to identify a set of anchor items with invariant item parameters across groups. Results showed that the LR and Wald tests with the forward anchor item search algorithm produced better calibrated Type I error rates than the ordinary LR and Wald tests, especially when items were of low quality. A set of real data were also analyzed to illustrate the use of these DIF detection procedures.

Responsiveness to Change Over Time: An Examination of the Neuro-QoL Social Function Measures in Persons with Huntington's Disease.

Social health is an important concern in persons with Huntington's disease (HD); however, there is little literature examining this construct in this population. While cross-sectional data supports the clinical utility of two Neuro-QoL social health measures in persons with HD, data is still needed to establish their longitudinal validity. Participants (N = 358) completed baseline and at least one follow-up (12- and 24-month) assessment that included the completion of Neuro-QoL Social Health computer adaptive tests (CATs) and short forms (for Ability to Participate in Social Roles and Activities [SRA] and Satisfaction with SRA). Test-retest reliability was examined using intra class correlations, and one-way ANOVAs with Bonferroni post-hoc contrasts were used to determine whether there were group differences among premanifest, early- and late-stage HD participants on the Social health measures. In addition, standardized response means were used to examine longitudinal responsiveness, and mixed or general linear models were used to examine change over time (relative to self-reported change on an associated anchor item about social health and clinician-rated change based on Total Functional Capacity scores from the UHDRS). Test-retest reliability of the measures was excellent (ICCs ranged from 0.82 to 0.87 across the different measures) and persons with greater disease burden reported more problems with social health than those at earlier stages in the disease process (all p < 0.0001). Responsiveness was supported for all measures except the Ability to Participate in SRA CAT; participants who had self-reported or clinician-rated declines in health generally had 12- and 24-month declines on the Neuro-QoL measures. Findings indicate that these measures may be useful for studies attempting to assess change in social health over time.

Investigating the Classification Accuracy of Rasch and Nominal Weights Mean Equating with Very Small Samples

ABSTRACTMaintaining equivalent performance standards across forms is a psychometric challenge exacerbated by small samples. In this study, the accuracy of two equating methods (Rasch anchored calibration and nominal weights mean) and four anchor item selection methods were investigated in the context of very small samples (N = 10). Overall, nominal weights mean equating slightly outperformed Rasch equating for three of the four anchor item selection methods, but Rasch equating slightly outperformed nominal weights mean equating when anchor items were selected to be near the cut score. The results largely confirmed previous research on the utility of nominal weights mean equating for very small samples. In addition, the results provide useful guidance for small volume programs who wish to consider using Rasch for building and equating new forms. Lastly, the results underscored the importance of being mindful about the method for selecting anchor items when building new forms.

The Effect of Mini and Midi Anchor Tests on Test Equating

The main purpose of this study is to compare the test forms to the midi anchor test and the mini anchor test performance based on item response theory. The research was conducted with using simulated data which were generated based on Rasch model. In order to equate two test forms the anchor item nonequivalent groups (internal anchor test) was used in this research. The data were generated and analyzed using the R software. 100 replications were done for each different condition. The results obtained from this simulation study were evaluated according to the equating error (RMSE) and bias (BIAS) criterions. In the study, midi anchor generally produced better equating results than the mini anchor test with a few exceptions for most conditions.

Agreement on the Classification of Latent Class Membership Between Different Identification Constraint Approaches in the Mixture Rasch Model

Abstract. When using the mixture Rasch model, the model identification constraints are either to set the equal means for all classes in the assumed normal ability distributions (equal ability mean constraint in short), or to set the sum of item difficulties to be zero for each class. In real data analysis, however, both constraints are not always sufficient to establish a common scale across latent classes unless some items are specified as anchor items in the estimation. If these two conventional constraint approaches recover the class membership as good as the anchor item constraint approach, the conventional constraint approaches may be considered useful for the purpose of class membership classification. This study investigated agreement on class membership between one conventional constraint (the equal ability mean) and the anchor item constraint approaches. Results showed high agreement between these two constraint approaches, indicating that the conventional constraint of the equal mean ability approach may be used to recover the latent class membership although item profiles are not correctly estimated across latent classes.

Local Lecturers, International Students and Their Linguistic Compatibility

The aim of this study is to compare the linguistic proficiency levels of lecturers and students, using an online lecture passage dictation as an anchor item between the two tests developed for students and lecturers. The article also reports on the use of the said dictation as a means of the assessment justification (Bachman, 2013) of the proficiency test offered via at a University in Latvia Moodle (Modular Object-Oriented Dynamic Learning Environment) system (Online 1), making the test available in different parts of the world. The main finding is that the lecturers and their students’ language proficiency is comparable when using Gary Buck’s (2001) and Common European Framework of Reference (2006) frameworks of phonological, orthographic, lexical, syntactical and discourse competence levels.

Examining the Impact of Covariates on Anchor Tests to Ascertain Quality Over Time in a College Admissions Test

We propose a comprehensive procedure for the implementation of a quality control process of anchor tests for a college admissions test with multiple consecutive administrations. We propose to examine the anchor tests and their items in connection with covariates to investigate if there was any unusual behavior in the anchor test results over time and if the test results differed for different groups of test takers. Descriptive statistics, ANOVA, and linear mixed effect models were used to examine the impact of covariates on the anchor test over time. Descriptive statistics and logistic regression were used to examine the quality of each anchor item over time. It is concluded that examining the impact of the test takers' covariates at different administrations can help in the process of providing fair tests for all test takers. The results are discussed with recommendations how it can be used in other large testing programs.

Linking Methods for the Zinnes-Griggs Pairwise Preference IRT Model.

Forced-choice item response theory (IRT) models are being more widely used as a way of reducing response biases in noncognitive research and operational testing contexts. As applications have increased, there has been a growing need for methods to link parameters estimated in different examinee groups as a prelude to measurement equivalence testing. This study compared four linking methods for the Zinnes and Griggs (ZG) pairwise preference ideal point model. A Monte Carlo simulation compared test characteristic curve (TCC) linking, item characteristic curve (ICC) linking, mean/mean (M/M) linking, and mean/sigma (M/S) linking. The results indicated that ICC linking and the simpler M/M and M/S methods performed better than TCC linking, and there were no substantial differences among the top three approaches. In addition, in the absence of possible contamination of the common (anchor) item subset due to differential item functioning, five items should be adequate for estimating the metric transformation coefficients. Our article presents the necessary equations for ZG linking and provides recommendations for practitioners who may be interested in developing and using pairwise preference measures for research and selection purposes.

Multidimensional Extension of Multiple Indicators Multiple Causes Models to Detect DIF.

A number of studies have found multiple indicators multiple causes (MIMIC) models to be an effective tool in detecting uniform differential item functioning (DIF) for individual items and item bundles. A recently developed MIMIC-interaction model is capable of detecting both uniform and nonuniform DIF in the unidimensional item response theory (IRT) framework. The goal of the current study is to extend the MIMIC-interaction model for detecting DIF in the context of multidimensional IRT modelling and examine the performance of the multidimensional MIMIC-interaction model under various simulation conditions with respect to Type I error and power rates. Simulation conditions include DIF pattern and magnitude, test length, correlation between latent traits, sample size, and latent mean differences between focal and reference groups. The results of this study indicate that power rates of the multidimensional MIMIC-interaction model under uniform DIF conditions were higher than those of nonuniform DIF conditions. When anchor item length and sample size increased, power for detecting DIF increased. Also, the equal latent mean condition tended to produce higher power rates than the different mean condition. Although the multidimensional MIMIC-interaction model was found to be a reasonably useful tool for identifying uniform DIF, the performance of the model in detecting nonuniform DIF appeared to be questionable.