Anchor Items Research Articles

Anchors Aweigh: How the Choice of Anchor Items Affects the Vertical Scaling of 3PL Data with the Rasch Model

ABSTRACT Vertical scaling is used to put tests of different difficulty onto a common metric. The Rasch model is often used to perform vertical scaling, despite its strict functional form. Few, if any, studies have examined anchor item choice when using the Rasch model to vertically scale data that do not fit the model. The purpose of this study was to investigate the implications of anchor item choice on bias in growth estimates when data do not fit the Rasch model. Data were generated with varying levels of true difference between grades and levels of the lower asymptote. When true growth or the lower asymptote were zero, estimates were unbiased and anchor item choice was not consequential. As true growth and the lower asymptote both increased, growth was underestimated and choice of anchor items had an impact. Easy anchor items led to less biased estimates of growth than hard anchor items.

The Impact of Covariate Variables on Kernel Equating under the Non-equivalent Groups

This study aims to use covariate variables correlated with the test scores instead of common items for non-equivalent groups with covariates (NEC) design in kernel equating. This study used the 2016 Monitoring and Evaluation of Academic Skills Project in Turkey. The study used data from 6,000 students, randomly selected from the Turkish Ministry of National Education’s current student data. Three thousand of the students took form A, and 3,000 of them took form B. The data include mathematics test scores and consist of 18 items, nine of which are the first items, and nine of which are anchor items. The equated scores from the NEC design were compared with equated scores from the non-equivalent group (NEAT) design. From the equating results, the root mean squared difference (RMSD) and standard error of equating (SEE) values were calculated. The results showed that NEC design could produce lower standard errors compared with the NEAT design, and the least RMSD was provided by NEAT PSE methods and NEC methods. The general result of this research is that test forms can be equated using covariates when there are no anchor items.

Improving the assessment of measurement invariance: Using regularization to select anchor items and identify differential item functioning.

A common challenge in the behavioral sciences is evaluating measurement invariance, or whether the measurement properties of a scale are consistent for individuals from different groups. Measurement invariance fails when differential item functioning (DIF) exists, that is, when item responses relate to the latent variable differently across groups. To identify DIF in a scale, many data-driven procedures iteratively test for DIF one item at a time while assuming other items have no DIF. The DIF-free items are used to anchor the scale of the latent variable across groups, identifying the model. A major drawback to these iterative testing procedures is that they can fail to select the correct anchor items and identify true DIF, particularly when DIF is present in many items. We propose an alternative method for selecting anchors and identifying DIF. Namely, we use regularization, a machine learning technique that imposes a penalty function during estimation to remove parameters that have little impact on the fit of the model. We focus specifically here on a lasso penalty for group differences in the item parameters within the two-parameter logistic item response theory model. We compare lasso regularization with the more commonly used likelihood ratio test method in a 2-group DIF analysis. Simulation and empirical results show that when large amounts of DIF are present and sample sizes are large, lasso regularization has far better control of Type I error than the likelihood ratio test method with little decrement in power. This provides strong evidence that lasso regularization is a promising alternative for testing DIF and selecting anchors. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Detecting Differential Item Functioning Using Multiple-Group Cognitive Diagnosis Models.

This study proposes a multiple-group cognitive diagnosis model to account for the fact that students in different groups may use distinct attributes or use the same attributes but in different manners (e.g., conjunctive, disjunctive, and compensatory) to solve problems. Based on the proposed model, this study systematically investigates the performance of the likelihood ratio (LR) test and Wald test in detecting differential item functioning (DIF). A forward anchor item search procedure was also proposed to identify a set of anchor items with invariant item parameters across groups. Results showed that the LR and Wald tests with the forward anchor item search algorithm produced better calibrated Type I error rates than the ordinary LR and Wald tests, especially when items were of low quality. A set of real data were also analyzed to illustrate the use of these DIF detection procedures.

Linking Ability Scales Under the Multidimensional Nominal Response Model

본 연구의 목적은 탐색적 다차원 문항반응이론(item response theory: IRT) 모형 중 가장 일반적인 형태인 다차원 명명반응(nominal response: NR) 모형을 위한 공통-문항 척도연계 방법을 제시하고 이 방법의 적절성을 탐구하는 데 있다. 이를 위해 최소제곱(LSQ) 방법과 문항유목반응함수(IRF) 방법의 두 가지 방법을 제시하고 두 방법의 기능을 모의실험을 통해 검토하였다. 모의실험 요인으로 기저 능력분포의 유형, 표본의 크기, 가교 문항의 수 등을 포함하였다. 대부분의 모의실험 조건에서 LSQ 방법은 임의의 다차원 능력척도를 기준 능력척도로 변환하는 데 사용되는 척도연계 계수(회전행렬과 이동벡터)를 추정함에 있어 IRF 방법보다 열등한 기능을 보였다. 특히 LSQ 방법은 척도연계를 통해 기저 능력분포를 추정할 때 능력 변수의 분산을 상당한 정도로 과소 혹은 과대 추정하는 경향을 보였다. 이에 비해 IRF 방법은, 가교 문항의 수가 5인 조건을 제외하고, 척도연계를 통해 기저 능력분포의 모수뿐만 아니라 문항 모수 또한 적절히 추정하였다. 논문의 말미에 다차원 NR 모형을 적용한 검사 자료의 분석 과정에서 IRF 척도연계 방법이 어떻게 실시되어야 하는지에 대해 논의하였다.In item response theory (IRT), the exploratory multidimensional nominal response (NR) model is one of the most general models, from which several dichotomous and polytomous models can be derived. The purpose of this paper is to present common-item scale-linking methods for the multidimensional NR model and investigate their characteristics and performances. For this, the least squares (LSQ) method and the item-category response function (IRF) method were presented and their functions were examined through computer simulations. The simulation study included three factors of (a) type of population ability distribution, (b) sample size, and (c) number of anchor items. Simulation results showed that overall, the LSQ method was inferior to the IRF method in estimating the linking coefficients (rotation matrix and translation vector) used to transform an arbitrary multidimensional ability scale to the base scale. In particular, the LSQ method tended to considerably under-estimate or over-estimate the variances of ability variables when recovering the underlying ability distribution through scale linking. On the other hand, the IRF method recovered well the parameters of the underlying ability distribution as well as the item parameters through scale linking. At the end of the paper, a discussion is provided as to how the IRF linking method should be implemented in association with the process of test data analysis based on the multidimensional NR model.

A proposal of a financial knowledge scale based on item response theory

The evaluation of financial knowledge is still a controversial subject in the literature, and this work seeks to advance by proposing instruments for this evaluation. Based on the related literature, twenty-four questions were adapted in the initial instrument. Using Item Response Theory, a synthetic instrument was developed, with twelve questions that evaluate three levels of items: basic, intermediate and advanced. From the skill and anchor items scale, two smaller scales of nine and seven items were also derived for financial knowledge assessment. The scales proposed in this study innovate by differing from those commonly used in the literature, by considering not only the percentage of correct answers but also the parameters of the item that the individual answers correctly. The three scales have items from different levels of knowledge, allowing researchers, educators, and policymakers to choose which is the most appropriate to their objectives.

Evaluating the Thresholds for Clinical Importance of the EORTC QLQ-C15-PAL in Patients Receiving Palliative Treatment.

Background: The EORTC QLQ-C15-PAL is a shortened version of the widely used EORTC QLQ-C30. This questionnaire was developed to measure the symptoms and functional health of patients receiving palliative care. Objective: To enhance clinical interpretability of the EORTC QLQ-C15-PAL, our aim was to evaluate the sensitivity and specificity of thresholds for clinical importance developed previously for the QLQ-C30 when applied to the QLQ-C15-PAL scales. Design: Cross-sectional observational study. Setting/Subjects: Patients with cancer receiving any type of palliative treatment. Measurement: Patients completed the EORTC QLQ-C15-PAL and anchor items on limitations, worries, and need for help for each of the health domains covered by the questionnaire. The anchor items were summarized in a binary criterion for clinical importance to calculate the sensitivity and specificity of the thresholds for clinical importance. Results: In total, 225 patients participated in the study (mean age 64.5 years). Patients were recruited from Austria, Italy, the Netherlands, Poland, Spain, and the United Kingdom. The thresholds for clinical importance for the QLQ-C15-PAL scales showed a median sensitivity of 0.88 (range: 0.82 for sleep disturbances to 1.00 for dyspnea) and a median specificity of 0.74 (range: 0.54 for dyspnea to 0.89 for constipation). Conclusion: The thresholds for clinical importance showed high sensitivity and mostly high specificity in identifying clinically important symptoms and functional health impairments as assessed by the QLQ-C15-PAL. These thresholds will facilitate interpretation of EORTC QLQ-C15-PAL scores in daily clinical practice and clinical research.

Assessing the Word Recognition Skills of German Elementary Students in Silent Reading—Psychometric Properties of an Item Pool to Generate Curriculum-Based Measurements

Given the high proportion of struggling readers in school and the long-term negative consequences of underachievement for those affected, the question of prevention options arises. The early identification of central indicators for reading literacy is a noteworthy starting point. In this context, curriculum-based measurements have established themselves as reliable and valid instruments for monitoring the progress of learning processes. This article is dedicated to the assessment of word recognition in silent reading as an indicator of adequate reading fluency. The process of developing an item pool is described, from which instruments for learning process diagnostics can be derived. A sample of 4268 students from grades 1–4 processed a subset of items. Each student template included anchor items, which all students processed. Using Item Response Theory, item statistics were estimated for the entire sample and all items. After eliminating unsuitable items (N = 206), a one-dimensional, homogeneous pool of items remained. In addition, there are high correlations with another established reading test. This provides the first evidence that the recording of word recognition skills for silent reading can be seen as an economic indicator for reading skills. Although the item pool forms an important basis for the extraction of curriculum-based measurements, further investigations to assess the diagnostic suitability (e.g., the measurement invariance over different test times) are still pending.

Investigating the Classification Accuracy of Rasch and Nominal Weights Mean Equating with Very Small Samples

ABSTRACTMaintaining equivalent performance standards across forms is a psychometric challenge exacerbated by small samples. In this study, the accuracy of two equating methods (Rasch anchored calibration and nominal weights mean) and four anchor item selection methods were investigated in the context of very small samples (N = 10). Overall, nominal weights mean equating slightly outperformed Rasch equating for three of the four anchor item selection methods, but Rasch equating slightly outperformed nominal weights mean equating when anchor items were selected to be near the cut score. The results largely confirmed previous research on the utility of nominal weights mean equating for very small samples. In addition, the results provide useful guidance for small volume programs who wish to consider using Rasch for building and equating new forms. Lastly, the results underscored the importance of being mindful about the method for selecting anchor items when building new forms.

Equating with Small and Unbalanced Samples

ABSTRACTRecent research has suggested that re-setting the standard for each administration of a small sample examination, in addition to the high cost, does not adequately maintain similar performance expectations year after year. Small-sample equating methods have shown promise with samples between 20 and 30. For groups that have fewer than 20 students, options are scarcer. This simulation study examined balanced and unbalanced designs across nine equating models including both classic equating models and small-sample models. The study also accounted for varying sample sizes, differences in form difficulty and candidate population, and the size of the anchor set. This study supports the use of nominal weights approaches in combination with either circle-arc or mean equating. Consistent with other research, this study found that the best ways to improve equating results are increases in sample size and/or the number of anchor items across the old and new forms. However, a testing program’s tolerance for reuse will influence the decision to pool administrations.

Supplemental Material for Improving the Assessment of Measurement Invariance: Using Regularization to Select Anchor Items and Identify Differential Item Functioning

Supplemental Material for Improving the Assessment of Measurement Invariance: Using Regularization to Select Anchor Items and Identify Differential Item Functioning

Developing of Bloomian HOTS Physics Test: Content and Construct Validation of The PhysTeBloHOTS

Higher order thinking skills (HOTS) of the students in physics learning need to be assessed. There are several kinds of HOTS, one of which is Bloomian HOTS. The Bloomian HOTS Physics test for Senior High School Students called PhysTeBloHOTS has been developed to assess students’ HOTS. The purpose of this study is to prove validation of the PhysTeBloHOTS. There were two steps of validation: proving the content validity and construct validity. The content validity analysis was conducted using Aiken’s V and the analysis of construct validation was conducted using exploratory factor analysis (EFA). The PhysTeBloHOTS consisted of four test sets: A, B, C, and D. The each of test set consists of 40 items with 8 anchor items. For the test trial, PhysTeBloHOTS were administrated to 301 Senior High School Students in Yogyakarta. The result of the study shows that PhysTeBloHOTS is valid according to the content and construct validity. The PhysTeBloHOTS items are valid according to the expert judgments with Aiken’s V index ranged from 0.83 to 0.97, respectively. The result of EFA and CFA show that PhysTeBloHOTS consist of three aspects: analysis, evaluate, and create.

Evaluating the Quantity-Quality Trade-off in the Selection of Anchor Items: a Vertical Scaling Approach

When administering tests across grades, vertical scaling is often employed to place scores from different tests on a common overall scale so that test-takers’ progress can be tracked. In order to be able to link the results across grades, however, common items are needed that are included in both test forms. In the literature there seems to be no clear agreement about the ideal number of common items. In line with some scholars, we argue that a greater number of anchor items bear a higher risk of unwanted effects like displacement, item drift, or undesired fit statistics and that having fewer psychometrically well-functioning anchor items can sometimes be more desirable. In order to demonstrate this, a study was conducted that included the administration of a reading-comprehension test to 1,350 test-takers across grades 6 to 8. In employing a step-by-step approach, we found that the paradox of high item drift in test administrations across grades can be mitigated and eventually even be eliminated. At the same time, a positive side effect was an increase in the explanatory power of the empirical data. Moreover, it was found that scaling adjustment can be used to evaluate the effectiveness of a vertical scaling approach and, in certain cases, can lead to more accurate results than the use of calibrated anchor items.

Thresholds for clinical importance were defined for the European Organisation for Research and Treatment of Cancer Computer Adaptive Testing Core—an adaptive measure of core quality of life domains in oncology clinical practice and research

ObjectivesThe aim of this article was to establish thresholds for clinical importance (TCIs) for the European Organisation for Research and Treatment of Cancer (EORTC) Computer Adaptive Testing (CAT) Core measure, the new adaptive version of the EORTC QLQ-C30. Study Design and SettingFor our diagnostic study, we recruited cancer patients with mixed diagnoses and treatments from six European countries. Patients completed the EORTC CAT Core and a questionnaire with anchor items assessing criteria for clinical importance (limitations in everyday life, need for help/care, and worries by the patient/family/partner) for each EORTC CAT Core domain. We used a binary variable summarizing the anchor items for determining TCIs and for calculating the area under the curve (AUC) in receiving operator characteristic analysis as a measure of diagnostic accuracy. ResultsUsing data from 498 cancer patients (mean age 60.4 years, 55.2% women), we established TCIs for the 14 domains of the EORTC CAT Core. Median AUC across domains was 0.93 (range 0.84–0.94). Median sensitivity and specificity of the TCIs were 0.91 (range 0.80–0.96) and 0.77 (range 0.66–0.84), respectively. TCIs and AUCs were largely consistent across patient groups. ConclusionWe have generated TCIs for the 14 functional health and symptom domains of the EORTC CAT Core. The EORTC CAT Core showed high diagnostic accuracy in identifying clinically important symptoms and functional impairments.

Application of IRT Fixed Parameter Calibration to Multiple-Group Test Data

ABSTRACTIn applications of item response theory (IRT), fixed parameter calibration (FPC) has been used to estimate the item parameters of a new test form on the existing ability scale of an item pool. The present paper presents an application of FPC to multiple examinee groups test data that are linked to the item pool via anchor items, and investigates the performance of FPC relative to an alternative approach, namely independent 0–1 calibration and scale linking. Two designs for linking to the pool are proposed that involve multiple groups and test forms, for which multiple-group FPC can be effectively used. A real-data study shows that the multiple-group FPC method performs similarly to the alternative method in estimating ability distributions and new item parameters on the scale of the item pool. In addition, a simulation study shows that the multiple-group FPC method performs nearly equally to or better than the alternative method in recovering the underlying ability distributions and the new item parameters.

Definition and construct validation of clinically relevant cutoffs on the Flare Assessment in Rheumatoid Arthritis (FLARE-RA) questionnaire

ObjectiveThe Flare Assessment in Rheumatoid Arthritis (FLARE-RA) questionnaire was devised for the detection of flares in patients with RA. We aimed to define construct validity and cut-off(s) for the FLARE-RA questionnaire. MethodsThis cross-sectional study included adult patients with prevalent RA (2010 ACR/EULAR criteria) attending outpatient rheumatology clinics in France (n = 138), Denmark (n = 253), USA (n = 75), and Argentina (n = 105). Flare occurrence over the past 3 months was assessed with the FLARE-RA questionnaire scoring from 0 (no flare) to 10 (maximum flare). The cut-offs for the FLARE-RA score were defined using the following anchor items obtained at the same encounter: (1) Patient report of flare; (2) DAS28-CRP > 3.2; (3) Change of anti-rheumatic treatment, based on the area under the receiver operating characteristic curve (AUC) and distance to (0,1). ResultsFour hundred seventy four patients with RA duration ≥2 years (mean age 58.6 years, 74.9% female) were included in the main analysis. The discrimination for the FLARE-RA cut-offs was acceptable-to-excellent: AUC for the global FLARE-RA score ranged from 0.71 to 0.92. The cut-offs for the FLARE-RA score were lower using “patient report of flare” than DAS28-CRP and “change of anti-rheumatic treatment”. Proposed FLARE-RA cut-offs for clinical detection and change of anti-rheumatic treatment are 2 and 5, respectively, for patients with RA duration 2–5 years, and 2 and 3.5, respectively, for patients with RA duration >5 years. ConclusionsProposed FLARE-RA cut-offs have acceptable discriminative capacity across the tested anchor items and are expected to aid in early recognition and timely management of RA flares.

A Comparison of Aggregation Rules for Selecting Anchor Items in Multigroup DIF Analysis

AbstractThis study addresses the topic of how anchoring methods for differential item functioning (DIF) analysis can be used in multigroup scenarios. The direct approach would be to combine anchoring methods developed for two‐group scenarios with multigroup DIF‐detection methods. Alternatively, multiple tests could be carried out. The results of these tests need to be aggregated to determine the anchor for the final DIF analysis. In this study, the direct approach and three aggregation rules are investigated. All approaches are combined with a variety of anchoring methods, such as the “all‐other purified” and “mean p‐value threshold” methods, in two simulation studies based on the Rasch model. Our results indicate that the direct approach generally does not lead to more accurate or even to inferior results than the aggregation rules. The min rule overall shows the best trade‐off between low false alarm rate and medium to high hit rate. However, it might be too sensitive when the number of groups is large. In this case, the all rule may be a good compromise. We also take a closer look at the anchor selection method “next candidate,” which performed rather poorly, and suggest possible improvements.

Alignment as an Alternative to Anchor Purification in DIF Analyses

Unbalanced differential item functioning (DIF) occurs when the items favoring group 1 are not balanced by items favoring group 2. This creates false differences in the total score used for matching examinee subgroups or inaccuracy in the scaling used to bring the groups to commensurable metrics. Historically, unbalanced DIF has been addressed by removing DIF items from the set of anchor items, often called purification. A recently developed alignment procedure minimizes the impact of the DIF items in scaling the groups without removing them. In this study, when the data followed a two-parameter model, the results using alignment or a purified anchor were generally comparable when the DIF was balanced or when the DIF was unbalanced but no more than 20% of the items contained DIF. With a larger percentage of unbalanced DIF, the alignment procedure yielded more accurate estimates of the group means and the magnitude of the DIF.

Development of HOTS Oriented Cognitive Problems in Thermodynamics for Senior High Schools

One of cognitive abilities of students that should be trained and developed in the 21st century learning is high-level thinking skills. As higher order thinking skills are emerging trend in education, assessment instruments based on high order thinking skills (HOTS) principles are required. This study aims to develop a high-level thinking ability instrument for Physics subject of senior high schools which is intended to measure HOTS of senior high school students for Thermodynamic Materials in City of Nganjuk in East Java. The study was conducted in three stages, including initial development, trials, and measurement. The initial development stage included compilation, review, and validation, including content validation by experts. There are two sets questions A and B, consisting 20 questions respectively, with anchor items as much as 5 items. The measurement results show that HOTS-oriented test kits of Thermodynamics have obtained empirical evidence are suitable with partial credit model. The difficulty level of the test battery is in the range of -2.0 to +2.0, with the most difficult aspects in the sequence are analyzing, evaluating, and creating. The reliability of HOTS-oriented tests has met the high categorical requirements, and based on the total information function of the tests, it is appropriately used to measure the HOTS of learners with capability -2.3 to +1.6. Based on students’ responses, the analysis ability is the highest skills followed by the ability to evaluate and create. HOTS-oriented tests developed can be used as the reference for the teachers to accommodate student’s higher order thinking skills.

Methods used by mathematics teachers in developing parallel multiple-choice test items in school

The study was aimed at describing five methods of the development of parallel test items of the multiple-choice type in mathematics at Yogyakarta (primary education level). The study was descriptive research involving 22 mathematics teachers as the respondents. Data collection was conducted through interviews and document reviews concerning the developed test packages. A questionnaire was used to gather data about the procedure the teachers employed in developing the tests. Findings show that the teachers used five methods in developing the test item; namely (1) randomizing the item numbers; (2) randomizing the sequences of response options; (3) writing items using the same contexts but different figures; (4) using anchor items; and (5) writing different items based on the same specification table. All of the respondents stated that they developed the table of the specification before developing the test items and that most of them (77%) did the validation of the instruments in content and language.