Anchor Items Research Articles

Differential Item Functioning via Robust Scaling.

This paper proposes a method for assessing differential item functioning (DIF) in item response theory (IRT) models. The method does not require pre-specification of anchor items, which is its main virtue. It is developed in two main steps: first by showing how DIF can be re-formulated as a problem of outlier detection in IRT-based scaling and then tackling the latter using methods from robust statistics. The proposal is a redescending M-estimator of IRT scaling parameters that is tuned to flag items with DIF at the desired asymptotic type I error rate. Theoretical results describe the efficiency of the estimator in the absence of DIF and its robustness in the presence of DIF. Simulation studies show that the proposed method compares favorably to currently available approaches for DIF detection, and a real data example illustrates its application in a research context where pre-specification of anchor items is infeasible. The focus of the paper is the two-parameter logistic model in two independent groups, with extensions to other settings considered in the conclusion.

Linking Methods for Multidimensional Forced Choice Tests Using the Multi-Unidimensional Pairwise Preference Model.

Applications of multidimensional forced choice (MFC) testing have increased considerably over the last 20years. Yet there has been little, if any, research on methods for linking the parameter estimates from different samples. This research addressed that important need by extending four widely used methods for unidimensional linking and comparing the efficacy of new estimation algorithms for MFC linking coefficients based on the Multi-Unidimensional Pairwise Preference model (MUPP). More specifically, we compared the efficacy of multidimensional test characteristic curve (TCC), item characteristic curve (ICC; Haebara, 1980), mean/mean (M/M), and mean/sigma (M/S) methods in a Monte Carlo study that also manipulated test length, test dimensionality, sample size, percentage of anchor items, and linking scenarios. Results indicated that the ICC method outperformed the M/M method, which was better than the M/S method, with the TCC method being the least effective. However, as the number of items "per dimension" and the percentage of anchor items increased, the differences between the ICC, M/M, and M/S methods decreased. Study implications and practical recommendations for MUPP linking, as well as limitations, are discussed.

DIF Analysis with Unknown Groups and Anchor Items

Ensuring fairness in instruments like survey questionnaires or educational tests is crucial. One way to address this is by a Differential Item Functioning (DIF) analysis, which examines if different subgroups respond differently to a particular item, controlling for their overall latent construct level. DIF analysis is typically conducted to assess measurement invariance at the item level. Traditional DIF analysis methods require knowing the comparison groups (reference and focal groups) and anchor items (a subset of DIF-free items). Such prior knowledge may not always be available, and psychometric methods have been proposed for DIF analysis when one piece of information is unknown. More specifically, when the comparison groups are unknown while anchor items are known, latent DIF analysis methods have been proposed that estimate the unknown groups by latent classes. When anchor items are unknown while comparison groups are known, methods have also been proposed, typically under a sparsity assumption – the number of DIF items is not too large. However, DIF analysis when both pieces of information are unknown has not received much attention. This paper proposes a general statistical framework under this setting. In the proposed framework, we model the unknown groups by latent classes and introduce item-specific DIF parameters to capture the DIF effects. Assuming the number of DIF items is relatively small, an L1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$L_1$$\\end{document}-regularised estimator is proposed to simultaneously identify the latent classes and the DIF items. A computationally efficient Expectation-Maximisation (EM) algorithm is developed to solve the non-smooth optimisation problem for the regularised estimator. The performance of the proposed method is evaluated by simulation studies and an application to item response data from a real-world educational test.

Development of Student Cognitive Learning Outcomes Tests Based on Differentiated Learning

Differentiated learning is the core of the independent curriculum now. The aim of this research is to develop an instrument for measuring the cognitive learning outcomes of high school students based on differentiated learning on elasticity material and to obtain its characteristics. Differentiation, in this case, is differentiation of student learning styles. The instrument grid is arranged based on indicators of cognitive learning outcomes, which are then used to compile question items. The test instrument consists of two test sets, each of which has 9 items including two anchor items so that the total number of items is 16 items and has been validated by 6 experts. The validated instrument was tested on 252 respondents spread across high schools in the low, medium and high categories. Dichotomous data were analyzed using the Partial Credit Model (PCM). The trial results showed that all 16 items and the differentiated learning-based student cognitive learning outcomes test instrument were proven to be valid, fit the PCM model, and reliable, which means all items were in the good category. Thus, the test instrument developed meets the requirements for measuring the cognitive learning outcomes of high school students on elasticity material based on differentiated learning.

Bayesian inference with spike-and-slab priors for differential item functioning detection in a multiple-group IRT tree model

Group differences have practical implications in analysing data from achievement tests or questionnaires. In the current study, we develop a model that accounts for between-group differences, differential item functioning (DIF), latent factors, and missing item response data simultaneously. Different from most of the present DIF studies where one has to iteratively select anchor items and detect DIF items, we achieve DIF detection and parameter estimation simultaneously by properly reparameterizing model parameters and applying some spike-and-slab priors (Ishwaran & Rao, Spike and slab variable selection: frequentist and Bayesian strategies. Ann Stat. 2005a;33:730–773; Ročková & George, The spike-and-slab LASSO. J Am Stat Assoc. 2018;113:431–444) in Bayesian estimation. Simulation studies are conducted to illustrate the validation of the proposed estimation procedure and the efficiency of DIF detection. The proposed method is further applied to a real dataset for illustration.

Measuring Mathematical Competence of Grade 3 Students in New General Curriculum: A Case of Vietnam

The general education curriculum promulgated by the Vietnamese Ministry of Education and Training in 2018 has identified 10 common competencies that need to be formed and developed for students in the coming period, including computing capacity. The Math general education curriculum issued in 2018 helps students form and develop mathematical competence (the most concentrated expression of computing capacity) including the following core elements: mathematical thinking and reasoning; mathematical modeling; mathematical problem solving; mathematical communication; use mathematical learning tools and media. The aim of this study was to design an assessment framework for mathematical competence. From the Grade 3 Math curriculum, the research team has proposed a math competency standard with 90 items. From these 90 items, design 6 illustrative test questions. Each test consists of 30 questions, between which there are 15 fixed questions designed to measure students’ mathematical competence. A sample of 383 students in grade 3 were recruited for the study with the participation of 3 researchers and 5 elementary school teachers. A mathematical competence framework and six tests with anchor items were designed to measure students’ mathematical competence. The results of analysis using an approach to item response theory showed that the items had good fits with the model and they could be used to describe mathematical competence learning progression with different levels of proficiency. The tests were reliable and valid, and the anchor items were good for connecting students’ mathematical competence.

Detecting Item Parameter Drift in Small Sample Rasch Equating

ABSTRACT Screening items for parameter drift helps protect against serious validity threats and ensure score comparability when equating forms. Although many high-stakes credentialing examinations operate with small sample sizes, few studies have investigated methods to detect drift in small sample equating. This study demonstrates that several newly researched drift detection strategies can improve equating accuracy under certain conditions with small samples where some anchor items display item parameter drift. Results showed that the recently proposed methods mINFIT and mOUTFIT as well as the more conventional Robust-z helped mitigate the adverse effects of drifting anchor items in conditions with higher drift levels or with more than 75 examinees. In contrast, the Logit Difference approach excessively removed invariant anchor items. The discussion provides recommendations on how practitioners working with small samples can use the results to make more informed decisions regarding item parameter drift.

Assessment of the Brazilian Version of the Epworth Sleepiness Scale Using Item Response Theory: A Psychometric Study.

There seems to be a consensus that adolescents worldwide are not getting enough sleep. This study aimed to evaluate the psychometric properties of the Epworth Sleepiness Scale (ESS) in adolescents using the item response theory. A psychometric study was conducted with 2206 adolescents aged 18 and 19 years in the city of São Luís, Maranhão, Brazil. The dimensionality of the ESS was assessed by principal component analysis. A Samejima's graded response model (SGRM) was fitted to it. The findings of this study showed a good internal consistency and the unidimensionality of the ESS. Considering the latent trait continuum, we obtained three levels, with anchor items. For the item 'Sitting still in a public place', the adolescents presented a small possibility of dozing in level 1, and a medium and great possibility in level 2. The item 'Sitting around talking to someone' presented small, medium, and great possibilities of dozing in level 3. The ESS with five items showed satisfactory psychometric properties. In addition, the results from the SGRM showed that adolescents with high levels of sleepiness are likely to nod off or sleep sitting up while talking to someone. This study allows us to understand excessive daytime sleepiness in adolescents.

DIF Statistical Inference Without Knowing Anchoring Items

Establishing the invariance property of an instrument (e.g., a questionnaire or test) is a key step for establishing its measurement validity. Measurement invariance is typically assessed by differential item functioning (DIF) analysis, i.e., detecting DIF items whose response distribution depends not only on the latent trait measured by the instrument but also on the group membership. DIF analysis is confounded by the group difference in the latent trait distributions. Many DIF analyses require knowing several anchor items that are DIF-free in order to draw inferences on whether each of the rest is a DIF item, where the anchor items are used to identify the latent trait distributions. When no prior information on anchor items is available, or some anchor items are misspecified, item purification methods and regularized estimation methods can be used. The former iteratively purifies the anchor set by a stepwise model selection procedure, and the latter selects the DIF-free items by a LASSO-type regularization approach. Unfortunately, unlike the methods based on a correctly specified anchor set, these methods are not guaranteed to provide valid statistical inference (e.g., confidence intervals and p-values). In this paper, we propose a new method for DIF analysis under a multiple indicators and multiple causes (MIMIC) model for DIF. This method adopts a minimal L_1 norm condition for identifying the latent trait distributions. Without requiring prior knowledge about an anchor set, it can accurately estimate the DIF effects of individual items and further draw valid statistical inferences for quantifying the uncertainty. Specifically, the inference results allow us to control the type-I error for DIF detection, which may not be possible with item purification and regularized estimation methods. We conduct simulation studies to evaluate the performance of the proposed method and compare it with the anchor-set-based likelihood ratio test approach and the LASSO approach. The proposed method is applied to analysing the three personality scales of the Eysenck personality questionnaire-revised (EPQ-R).

Wald χ2 Test for Differential Item Functioning Detection with Polytomous Items in Multilevel Data

Identifying items with differential item functioning (DIF) in an assessment is a crucial step for achieving equitable measurement. One critical issue that has not been fully addressed with existing studies is how DIF items can be detected when data are multilevel. In the present study, we introduced a Lord’s Wald [Formula: see text] test-based procedure for detecting both uniform and non-uniform DIF with polytomous items in the presence of the ubiquitous multilevel data structure. The proposed approach is a multilevel extension of a two-stage procedure, which identifies anchor items in its first stage and formally evaluates candidate items in the second stage. We applied the Metropolis–Hastings Robbins–Monro (MH-RM) algorithm to estimate multilevel polytomous item response theory (IRT) models and to obtain accurate covariance matrices. To evaluate the performance of the proposed approach, we conducted a preliminary simulation study that considered various conditions to mimic real-world scenarios. The simulation results indicated that the proposed approach has great power for identifying DIF items and well controls the Type I error rate. Limitations and future research directions were also discussed.

The Korean brain aging study for the early diagnosis and prediction of Alzheimer’s disease (KBASE): Cognitive data harmonization

AbstractBackgroundKBASE is a prospective cohort study launched at Seoul National University (SNU), South Korea, in 2014 using similar design and methods as the North American Alzheimer’s Disease Neuroimaging Initiative (ADNI). The KBASE cohort consists of well‐characterized participants (420 cognitively normal, 140 mild cognitive impairment, and 90 AD dementia) (Figure 1). It includes systematic longitudinal collection of comprehensive clinical, cognitive and lifestyle data, multimodal neuroimaging, and bio‐specimens for the first six years (“KBASE1”) at a single center. “KBASE2: Korean Brain Aging Study, Longitudinal Endophenotypes and Systems Biology” (NIH Grant U01AG072177) is the second phase of KBASE, launched in collaboration with Indiana University and other institutions. KBASE2 began in 2021, and has similar assessment protocols to the KBASE1. The NIA AD Sequencing Project (ADSP) will perform GWAS and whole genome sequencing (WGS).MethodNeuropsychological data will be harmonized by the study team and members of the ADSP Phenotype Harmonization Consortium. Comprehensive cognitive data were collected using a battery of standardized tests (Table 1). Harmonization and co‐calibration will enable comparative analyses with cognitive data from similar studies with largely non‐Hispanic white (NHW) cohorts, including the ADNI and AIBL, and ultimately the range of other aging and dementia cohorts including those with underrepresented multi‐ethnic populations. WGS and GWAS data will be harmonized and shared by the ADSP.ResultCognitive data harmonization processes are underway and have begun to address the challenges of language and cultural differences. Initial steps include mapping tests and items into an English template. Harmonization and co‐calibration efforts will include identifying appropriate anchor items and accounting for differences in test items that were made to be more culturally and linguistically appropriate. Additional challenges include the wide range of educational attainment as well as gender effects on educational opportunities. A bilingual neuropsychologist who is trained and licensed in the U.S. and a bilingual psychometrics expert are deeply involved in harmonization processes.ConclusionHarmonized KBASE cognitive data will be shared with the ADSP and other investigators for use in quantitative phenotype analyses, as well as other pooled, integrative, and comparative studies of other multiethnic, multilingual, U.S., and international cohorts.

Cognitive domain harmonization and cocalibration in studies of older adults.

Studies use different instruments to measure cognitirating cognitive tests permit direct comparisons of individuals across studies and pooling data for joint analyses. We began our legacy item bank with data from the Adult Changes in Thought study (n = 5,546), the Alzheimer's Disease Neuroimaging Initiative (n = 3,016), the Rush Memory and Aging Project (n = 2,163), and the Religious on such as the Mini-Mental State Examination, the Alzheimer's Disease Assessment Scale-Cognitive Subscale, the Wechsler Memory Scale, and the Boston Naming Test. CocalibOrders Study (n = 1,456). Our workflow begins with categorizing items administered in each study as indicators of memory, executive functioning, language, visuospatial functioning, or none of these domains. We use confirmatory factor analysis models with data from the most recent visit on the pooled sample across these four studies for cocalibration and derive item parameters for all items. Using these item parameters, we then estimate factor scores along with corresponding standard errors for each domain for each study. We added additional studies to our pipeline as available and focused on thorough consideration of candidate anchor items with identical content and administration methods across studies. Prestatistical harmonization steps such qualitative and quantitative assessment of granular cognitive items and evaluating factor structure are important steps when trying to cocalibrate cognitive scores across studies. We have cocalibrated cognitive data and derived scores for four domains for 76,723 individuals across 10 studies. We have implemented a large-scale effort to harmonize and cocalibrate cognitive domain scores across multiple studies of cognitive aging. Scores on the same metric facilitate meta-analyses of cognitive outcomes across studies or the joint analysis of individual data across studies. Our systematic approach allows for cocalibration of additional studies as they become available and our growing item bank enables robust investigation of cognition in the context of aging and dementia. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Ceiling effects and differential measurement precision across calibrated cognitive scores in the Framingham Study.

To calibrate cognitive assessment data across multiple waves of the Framingham Heart Study (FHS), addressing study design considerations, ceiling effects, and measurement precision. FHS participants completed several cognitive assessments including screening instruments and more comprehensive batteries at different study visits. We used expert opinion to assign each cognitive test item to a single domain-memory, executive function, language, visuospatial abilities, or none of the above. As part of a larger cross-study harmonization effort, we calibrated each domain separately using bifactor confirmatory factor analysis (CFA) models, incorporating item parameters for anchor items previously calibrated from other studies and freely estimating item parameters for FHS-specific items. We obtained scores and standard errors (SEs) for each participant at each study visit. We addressed psychometric considerations of ceiling effects and measurement precision. Overall, memory domain scores were the most precisely estimated. Scores for all domains from visits where the Mini-Mental State Examination (MMSE) was the only test administered were imprecisely estimated and suffered from ceiling effects. Scores from visits with a more extensive battery were estimated more precisely and better differentiated between ability levels. The harmonized and calibrated cognitive data from the FHS should prove useful for future analyses examining cognition and cognitive decline. They will be of particular interest when combining FHS with other studies that have been similarly calibrated. Researchers should be aware of varying levels of measurement precision and the possibility of ceiling effects in their planned analyses of data from the FHS and similar studies. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Comparing Drift Detection Methods for Accurate Rasch Equating in Different Sample Sizes

ABSTRACT Rasch common-item equating is often used in high-stakes testing to maintain equivalent passing standards across test administrations. If unaddressed, item parameter drift poses a major threat to the accuracy of Rasch common-item equating. We compared the performance of well-established and newly developed drift detection methods in small and large sample sizes, varying the proportion of test items used as anchor (common) items and the proportion of drifted anchors. In the simulated-data study, the most accurate equating was obtained in large-sample conditions with a small-moderate number of drifted anchors using the mINFIT/mOUTFIT methods. However, when any drift was present in small-sample conditions and when a large number of drifted anchors were present in large-sample conditions, all methods performed ineffectively. In the operational-data study, percent-correct standards and failure rates varied across the methods in the large-sample exam but not in the small-sample exam. Different recommendations for high- and low-volume testing programs are provided.

혼합형 가교문항 구성 방법이 혼합형 검사 동등화에 미치는 영향

With the introduction of innovative items in the computer-based NAEA, the item format became diversified, and the current method of selecting anchor items from multiple-choice (MC) items became burdensome for test developers. The purpose of this study is to examine whether a mixed-format anchor items including both MC and constructed-response (CR) items could hold the stability of equating in the NAEA, and give desirable direction to expand the range of anchor items and to make test construct flexible. A simulation study was conducted based on a common-item non-equivalent group design with the test conditions of the Korean subject in the NAEA. The treatment conditions include (1) a type of anchor item set (a single MC or a mixed), (2) a composition type of mixed anchor item (MC:CR = 1:1, 3:2, or 3:4), (3) a length of mixed anchor items within a test (25%, 22%, or 19%), and (4) the presence of multidimensionality according to the item format were assumed. The main result is that there was no significant difference in the equating error between the single MC and the mixed anchor item condition. Specifically the equating error of the single MC was smaller than that of the mixed-format cases where the ratios between MC and CR were 1:1 and 3:2. In the 3:4-ratio mixed condition, the equating error was smaller than that of the single MC condition. A length of mixed anchor items within a test showed somewhat mixed results, probably because there was not much variation between the conditions in the composition ratio, about 20% in test. When there was multidimensionality according to the item format in the test, the equating error increased significantly with mixed anchor items compared to that of unidimensional test condition. In sum, about 20% of the mixed anchor items with slightly higher CR ratio was found to be suitable in the unidimensional test and caution was called for in using mixed anchor items in multidimensional test.

Don't worry about the anchor-item setting in longitudinal learning diagnostic assessments.

Previous longitudinal assessment experiences for multidimensional continuous latent constructs suggested that the set of anchor items should be proportionally representative of the total test forms in content and statistical characteristics and that they should be loaded on every domain in multidimensional tests. In such cases, the set of items containing the unit Q-matrix, which is the smallest unit representing the whole test, seems to be the natural choice for anchor items. Two simulation studies were conducted to verify the applicability of these existing insights to longitudinal learning diagnostic assessments (LDAs). The results mainly indicated that there is no effect on the classification accuracy regardless of the unit Q-matrix in the anchor items, and even not including the anchor items has no impact on the classification accuracy. The findings of this brief study may ease practitioners' worries regarding anchor-item settings in the practice application of longitudinal LDAs.

Profiles of Quality in Three Distinct Early Childhood Programs Using the Brief Early Childhood Quality Inventory (BEQI)

Quality early childhood care and education (ECCE) is important for young children’s holistic healthy development. As ECCE scales, contextually relevant and feasible measurement is needed to inform policy and programs on strengths and areas for improvement. However, few measures have been designed for use across diverse contexts. Drawing on principles of mixed methods design, this study reports on a new approach to ECCE quality measurement: the Brief Early Childhood Quality Inventory. Using data from the USA, Liberia, and Colombia, results indicate variation in the items perceived as highly relevant to each setting and in the characteristics of classrooms including the degree of child autonomy, the types of activities, and in child/educator interactions and dialogue. However, despite this variation, a small set of items indicate potential functionality as cross-country anchor items. Findings lend support to the idea that quality measures can have some common elements with room for adaptation within and across settings. Future work in this area should address the possibility that the significance of these practices for child development also varies across settings.

Lack of voluntary interest and difficulty making eye contact are the most discriminative behaviors of the ASQ:SE and might suggest delays: Results from a large-scale assessment

Background Every child is unique, but development tends to occur in predictable steps and stages. The early identification of infants who face developmental delays is critical, boosting the use of screening tools to determine risks for delays. The city of Rio de Janeiro conducted a large-scale assessment of children who were enrolled in educational facilities using the Ages & Stages Questionnaires: Social-Emotional (ASQ:SE). Objectives We examined the internal structure of the ASQ:SE and its most discriminative items of risks of delays in development among 12- to 60-month-old children. The trajectory of the discrimination parameter of eight anchor items was used to check how well they inform the risk of social-emotional competence delays throughout development. Methods Data from 79,332 children (1–5 years) were analyzed via Samejima Graded Response model of Item Response Theory (IRT). The discrimination (a) and threshold (b) parameters were computed, and errors were achieved via maximum likelihood. Data/codes are available at https://osf.io/by6sf/. Results (a) Item Response Theory analyses supported the unidimensionality of data via the root mean square error of approximation and standardized root mean square residual results (RMSEA). (b) The lack of voluntary interest was the most discriminative risk behavior in the first 5 years. (c) Lack of interest was the most persistent risk behavior. (d) Difficulty making eye contact was nearly as informative as lack of interest. Conclusion Lack of voluntary interest in things should be considered a critical risk-related behavior, and making eye contact is a vital aspect of typical development. Both behaviors may be predictors of children’s delays. MAIN OUTCOMES The ASQ:SE is a valid and reliable tool to measure child development. The internal structure of the ASQ:SE is well-fitted with a unidimensional solution. A child’s age is a vital aspect of the discrimination parameter of the IRT model. Lack of interest in things and difficulty making eye contact are critical risk-related behaviors.

Crowdsourcing ratings for single lexical items

In this study, we investigate theoretical and practical issues connected to differentiating between core and peripheral vocabulary at different levels of linguistic proficiency using statistical approaches combined with crowdsourcing. We also investigate whether crowdsourcing second language learners’ rankings can be used for assigning levels to unseen vocabulary. The study is performed on Swedish single-word items. The four hypotheses we examine are: (1) there is core vocabulary for each proficiency level, but this is only true until CEFR level B2 (upper-intermediate); (2) core vocabulary shows more systematicity in its behavior and usage, whereas peripheral items have more idiosyncratic behavior; (3) given that we have truly core items (aka anchor items) for each level, we can place any new unseen item in relation to the identified core items by using a series of comparative judgment tasks, this way assigning a “target” level for a previously unseen item; and (4) non-experts will perform on par with experts in a comparative judgment setting. The hypotheses have been largely confirmed: In relation to (1) and (2), our results show that there seems to be some systematicity in core vocabulary for early to mid-levels (A1-B1) while we find less systematicity for higher levels (B2-C1). In relation to (3), we suggest crowdsourcing word rankings using comparative judgment with known anchor words as a method to assign a “target” level to unseen words. With regard to (4), we confirm the previous findings that non-experts, in our case language learners, can be effectively used for the linguistic annotation tasks in a comparative judgment setting.

Exploring Numeracy Skills of Lower Secondary School Students in Mountainous Areas of Northern Vietnam

In 2018, among 10 key competencies that the Vietnam Ministry of Education and Training has proposed in the new education curriculum, numeracy was identified as an essential competency that needs to be developed for students in the future. However, little is known about the numeracy of students in general, and of students in mountainous areas in particular. The aim of this study was to design an assessment of numeracy for students in the mountainous areas to understand their performance in numeracy. The study used the quantitative approach. A sample of 730 students in grades 6-8 from eight provinces in the mountainous areas of Northern Vietnam was recruited to participate in the study. A numeracy framework and three tests with anchor items were designed and developed to measure students’ numeracy. The results of the analysis using an approach to item response theory showed that the items had good fits with the model, and they could be used to describe numeracy learning progression with different levels of proficiency. The tests were reliable and valid, and the anchor items were good for connecting students’ competency among grades. The results also showed that Vietnamese students in mountainous areas tended to perform better in Arithmetic and Algebra problems than in real-life problems. The results provide convincing evidence of the practical performance in numeracy of students in various ethnic minority groups in Northern Vietnam.