Item Parameters Research Articles

Exploring the Effects of Item-Specific Factors in Sequential and IRTree Models.

Test items for which the item score reflects a sequential or IRTree modeling outcome are considered. For such items, we argue that item-specific factors, although not empirically measurable, will often be present across stages of the same item. In this paper, we present a conceptual model that incorporates such factors. We use the model to demonstrate how the varying conditional distributions of item-specific factors across stages become absorbed into the stage-specific item discrimination and difficulty parameters, creating ambiguity in the interpretations of item and person parameters beyond the first stage. We discuss implications in relation to various applications considered in the literature, including methodological studies of (1) repeated attempt items; (2) answer change/review, (3) on-demand item hints; (4) item skipping behavior; and (5) Likert scale items. Our own empirical applications, as well as several examples published in the literature, show patterns of violations of item parameter invariance across stages that are highly suggestive of item-specific factors. For applications using sequential or IRTree models as analytical models, or for which the resulting item score might be viewed as outcomes of such a process, we recommend (1) regular inspection of data or analytic results for empirical evidence (or theoretical expectations) of item-specific factors; and (2) sensitivity analyses to evaluate the implications of item-specific factors for the intended inferences or applications.

Using Item Scores and Distractors to Detect Test Speededness.

Test speededness refers to a situation in which examinee performance is inadvertently affected by the time limit of the test. Because speededness has the potential to severely bias both person and item parameter estimates, it is crucial that speeded examinees are detected. In this article, we develop a change-point analysis (CPA) procedure for detecting test speededness. Our procedure distinguishes itself from existing CPA procedures by using information from both item scores and distractors. Using detailed simulations, we show that under most conditions, the new CPA procedure improves the detection of speeded examinees and produces more accurate change-point estimates. It therefore seems there is a considerable amount of information to be gained from the item distractors, which, quite notably are available in all multiple-choice data. A real data example is also provided.

Measuring PROMIS pain interference in German patients with chronic conditions: calibration, validation, and cross-cultural use of item parameters

PurposeTo calibrate the item parameters of the German PROMIS® Pain interference (PROMIS PI) items using an item-response theory (IRT) model and investigate psychometric properties of the item bank.MethodsForty items of the PROMIS PI item bank were collected in a convenience sample of 660 patients, which were recruited during inpatient rheumatological treatment or outpatient psychosomatic medicine visits in Germany. Unidimensionality, monotonicity, and local independence were tested as required for IRT analyses. Unidimensionality was examined using confirmatory factor analyses (CFA) and exploratory factor analysis (EFA). Unidimensional and bifactor graded-response IRT models were fitted to the data. Bifactor indices were used to investigate whether multidimensionality would lead to biased scores. To evaluate convergent and discriminant validity, the item bank was correlated with legacy pain instruments. Potential differential item functioning (DIF) was examined for gender, age, and subsample. To investigate whether U.S. item parameters may be used to derive T-scores in German patients, T-scores based on previously published U.S. and newly estimated German item parameters were compared with each other after adjusting for sample specific differences.ResultsAll items were sufficiently unidimensional, locally independent, and monotonic. Whereas the fit of the unidimensional IRT model was not acceptable, a bifactor IRT model demonstrated acceptable fit. Explained common variance and Omega hierarchical suggested that using the unidimensional model would not lead to biased scores. One item demonstrated DIF between subsamples. High correlations with legacy pain instruments supported construct validity of the item bank. T-scores based on U.S. and German item parameters were similar suggesting that U.S. parameters could be used in German samples.ConclusionThe German PROMIS PI item bank proved to be a clinically valid and precise instrument for assessing pain interference in patients with chronic conditions.

The use of measurement frameworks to explore the qualities of test items

<span lang="EN-US">Psychometricians and researchers have grown increasingly concern about students’ performance. It is noteworthy from observation in the selected university of education in Nigeria that many of these students do not attend lectures for general courses particularly EDU 222 (tests and measurement) due to overcrowded lecture halls, the classroom climate factor, and other latent factors, such as exam conditions, test anxiety among others. Consequently therefore, students’ performance in EDU 222 does not significantly correlate with these challenges. The characteristics of the item parameter of the Tests and Measurement achievement test at the selected University of Education in Ogun State, Nigeria, were therefore examined in this study using measurement theories. The study employed an ex-post facto research of non-experimental design. Purposive sampling was used to select 6,203 second-year undergraduates offering EDU 222 for the 2018/2019 academic year. To analysis students’ responses to 30 multiple-choice questions on tests and measurement, factor analysis, item calibration, an independent sample t-test with a significance threshold of 0.05 were utilized with empirical reliability of 0.64. The results showed the test items satisfied the three-parameter logistics model and uni-dimensionality assumption. The findings also showed that the item response theory (IRT) eliminated 7 items due to guessing, while the classical test theory (CTT) eliminated 13 items. The test is prone to guessing, thus, helps pupils do better on the exam. To establish parameter estimates during test development, higher education institutions (HEIs) should employ the use of IRT.</span>

Matematik Başarısında Grup Farklılıklarının İncelenmesi: Açıklayıcı Madde Tepki Modeli Uygulaması

Students take many different exams throughout their educational lives. In these exams, various individual and item characteristics can affect the responses of individuals to the items. In this study, it was aimed to examine the effects of person and item predictors on the mathematics common exam results of 365 9th grade students with explanatory item response models. Gender and school type as person variables and cognitive domain, content domain and booklet type as item variables were added to the models due to their widespread inclusion in the literature. When the predicted item parameters were examined, it was seen that the smallest parameter values were obtained for all items with the Rasch model. When the model data fit values of four different models were examined, it was concluded that the latent regression and latent regression linear logistic test models showed better fit than the Rasch model. By adding person and item predictors to the model, the parameters obtained for each variable group were compared, and differences were observed between the groups for school type, cognitive domain, and content domain variables. It was concluded that the item parameters did not differ for the variables of gender and booklet type. It is thought that it would be beneficial to use these models more widely in studies to be conducted in the field of education and psychology since they provide more detailed information about the reasons for the differences in the estimated parameters.

Dinlediğini Anlama Testinin Türk Kültürüne Uyarlanması

Introduction: Listening, the first language skill developed by many children, serves as the basis of subsequent language skills. In addition, listening comprehension skills affect reading success in many different groups such as typically developing readers, children with learning disabilities and children with specific language disorders. The purpose of this research is to adapt the listening comprehension measurement tool to Turkish culture. Method: The purpose of this study is to examine the psychometric properties of the measurement tool within the framework of item response theory (IRT) through two-parameter logistic model (2PLM). Among model assumptions, confirmatory factor analysis (CFA) for unidimensionality Q3 statistics and item characteristic curv for the monotonic increase assumption showing the response probabilities conditional to the ability continuum were examined, respectively. The results for the assumptions were satisfactory. Then, general model-data fit, item-based model-data fit, discrimination parameters, difficulty parameters and information functions were examined with factor loadings. Findings: The results showed that item parameters were in the expected ranges in practical applications, and test information function peaked at lower ability level similar to the original measurement tool. Examining factor loadings, it was shown that they were higher than generally acceptable limit except for 23rd item. When contribution of this item to the model was tested statistically, it was concluded that it would be appropriate to include it in the measurement tool to keep the other items of associated with story block. Moreover, marginal reliability coefficient shows that the measurement results were highly reliable. Discussion: It was concluded that sufficient evidence of validity and reliability was obtained for one-dimensional structure of 36-item form of the measurement tool.

Examining the Psychometric Integrity of the Social Skills Improvement System Teacher Rating Scale Scores for a Sample of Preschool-Age Children

The Social Skills Improvement System–Teacher Rating Scale (SSIS-TRS) has been widely used to measure the social skills and behaviors of children and adolescents that are challenging. Studies examining the psychometric properties of the SSIS-TRS have been conducted, but the dimensional structure and item properties of the SSIS-TRS have not been extensively examined with samples of preschool-age children at risk for social-emotional and behavioral difficulties. This current study was designed to examine (a) the dimensional structure of the Social Skills scale and Problem Behaviors scale of the SSIS-TRS and (b) the item function of the SSIS-TRS using item response theory analyses with data collected from teachers of a sample of 469 preschool-age children at risk for social–emotional and behavioral difficulties. A four-dimensional structure (i.e., Cooperation, Empathy and Relationship, Engagement and Interaction, and Self-Control) and a three-dimensional structure (i.e., Aggression and Defiance, Disruptiveness and Stereotypy, and Withdrawal) were identified for the Social Skills and Problem Behaviors scales, respectively. Item parameters of the SSIS-TRS were estimated using the generalized partial credit model. Item information curves were plotted for individual items; behavior indicators providing higher information for measuring corresponding social skills and problem behaviors constructs were identified. The limitations and implications for the present study are discussed.

SAS PROC IRT and the R mirt Package: A Comparison of Model Parameter Estimation for Multidimensional IRT Models

This study investigates the performance of estimation methods for multidimensional IRT models with dichotomous and polytomous data in two well-known IRT programs: SAS PROC IRT and the mirt package in R. A simulation study was used to compare performance on a simple structure Rasch model, complex structure 2PL model, and bifactor graded response model. Under RMSE and bias criteria regarding item parameter recovery, PROC IRT and mirt showed nearly identical performance in the simple structure condition. When a complex structure was used, mirt performed better in terms of the recovery of intercept parameters, while the recovery of slope parameters depended on the program and the sample sizes: PROC IRT tended to be better with small samples (N=500) according to RMSE, and mirt was better for larger samples (N=1000 and 2500) according to RMSE and bias for the slope parameter recovery. When a bifactor structure was used, mirt was preferred in all cases; differences lessened as sample size increased.

Cognitive Diagnosis Testlet Model for Multiple-Choice Items

The testlet design is very popular in educational and psychological assessments. This article proposes a new cognitive diagnosis model, the multiple-choice cognitive diagnostic testlet (MC-CDT) model for tests using testlets consisting of MC items. The MC-CDT model uses the original examinees’ responses to MC items instead of dichotomously scored data (i.e., correct or incorrect) to retain information of different distractors and thus enhance the MC items’ diagnostic power. The Markov chain Monte Carlo algorithm was adopted to calibrate the model using the WinBUGS software. Then, a thorough simulation study was conducted to evaluate the estimation accuracy for both item and examinee parameters in the MC-CDT model under various conditions. The results showed that the proposed MC-CDT model outperformed the traditional MC cognitive diagnostic model. Specifically, the MC-CDT model fits the testlet data better than the traditional model, while also fitting the data without testlets well. The findings of this empirical study show that the MC-CDT model fits real data better than the traditional model and that it can also provide testlet information.

Chance-Constrained Automated Test Assembly

Classical automated test assembly (ATA) methods assume fixed and known coefficients for the constraints and the objective function. This hypothesis is not true for the estimates of item response theory parameters, which are crucial elements in test assembly classical models. To account for uncertainty in ATA, we propose a chance-constrained version of the maximin ATA model, which allows maximizing the α-quantile of the sampling distribution of the test information function obtained by applying the bootstrap on the item parameter estimation. A heuristic inspired by the simulated annealing optimization technique is implemented to solve the ATA model. The validity of the proposed approach is empirically demonstrated by a simulation study. The applicability is proven by using the real responses to the Trends in International Mathematics and Science Study (TIMSS) 2015 science test.

An Iterative Scale Purification Procedure on l z for the Detection of Aberrant Responses

Many person-fit statistics have been proposed to detect aberrant response behaviors (e.g., cheating, guessing). Among them, l z is one of the most widely used indices. The computation of l z assumes the item and person parameters are known. In reality, they often have to be estimated from data. The better the estimation, the better l z will perform. When aberrant behaviors occur, the person and item parameter estimations are inaccurate, which in turn degrade the performance of l z. In this study, an iterative procedure was developed to attain more accurate person parameter estimates for improved performance of l z. A series of simulations were conducted to evaluate the iterative procedure under two conditions of item parameters, known and unknown, and three aberrant response styles of difficulty-sharing cheating, random-sharing cheating, and random guessing. The results demonstrated the superiority of the iterative procedure over the non-iterative one in maintaining control of Type-I error rates and improving the power of detecting aberrant responses. The proposed procedure was applied to a high-stake intelligence test.

Applications and Extensions of Metric Stability Analysis

Item response theory models and applications are affected by many sources of variability, including errors associated with item parameter estimation. Metric stability analysis (MSA) is one method to evaluate the effects of item parameter standard errors that quantifies how well a model determines the latent trait metric. This paper describes how to evaluate MSA in dichotomous and polytomous data and describes a Bayesian implementation of MSA that does not require a positive definite variance–covariance matrix among item parameters. MSA analyses are illustrated in the context of an oral-health-related quality of life measure administered before and after prosthodontic treatment. The R code to implement the methods described in this paper is provided.

Cognitive domain harmonization and cocalibration in studies of older adults.

Studies use different instruments to measure cognitirating cognitive tests permit direct comparisons of individuals across studies and pooling data for joint analyses. We began our legacy item bank with data from the Adult Changes in Thought study (n = 5,546), the Alzheimer's Disease Neuroimaging Initiative (n = 3,016), the Rush Memory and Aging Project (n = 2,163), and the Religious on such as the Mini-Mental State Examination, the Alzheimer's Disease Assessment Scale-Cognitive Subscale, the Wechsler Memory Scale, and the Boston Naming Test. CocalibOrders Study (n = 1,456). Our workflow begins with categorizing items administered in each study as indicators of memory, executive functioning, language, visuospatial functioning, or none of these domains. We use confirmatory factor analysis models with data from the most recent visit on the pooled sample across these four studies for cocalibration and derive item parameters for all items. Using these item parameters, we then estimate factor scores along with corresponding standard errors for each domain for each study. We added additional studies to our pipeline as available and focused on thorough consideration of candidate anchor items with identical content and administration methods across studies. Prestatistical harmonization steps such qualitative and quantitative assessment of granular cognitive items and evaluating factor structure are important steps when trying to cocalibrate cognitive scores across studies. We have cocalibrated cognitive data and derived scores for four domains for 76,723 individuals across 10 studies. We have implemented a large-scale effort to harmonize and cocalibrate cognitive domain scores across multiple studies of cognitive aging. Scores on the same metric facilitate meta-analyses of cognitive outcomes across studies or the joint analysis of individual data across studies. Our systematic approach allows for cocalibration of additional studies as they become available and our growing item bank enables robust investigation of cognition in the context of aging and dementia. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Protocol for development of an assessment tool for competency of ECG interpretation: expert consensus by the RAND/UCLA appropriateness method and cross-sectional testing using multidimensional item response theory

IntroductionAlthough the ECG is an important diagnostic tool in medical practice, the competency of ECG interpretation is considered to be poor. Diagnostic inaccuracy involving the misinterpretation of ECG can lead...

Ceiling effects and differential measurement precision across calibrated cognitive scores in the Framingham Study.

To calibrate cognitive assessment data across multiple waves of the Framingham Heart Study (FHS), addressing study design considerations, ceiling effects, and measurement precision. FHS participants completed several cognitive assessments including screening instruments and more comprehensive batteries at different study visits. We used expert opinion to assign each cognitive test item to a single domain-memory, executive function, language, visuospatial abilities, or none of the above. As part of a larger cross-study harmonization effort, we calibrated each domain separately using bifactor confirmatory factor analysis (CFA) models, incorporating item parameters for anchor items previously calibrated from other studies and freely estimating item parameters for FHS-specific items. We obtained scores and standard errors (SEs) for each participant at each study visit. We addressed psychometric considerations of ceiling effects and measurement precision. Overall, memory domain scores were the most precisely estimated. Scores for all domains from visits where the Mini-Mental State Examination (MMSE) was the only test administered were imprecisely estimated and suffered from ceiling effects. Scores from visits with a more extensive battery were estimated more precisely and better differentiated between ability levels. The harmonized and calibrated cognitive data from the FHS should prove useful for future analyses examining cognition and cognitive decline. They will be of particular interest when combining FHS with other studies that have been similarly calibrated. Researchers should be aware of varying levels of measurement precision and the possibility of ceiling effects in their planned analyses of data from the FHS and similar studies. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Identifying Disengaged Responding in Multiple-Choice Items: Extending a Latent Class Item Response Model With Novel Process Data Indicators.

Disengaged responding poses a severe threat to the validity of educational large-scale assessments, because item responses from unmotivated test-takers do not reflect their actual ability. Existing identification approaches rely primarily on item response times, which bears the risk of misclassifying fast engaged or slow disengaged responses. Process data with its rich pool of additional information on the test-taking process could thus be used to improve existing identification approaches. In this study, three process data variables-text reread, item revisit, and answer change-were introduced as potential indicators of response engagement for multiple-choice items in a reading comprehension test. An extended latent class item response model for disengaged responding was developed by including the three new indicators as additional predictors of response engagement. In a sample of 1,932 German university students, the extended model indicated a better model fit than the baseline model, which included item response time as only indicator of response engagement. In the extended model, both item response time and text reread were significant predictors of response engagement. However, graphical analyses revealed no systematic differences in the item and person parameter estimation or item response classification between the models. These results suggest only a marginal improvement of the identification of disengaged responding by the new indicators. Implications of these results for future research on disengaged responding with process data are discussed.

Why Full, Partial, or Approximate Measurement Invariance Are Not a Prerequisite for Meaningful and Valid Group Comparisons

It is frequently stated in the literature that measurement invariance is a prerequisite for the comparison of group means or standard deviations of the latent variable in factor models. This article argues that measurement invariance is not necessary for meaningful and valid comparisons across groups. There is unavoidable ambiguity in how researchers can define comparisons if measurement invariance is violated. Moreover, there is no support for preferring the partial invariance approach over competing approaches, such as invariance alignment, robust linking, or Bayesian approximate invariance. Furthermore, we also argue why an intentionally misspecified multiple-group factor model with invariant item parameters can be justified if measurement invariance is violated.

Exploring Approaches for Estimating Parameters in Cognitive Diagnosis Models with Small Sample Sizes

Cognitive diagnostic models (CDMs) are increasingly being used in various assessment contexts to identify cognitive processes and provide tailored feedback. However, the most commonly used estimation method for CDMs, marginal maximum likelihood estimation with Expectation–Maximization (MMLE-EM), can present difficulties when sample sizes are small. This study compares the results of different estimation methods for CDMs under varying sample sizes using simulated and empirical data. The methods compared include MMLE-EM, Bayes modal, Markov chain Monte Carlo, a non-parametric method, and a parsimonious parametric model such as Restricted DINA. We varied the sample size, and assessed the bias in the estimation of item parameters, the precision in attribute classification, the bias in the reliability estimate, and computational cost. The findings suggest that alternative estimation methods are preferred over MMLE-EM under low sample-size conditions, whereas comparable results are obtained under large sample-size conditions. Practitioners should consider using alternative estimation methods when working with small samples to obtain more accurate estimates of CDM parameters. This study aims to maximize the potential of CDMs by providing guidance on the estimation of the parameters.

A Comparison of Response Time Threshold Scoring Procedures in Mitigating Bias From Rapid Guessing Behavior.

Rapid guessing (RG) is a form of non-effortful responding that is characterized by short response latencies. This construct-irrelevant behavior has been shown in previous research to bias inferences concerning measurement properties and scores. To mitigate these deleterious effects, a number of response time threshold scoring procedures have been proposed, which recode RG responses (e.g., treat them as incorrect or missing, or impute probable values) and then estimate parameters for the recoded dataset using a unidimensional or multidimensional IRT model. To date, there have been limited attempts to compare these methods under the possibility that RG may be misclassified in practice. To address this shortcoming, the present simulation study compared item and ability parameter recovery for four scoring procedures by manipulating sample size, the linear relationship between RG propensity and ability, the percentage of RG responses, and the type and rate of RG misclassifications. Results demonstrated two general trends. First, across all conditions, treating RG responses as incorrect produced the largest degree of combined systematic and random error (larger than ignoring RG). Second, the remaining scoring approaches generally provided equal accuracy in parameter recovery when RG was perfectly identified; however, the multidimensional IRT approach was susceptible to increased error as misclassification rates grew. Overall, the findings suggest that recoding RG as missing and employing a unidimensional IRT model is a promising approach.

Study Item Parameters of Classical and Modern Theory of Differential Aptitude Test: Is it Comparable?

<p style="text-align: justify;">This study aimed to find the Classical Test Theory (CTT) and Modern Test Theory (MTT) item parameters of the Differential Aptitude Test (DAT) and examined their comparability of them. The item parameters being studied are difficulty level and discrimination index. 5.024 data of the result sub-test DAT were documented by the Department of Psychology and Guidance and Counselling bureau. The parameter of classical and modern test items was estimated and correlated by examining the comparability between parameters. The results show that there is a significant correlation between item parameter estimates. The Rasch and IRT 1-PL models have the highest correlation toward CTT regarding the item difficulty level. In contrast, model 2-PL has the highest correlation toward CTT in the item discrimination index. Overall, the study concluded that CTT and MTT were comparable in estimating item parameters of DAT and thus could be used independently or complementary in developing DAT.</p>