Item Parameters Research Articles

Examining Measurement Invariance in Bayesian Item Response Theory Models: A Simulation Study

The aim of the study is to determine a measurement invariance cut-off point based on item parameter differences in Bayesian Item Response Theory Models. Within this scope, the Bayes factor is estimated for testing measurement invariance. For this purpose, a simulation study is conducted. The data were generated in the R software for each simulation condition under the one-parameter logistic model for 10 binary (1-0 scored) items. The invariance test was performed for various group sizes (n=500, 1000, 1500 and 2000) and difficulty parameters (dk=0, dk=0.1, dk=0.3, dk=0.5 and dk=0.7). The Bayesian analyzes were performed on the WINBUGS using the codes written in the R. A Bayes factor that provides evidence for measurement invariance was calculated depending on the parameter differences. The Savage–Dickey density ratio, one of the MCMC sampling schemas, was used to calculate the Bayes factor. As a result, if the item parameter difference is dk=0.3 and group sizes are 1500 or larger, the measurement invariance cannot be achieved. However, for small sample sizes (n=1000 or less) measurement invariance interpretation should be done carefully. When the dk=0.5, there are invariant items only in n=500. According to Bayes factor test results, evidence has been produced when dk=0.7 that measurement invariance cannot be achieved.

The Effects of Different Item Selection Methods on Test Information and Test Efficiency in Computer Adaptive Testing

The purpose of this study is to examine the effect of different item selection methods on test information function (TIF) and test efficiency in computer adaptive testing (CAT). TIF indicates the quantity of information the test has produced. Test efficiency resembles the amount of information from each item, and more efficient tests are produced from the smallest number of good-quality items. The study was conducted with simulated data, and the constants of the study are sample size, ability parameter distribution, item pool size, model of item response theory (IRT) and distribution of item parameters, ability estimation method, starting rule, item exposure control and stopping rule. The item selection methods, which are the independent variables of this study, are the interval information criterion, efficiency balanced information, matching -b value, Kullback-Leibler information, maximum fisher information, likelihood-weighted information, and random selection. In the comparison of these methods, the best performance in the aspect of TIF is provided by the maximum fisher information method. In terms of test efficiency, the performances of the methods were similar, except for the random selection method, which had the worst performance in terms of both TIF and test efficiency.

Pretest Item Calibration in Computerized Multistage Adaptive Testing

AbstractThe purpose of this study was to compare calibration and linking methods for placing pretest item parameter estimates on the item pool scale in a 1‐3 computerized multistage adaptive testing design in terms of item parameter recovery. Two models were used: embedded‐section, in which pretest items were administered within a separate module, and embedded‐items, in which pretest items were distributed across operational modules. The calibration methods were separate calibration with linking (SC) and fixed calibration (FC) with three parallel approaches under each (FC‐1 and SC‐1; FC‐2 and SC‐2; and FC‐3 and SC‐3). The FC‐1 and SC‐1 used only operational items in the routing module to link pretest items. The FC‐2 and SC‐2 also used only operational items in the routing module for linking, but in addition, the operational items in second stage modules were freely estimated. The FC‐3 and SC‐3 used operational items in all modules to link pretest items. The third calibration approach (i.e., FC‐3 and SC‐3) yielded the best results. For all three approaches, SC outperformed FC in all study conditions which were module length, sample size and examinee distributions.

A Monte Carlo study of IRTree models' ability to recover item parameters.

Item response tree (IRTree) models are theorized to extract response styles from self-report data by utilizing multidimensional item response theory (IRT) models based on theoretical decision processes. Despite the growing popularity of the IRTree framework, there has been little research that has systematically examined the ability of its most popular models to recover item parameters across sample size and test length. This Monte Carlo simulation study explored the ability of IRTree models to recover item parameters based on data created from the midpoint primary process model. Results indicate the IRTree model can adequately recover item parameters early in the decision process model, specifically the midpoint node. However, as the model progresses through the decision hierarchy, item parameters have increased associated error variance. The authors ultimately recommend caution when employing the IRTree framework.

A general nonparametric classification method for multiple strategies in cognitive diagnostic assessment.

Cognitive diagnosis models (CDMs) have been used as psychometric tools in educational assessments to estimate students' strengths and weaknesses in terms of cognitive skills learned and skills that need study. In practice, it is not uncommon that questions can often be solved using more than one strategy, which requires CDMs capable of accommodating multiple strategies. However, existing parametric multi-strategy CDMs need a large sample size to produce a reliable estimation of item parameters and examinees' proficiency class memberships, which obstructs their practical applications. This article proposes a general nonparametric multi-strategy classification method with promising classification accuracy in small samples for dichotomous response data. The method can accommodate different strategy selection approaches and different condensation rules. Simulation studies showed that the proposed method outperformed the parametric CDMs when sample sizes were small. A set of real data was analyzed as well to illustrate the application of the proposed method in practice.

A classification model for continuous responses: Identifying risk perception groups on health-related activities.

In the current literature on latent variable models, much effort has been put on the development of dichotomous and polytomous cognitive diagnostic models (CDMs) for assessments. Recently, the possibility of using continuous responses in CDMs has been brought to discussion. But no Bayesian approach has been developed yet for the analysis of CDMs when responses are continuous. Our work is the first Bayesian framework for the continuous deterministic inputs, noisy, and gate (DINA) model. We also propose new interpretations for item parameters in this DINA model, which makes the analysis more interpretable than before. In addition, we have conducted several simulations to evaluate the performance of the continuous DINA model through our Bayesian approach. Then, we have applied the proposed DINA model to a real data example of risk perceptions for individuals over a range of health-related activities. The application results exemplify the high potential of the use of the proposed continuous DINA model to classify individuals in thestudy.

Comparison of disengagement levels and the impact of disengagement on item parameters between PISA 2015 and PISA 2018 in the United States

BackgroundExaminees may not make enough effort when responding to test items if the assessment has no consequence for them. These disengaged responses can be problematic in low-stakes, large-scale assessments because they can bias item parameter estimates. However, the amount of bias, and whether this bias is similar across administrations, is unknown. This study compares the degree of disengagement (i.e., fast and non-effortful responses) and the impact of disengagement on item parameter estimates in the Programme for International Student Assessment (PISA) across the 2015 and 2018 administrations.MethodWe detected disengaged responses at the item level based on response times and response behaviors. We used data from the United States and analyzed 51 computer-based mathematics items administered in both PISA 2015 and PISA 2018. We compared the percentage of disengaged responses and the average scores of the disengaged responses for the 51 common items. We filtered disengaged responses at the response- and examinee-levels and compared item difficulty (P+ and b) and item discrimination (a) before and after filtering.ResultsOur findings suggested that there were only slight differences in the amount of disengagement in the U.S. results for PISA 2015 and PISA 2018. In both years, the amount of disengagement was less than 5.2%, and the average scores of disengaged responses were lower than the average scores of engaged responses. We did not find any serious impact of disengagement on item parameter estimates when we applied response-level filtering; however, we found some bias, particularly on item difficulty, when we applied examinee-level filtering.ConclusionsThis study highlights differences in the amount of disengagement in PISA 2015 and PISA 2018 as well as the implications of the decisions made for handling disengaged responses on item difficulty and discrimination. The results of this study provide important information for reporting trends across years.

Comparison of the methods of examining measurement equivalence under different conditions in terms of statistical power ratios

Validity is the most important psychometric feature that should be found in a measurement tool. Measurement equivalence is one of the evidences for the validity of measurement tools. Providing information to the researchers about the methods of identifying measurement equivalence may contribute to present a complete validity evidence. The purpose of this study is to compare the statistical power ratios of methods used for examining the measurement equivalence based on structural equation modeling and item response theory on the artificial data sets generated by diversifying variables of sample size, number of items, and the ratios of the items having differential item function. In accordance with this purpose, the variables have been varied to include three different levels. In the analysis, multi-group confirmatory factor analysis was used which is among the methods based on the structural equation modeling, and likelihood ratio test and comparison of the item parameters methods were used which are among the methods based on item response theory. Multi-group group confirmatory factor analysis (93,50%) and likelihood ratio test (96,75%) methods have reached the highest statistical power ratio in the condition that the sample size is 1000/1000, the number of items is 40 items and the ratios of the items having differential item function is 10%. The method of comparing the item parameters (94,50%) has reached the highest statistical power ratio in the condition of sample size is 1000/1000, the number of items is 20 and the ratios of the items having differential item function are 10-20%.

Scale Development Based on Item Response Theory: A Systematic Review

Classical test theory (CTT) and item response theory (IRT) are two fundamental approaches used in scale development research. Although CTT is the preferred methodology in scale development research in the Republic of Turkey, the IRT methodology has started to gain traction in recent years. In this study, we used the systematic review methodology to review scale development research in Turkey that used the IRT procedures. Results revealed that most of the studies were conducted in 2017. Most of the studies have a sufficient sample size. Exploratory factor analysis was used more than confirmatory factor analysis. Some studies failed to choose the proper IRT model, report and interpret item parameters, check IRT assumptions, and report and interpret information functions. We recommend some practical considerations to researchers in terms of IRT assumptions.

A Likelihood Approach to Item Response Theory Equating of Multiple Forms.

Test equating is a statistical procedure to make scores from different test forms comparable and interchangeable. Focusing on an IRT approach, this paper proposes a novel method that simultaneously links the item parameter estimates of a large number of test forms. Our proposal differentiates itself from the current state of the art by using likelihood-based methods and by taking into account the heteroskedasticity and the correlation of the item parameter estimates of each form. Simulation studies show that our proposal yields equating coefficient estimates which are more efficient than what is currently available in the literature.

Construction of a computerized adaptive test (CAT-CCNB) for efficient neurocognitive and clinical psychopathology assessment

BackgroundTraditional paper-and-pencil neurocognitive evaluations and semi-structured mental health interviews can take hours to administer and score. Computerized assessment has decreased that burden substantially, and contemporary psychometric tools such as item response theory and computerized adaptive testing (CAT) allow even further abbreviation. New methodThe goal of this paper was to describe the application of CAT and related methods to the Penn Computerized Neurocognitive Battery (CNB) and a well-validated clinical assessment in order to increase efficiency in assessment and relevant domain coverage. To calibrate item banks for CAT, N = 5053 participants (63% female; mean age 45 years, range 18–80) were collected from across the United States via crowdsourcing, providing item parameters that were then linked to larger item banks and used in individual test construction. Tests not amenable to CAT were abbreviated using complementary short-form methods. ResultsThe final “CAT-CCNB” battery comprised 21 cognitive tests (compared to 14 in the original) and five adaptive clinical scales (compared to 16 in the original). Comparison with existing methodsThis new battery, derived with contemporary psychometric approaches, provides further improvements over existing assessments that use collections of fixed-length tests developed for stand-alone administration. The CAT-CCNB provides an improved version of the CNB that shows promise as a maximally efficient tool for neuropsychiatric assessment. ConclusionsWe anticipate CAT-CCNB will help satisfy the clear need for broad yet efficient measurement of cognitive and clinical domains, facilitating implementation of large-scale, “big science” approaches to data collection, and potential widespread clinical implementation.

What Affects the Quality of Score Transformations? Potential Issues in True-Score Equating Using the Partial Credit Model.

This simulation study investigated to what extent departures from construct similarity as well as differences in the difficulty and targeting of scales impact the score transformation when scales are equated by means of concurrent calibration using the partial credit model with a common person design. Practical implications of the simulation results are discussed with a focus on scale equating in health-related research settings. The study simulated data for two scales, varying the number of items and the sample sizes. The factor correlation between scales was used to operationalize construct similarity. Targeting of the scales was operationalized through increasing departure from equal difficulty and by varying the dispersion of the item and person parameters in each scale. The results show that low similarity between scales goes along with lower transformation precision. In cases with equal levels of similarity, precision improves in settings where the range of the item parameters is encompassing the person parameters range. With decreasing similarity, score transformation precision benefits more from good targeting. Difficulty shifts up to two logits somewhat increased the estimation bias but without affecting the transformation precision. The observed robustness against difficulty shifts supports the advantage of applying a true-score equating methods over identity equating, which was used as a naive baseline method for comparison. Finally, larger sample size did not improve the transformation precision in this study, longer scales improved only marginally the quality of the equating. The insights from the simulation study are used in a real-data example.

Linking Error in the 2PL Model

The two-parameter logistic (2PL) item response model is likely the most frequently applied item response model for analyzing dichotomous data. Linking errors quantify the variability in means or standard deviations due to the choice of items. Previous research presented analytical work for linking errors in the one-parameter logistic model. In this article, we present linking errors for the 2PL model using the general theory of M-estimation. Linking errors are derived in the case of log-mean-mean linking for linking two groups. The performance of the newly proposed formulas is evaluated in a simulation study. Furthermore, the linking error estimation in the 2PL model is also treated in more complex settings, such as chain linking, trend estimation, fixed item parameter calibration, and concurrent calibration.

Accelerating item factor analysis on GPU with Python package xifa.

Item parameter estimation is a crucial step when conducting item factor analysis (IFA). From the view of frequentist estimation, marginal maximum likelihood (MML) seems to be the gold standard. However, fitting a high-dimensional IFA model by MML is still a challenging task. The current study demonstrates that with the help of a GPU (graphics processing unit) and carefully designed vectorization, the computational time of MML could be largely reduced for large-scale IFA applications. In particular, a Python package called xifa (accelerated item factor analysis) is developed, which implements a vectorized Metropolis-Hastings Robbins-Monro (VMHRM) algorithm. Our numerical experiments show that the VMHRM on a GPU may run 33 times faster than its CPU version. When the number of factors is at least five, VMHRM (on GPU) is much faster than the Bock-Aitkin expectation maximization, MHRM implemented by mirt (on CPU), and the importance-weighted autoencoder (on GPU). The GPU-implemented VMHRM is most appropriate for high-dimensional IFA with large data sets. We believe that GPU computing will play a central role in large-scale psychometric modeling in the near future.

Utrecht Work Engagement Scale (UWES): Psychometric parameters in Brazil

Introduction: The study of work engagement is key for better understanding the underlying mechanisms that lead people to feel more motivated at work. The construct has gained prominence over recent decades, and the Utrecht Work Engagement Scale (UWES) has emerged as the most popular tool to assess the construct. Though widely known and used, more psychometric evidence is needed for the UWES, especially its item parameters. Method: This study (N = 525) aimed to provide psychometric evidence for the UWES and its shortened versions in Brazil, using a range of robust statistical analyses (e.g., Confirmatory Factor Analysis, Item Response Theory). Results: Results reveal good model fit and high internal consistency for both unidimensional and three-dimensional UWES structures. UWES items also showed high discrimination, difficulty, and information levels. Finally, significant correlations between UWES and workaholism and job satisfaction provided evidence of the convergent validity of the UWES in Brazil. Conclusion: This study’s findings broaden the understanding of work engagement and underscore the utility of the UWES as an efficacious tool for measuring work engagement in Brazil, paving the way for effective interventions and policies in diverse workplace environments.

The numeric understanding measures: Developing and validating adaptive and nonadaptive numeracy scales

Abstract Numeracy—the ability to understand and use numeric information—is linked to good decision-making. Several problems exist with current numeracy measures, however. Depending on the participant sample, some existing measures are too easy or too hard; also, established measures often contain items well-known to participants. The current article aimed to develop new numeric understanding measures (NUMs) including a 1-item (1-NUM), 4-item (4-NUM), and 4-item adaptive measure (A-NUM). In a calibration study, 2 participant samples (n = 226 and 264 from Amazon’s Mechanical Turk [MTurk]) each responded to half of 84 novel numeracy items. We calibrated items using 2-parameter logistic item response theory (IRT) models. Based on item parameters, we developed the 3 new numeracy measures. In a subsequent validation study, 600 MTurk participants completed the new numeracy measures, the adaptive Berlin Numeracy Test, and the Weller Rasch-Based Numeracy Test, in randomized order. To establish predictive and convergent validities, participants also completed judgment and decision tasks, Raven’s progressive matrices, a vocabulary test, and demographics. Confirmatory factor analyses suggested that the 1-NUM, 4-NUM, and A-NUM load onto the same factor as existing measures. The NUM scales also showed similar association patterns to subjective numeracy and cognitive ability measures as established measures. Finally, they effectively predicted classic numeracy effects. In fact, based on power analyses, the A-NUM and 4-NUM appeared to confer more power to detect effects than existing measures. Thus, using IRT, we developed 3 brief numeracy measures, using novel items and without sacrificing construct scope. The measures can be downloaded as Qualtrics files (https://osf.io/pcegz/).

Escala de Avaliação da Síndrome de Burnout na Segurança Pública: Parâmetros Psicométricos

ABSTRACT The burnout variable has been studied in different professions; however, it presents challenges in its measurement. The present study provides evidence of validity based on the internal structure of the Burnout Syndrome Assessment Scale in Public Security Workers (EASB-SP) and presents the estimation of item parameters through the Item Response Theory (IRT). A total of 10,885 public security agents from all states of Brazil participated in the study. The analyses indicated a three-factor structure, with adequate reliability indices and discrimination of the items. The Professional Fulfillment factor presented a lower theta value to be endorsed. The instrument was shown to be an adequate measure, with evidence of validity for the context of Brazilian public security.

Propriedades Psicométricas da Escala Multidimensional de Perfeccionismo de Hewitt e Flett para Adultos Brasileiros

Abstract The aim of the present study was to investigate validity evidence based on internal structure and the relationship with other variables of the Brazilian version of the Hewitt and Flett Multidimensional Perfectionism Scale (H&F-MPS). To this end, a sample comprising 368 participants with a mean age of 31.17 years (SD=11,34), 74% female, answered the H&F-MPS perfectionism scale and instruments that measure personality and psychological inflexibility. Findings indicated a factor structure of three dimensions, like the original scale, good discriminative item parameters, highly satisfactory reliability indexes, and significant correlations of the hypothesized directions with Openness to Experiences, Neuroticism, and Conscientiousness factors from the personality measure, as well as with the psychological flexibility measure.

Instrument validation of perceived organizational support (POS) indonesian version: A rasch model analysis

The Perceived Organizational Support scale developed by Eisenberg is still quite widespread in Indonesia. The POS scale precisely measures how employees perceive organizational support. On the other hand, analysis of items on the POS scale continues. However, this scale analysis has yet to be found using the Rasch Model with a sample of Indonesian people. One of the advantages of the Rasch model is objectivity-specific, i.e., item parameters do not depend on individual characteristics, and vice versa. So this study aims to analyze the validity of the items using the Rasch model. The sample in this study consisted of 325 people consisting of 175 men (52.31%) and 150 women (47.69%). The age range of 18 years – to 68 years. From the item validation test, this scale has good unidimensionality, a standardized reliability index so that the scale is reliable enough, and has items that fall into the fit proper test criteria. The conclusion is that the POS scale has good potential to measure Perceived Organizational Support in Indonesia.

An iterative two-step method for online item calibration in CD-CAT.

The development and maintenance of the item bank is a critical element to a CD-CAT (cognitive diagnostic computerized adaptive testing; Cheng, 2009) system. For continuous testing, it is important to replenish the item bank with new items that have been calibrated. This requires pretesting to estimate the parameters of the new items. For CD-CAT, the structural parameters that need to be estimated include both item parameters and attribute vectors. In this paper, we propose three residual-statistic-based methods: RMA, ROEM, and RMEM, to estimate the attribute vectors and item parameters all together for new items. An iterative two-step online calibration procedure is developed to estimate the attribute vectors for the new items in the first step, and estimate the item parameters in the second step, then proceed iteratively until convergence is reached. An extensive simulation study was conducted to evaluate the performance of the three proposed methods and compare them with two existing methods, namely the Joint Estimation Algorithm (JEA; Chen & Xin, 2011) and Single Item Estimation (SIE; Chen et al., 2015) methods. In terms of the estimation of the attribute vector, the RMEM method performs the best in most of the cases. In terms of item parameter estimation, RMEM still has some advantages, and RMA outperforms JEA and SIE. Taken together, results suggest that the RMEM is superior to the other methods, especially when sample size is relatively small. A real-data example is provided to illustrate the application of RMEM in practice.