Item Response Theory Software Research Articles

Construction of a reading literacy test item bank for fourth graders based on item response theory.

Reading literacy is not only central to students' academic success during their school years but also crucial to their personal development in their later life. The development of assessment instruments for reading literacy has been of interest to researchers, educators and educational administrators. The purpose of the present study was to construct and validate a comparable item bank for assessing fourth-grade students' reading literacy. One hundred fifteen reading comprehension items were developed and administered to 2,174 Grade 4 students to construct an item bank. Using the test equating technique and balanced incomplete block design, we divided participants into 10 subgroups, and the 115 items were further assigned into 10 test forms. Item response theory software was used to estimate discrimination, items' threshold parameters, and students' ability parameters. The criterion-related validity was also examined in 135 Grade 4 students who completed the reading literacy test and verbal self-description questionnaire. The final item bank included 99 reading performance indicators to express high achievement. The correlation between the students' reading literacy and the verbal self-description questionnaire was significant and demonstrated the item bank's good criterion-related validity. The item bank developed in this study shows good psychometric characteristics and can be used to assess the reading literacy of fourth graders.

[RETRACTED ARTICLE] irtplay: An R Package for Unidimensional Item Response Theory Modeling

Item response theory (IRT) is a general framework in which mathematical models are formulated to explain the relationship between an examinee's observable response on an item and the latent ability measured by a test. The application of IRT models and related statistical methods are commonly found in educational and psychological research. An important step in applying IRT models to test data is estimating the IRT model parameters. Accordingly, the successful application of IRT rests on the satisfactory statistical techniques and software for accurately estimating the model parameters. The irtplay R package was developed to provide users with a user-friendly experience and convenience when analyzing test data using unidimensional IRT models. The package can be used to fit the IRT models to a mixture of dichotomous and polytomous item data using marginal maximum likelihood estimation via the expectation-maximization, calibrate pretest items, and estimate examinees' latent ability parameters. In addition, the package provides practical tools that conveniently enable users to conduct many analyses related to IRT such as evaluating IRT model-data fit, analyzing differential item functioning, computing asymptotic variance-covariance matrices of item parameter estimates, calculating the conditional probability distribution of observed scores using the Lord and Wingersky (1984) formula, and importing item and ability parameter estimates from the output of popular IRT software. The main features of the irtplay package are illustrated using three data examples.

OpenMx: A Modular Research Environment for Item Response Theory Method Development:

There are many item response theory software packages designed for users. Here, the authors introduce an environment tailored to method development and simulation. Implementations of a selection of...

Irtplay: An R Package for Online Item Calibration, Scoring, Evaluation of Model Fit, and Useful Functions for Unidimensional IRT.

The R package irtplay provides practical tools for unidimensional item response theory (IRT) models that conveniently enable users to conduct many analyses related to IRT. For example, the irtplay includes functions for calibrating online items, scoring test-takers' proficiencies, evaluating IRT model-data fit, and importing item and/or proficiency parameter estimates from the output of popular IRT software. In addition, the irtplay package supports mixed-item formats consisting of dichotomous and polytomous items.

PCN244 UNBIASED TREATMENT EFFICACY DETECTION METHODS WHEN USING PATIENT REPORTED OUTCOMES IN SURVIVAL-DEPENDENT DROP-OUT

Missing patient reported outcome (PRO) data can arise from several processes. In the context of survival designs, the missingness may very well be outcome-dependent. This type of missing data is known as missing not at random (MNAR). In this circumstance, the PRO scores will produce biased treatment efficacy estimates. This bias can lead to the incorrect inference of no difference in treatment efficacy across arms. Outcome-dependent missingness poses serious challenges and requires the use of statistical approaches that properly adjust for this type of missing data. Currently, the state of the art in modeling PROs in a survival analysis context is to simultaneously model observed scores and the drop-out mechanism. This is referred to as a joint process model. However, interest in item response theory (IRT) approaches has grown in recent years. An IRT-based framework has been developed for evaluating PRO-based scores in survival analysis context. A longitudinal IRT model that incorporates the drop-out mechanism has been created. This allows the IRT scores and the drop-out mechanism to be modeled simultaneously. A simulation study based on empirical PRO data is employed here to illustrate the advantages of such an approach. The results of the simulation study show that the IRT approach to jointly model Quality of Life (QoL) scores and drop-out allows for the accurate estimation of separation of the treatment arms. Other approaches, such as a longitudinal IRT model that ignores drop-out, fail to detect separation. An IRT-based method has been developed that can better estimate the treatment effect in a survival analysis context. Notably, this approach can be implemented in standard commercial IRT software.

PRM106 - Calculating Rasch-Based Person Parameters for Individual Items

The Rasch model is the most constrained of the item response theory (IRT) models and is often used for analyzing categorical data based on (a) the respondent's trait levels (e.g., their disease severity) and (b) the item difficulty (i.e., how unlikely it is for a given disease characteristic to be reported). Rasch data are calculated at the domain level, and may sometimes be appropriate for a total score. We can examine more specific information by looking at individual scores on the entire domain or item-level data for all individuals at once. There are times, however, when one may wish to examine scores for each individual on a single item. For example, one may wish to examine the influence of changing a raw-score scale range by comparing it to Rasch data for the same item. Unfortunately, none of the current IRT software permits these calculations. The goal of the current study was to refine the process for such analyses, thereby creating an Excel worksheet, and make them available for others. In 1998, Linacre [1] published the details of how to calculate individual-level Rasch parameters for each item of a domain. As the formulae are cumbersome and lengthy, an easy to use Excel worksheet was created. The Excel worksheet was first reviewed by members of our team and then tested with data from a depression patient-reported outcome measure. Qualitative and quantitative review of the worksheet found that, beyond being easy to use, it was an accurate and appropriate representation of the Linacre formulae. Programming details and Excel layout are provided in the poster presentation and will be made available from the first author. Linacre, J.M., Estimating Rasch measures with known polytomous item difficulties. Rasch Measurement Transactions, 1998. 12: p. 638.

Item level diagnostics and model - data fit in item response theory (IRT) using BILOG - MG v3.0 and IRTPRO v3.0 programmes

Item response theory (IRT) is a framework for modeling and analyzing item response data. Item-level modeling gives IRT advantages over classical test theory. The fit of an item score pattern to an item response theory (IRT) models is a necessary condition that must be assessed for further use of item and models that best fit the data. The study investigated item level diagnostic statistics and model- data fit with one-and two- parameter models using IRTPROV3.0 and BILOG- MG V3.0. Ex-post facto design was adopted. The population for the study consisted of 11,538 candidates’ responses who took Type L 2014 Unified Tertiary Matriculation Examination (UTME) Mathematics paper in Akwa Ibom State, Nigeria. The sample of 5,192(45%) responses was randomly selected through stratified sampling technique. BILOG-MG V3.0 and IRTPROV3.0 computer software was used to calibrate the candidates’ responses. Two research questions were raised to guide the study. Pearson’s χ2 and S - χ2 statistics as an item fit index for dichotomous item response theory models were used. The outputs from the two computer software were used to answer the questions. The findings revealed that only 1 item fitted 1- parameter model in BILOG- MG V3.0 and IRTPRO V3.0. Furthermore, the findings revealed that 26 items fitted 2-parameter models when using BILOG-MG V3.0. Five items fitted 2-parameter models in IRTPRO. It was recommended that the use of more than one IRT software programme offers more useful information for the choice of model that fit the data.KEYWORDS: Item Level, Diagnostics, Statistics, Model - Data Fit, Item Response Theory (IRT).

Implementation of Item Response Theory for Analysis of Test Items Quality and Students’ Ability in Chemistry

<p>This first aim of this study is to describe the quality of chemistry test item made by teacher. The test was developed for 11<sup>th</sup> grade students’ science class in the first semester on academic year 2015/2016. The second aim of this study is to describe the characteristic of measurement’s result for students’ ability in chemistry. This is descriptive research design with the 101 student’s responses patterns from multiple choice test device with 5 answer alternatives. The responses patterns were collected by documentation technique and analyzed quantitatively using Item Response Theory software such as BILOG MG V3.0 with 1-PL, 2-PL, and 3-PL models. The differences of students’ ability in chemistry in model 1-PL, 2-PL, dan 3-PL were analyzed using One-Way Anova Repeated Measure. The result showed that the mean of item difficulties level (b), item differentiate (a), and pseudo-guessing (c) are good. The measurement tools arranged by teacher were suitable for students who have the ability from -1.0 to +1.7. The maximum score of item information function is 68.83 (SEM =0.121) with ability in 0.2 logit. The highest ability’s estimation score was showed by Model 2-PL. The mean of students’ ability for 11<sup>th</sup> grade students is -0.0185 logit and consider as moderate category.</p><p> </p>

A comparative analysis of pre-equating and post-equating in a large-scale assessment, high stakes examination

Statistical procedure used in adjusting test score difficulties on test forms is known as “equating”. Equating makes it possible for various test forms to be used interchangeably. In terms of where the equating method fits in the assessment cycle, there are pre-equating and post-equating methods. The major benefits of pre-equating, when applied, are that it facilitates the operational processes of examination bodies in terms of rapid score reporting, quality control and flexibility in the assessment process. The purpose of this study is to ascertain if pre- and post-equating results are comparable. Data for this study, which adopted an equivalent group design method, was taken from the 2012 Unified Tertiary Matriculation Examination (UTME) pre-test and 2013 UTME post-test in Use of English (UOE) subject. A pre-equating model using the 3-parameter (3PL) Item Response Theory (IRT) model was used. IRT software was used for the item calibration. Pre- and post-equating were carried out using 100-items per test form in an UOE test. The results indicate that the raw-score and ability estimates between the pre-equated model and the post-equated model were comparable.

Www.common-metrics.org: a web application to estimate scores from different patient-reported outcome measures on a common scale.

BackgroundRecently, a growing number of Item-Response Theory (IRT) models has been published, which allow estimation of a common latent variable from data derived by different Patient Reported Outcomes (PROs). When using data from different PROs, direct estimation of the latent variable has some advantages over the use of sum score conversion tables. It requires substantial proficiency in the field of psychometrics to fit such models using contemporary IRT software. We developed a web application (http://www.common-metrics.org), which allows estimation of latent variable scores more easily using IRT models calibrating different measures on instrument independent scales.ResultsCurrently, the application allows estimation using six different IRT models for Depression, Anxiety, and Physical Function. Based on published item parameters, users of the application can directly estimate latent trait estimates using expected a posteriori (EAP) for sum scores as well as for specific response patterns, Bayes modal (MAP), Weighted likelihood estimation (WLE) and Maximum likelihood (ML) methods and under three different prior distributions. The obtained estimates can be downloaded and analyzed using standard statistical software.ConclusionsThis application enhances the usability of IRT modeling for researchers by allowing comparison of the latent trait estimates over different PROs, such as the Patient Health Questionnaire Depression (PHQ-9) and Anxiety (GAD-7) scales, the Center of Epidemiologic Studies Depression Scale (CES-D), the Beck Depression Inventory (BDI), PROMIS Anxiety and Depression Short Forms and others. Advantages of this approach include comparability of data derived with different measures and tolerance against missing values. The validity of the underlying models needs to be investigated in the future.

Effects of Initial Values and Convergence Criterion in the Two-Parameter Logistic Model When Estimating the Latent Distribution in BILOG-MG 3.

Parameters of the two-parameter logistic model are generally estimated via the expectation-maximization algorithm, which improves initial values for all parameters iteratively until convergence is reached. Effects of initial values are rarely discussed in item response theory (IRT), but initial values were recently found to affect item parameters when estimating the latent distribution with full non-parametric maximum likelihood. However, this method is rarely used in practice. Hence, the present study investigated effects of initial values on item parameter bias and on recovery of item characteristic curves in BILOG-MG 3, a widely used IRT software package. Results showed notable effects of initial values on item parameters. For tighter convergence criteria, effects of initial values decreased, but item parameter bias increased, and the recovery of the latent distribution worsened. For practical application, it is advised to use the BILOG default convergence criterion with appropriate initial values when estimating the latent distribution from data.

Using rasch measurement to score, evaluate, and improve examinations in an anatomy course

Any examination that involves moderate to high stakes implications for examinees should be psychometrically sound and legally defensible. Currently, there are two broad and competing families of test theories that are used to score examination data. The majority of instructors outside the high-stakes testing arena rely on classical test theory (CTT) methods. However, advances in item response theory software have made the application of these techniques much more accessible to classroom instructors. The purpose of this research is to analyze a common medical school anatomy examination using both the traditional CTT scoring method and a Rasch measurement scoring method to determine which technique provides more robust findings, and which set of psychometric indicators will be more meaningful and useful for anatomists looking to improve the psychometric quality and functioning of their examinations. Results produced by the more robust and meaningful methodology will undergo a rigorous psychometric validation process to evaluate construct validity. Implications of these techniques and additional possibilities for advanced applications are also discussed.

Numerical Differentiation Methods for Computing Error Covariance Matrices in Item Response Theory Modeling

In item response theory (IRT) modeling, the item parameter error covariance matrix plays a critical role define? in statistical inference procedures. When item parameters are estimated using the EM algorithm, the parameter error covariance matrix is not an automatic by-product of item calibration. Cai proposed the use of Supplemented EM algorithm for computing the item parameter error covariance matrix. This method has been subsequently implemented in commercial IRT software programs such as IRTPRO and flexMIRT. Jamshidian and Jennrich noted that Supplemented EM is among a class of methods based on numerically differentiating the EM map, and they proposed noniterative alternatives, such as forward difference and Richardson extrapolation, that are mathematically simpler and may lead to a reduction in computational burden when compared with Supplemented EM. However, the relative merits of the various numerical differentiation methods have not been evaluated in the context of IRT modeling. We perform such an evaluation, using both simulated and empirical data. It is found that the accuracy of the simpler noniterative alternatives is heavily dependent on the choice of the numerical differentiation perturbation constants. On the other hand, Supplemented EM consistently maintains accuracy and does not require the selection of perturbation constants. Furthermore, when implemented with an adaptive iteration scheme, an updated Supplemented EM algorithm can be as computationally efficient as the alternatives. The expected (Fisher) information matrix, while accurate, requires too heavy computation for realistic test lengths. Therefore, we recommend the routine use of the updated Supplemented EM algorithm in IRT applications.

Advancing PROMIS’s methodology: results of the Third Patient-Reported Outcomes Measurement Information System (PROMIS®) Psychometric Summit

In 2002, the NIH launched the ‘Roadmap for Medical Research’. The Patient-Reported Outcomes Measurement Information System (PROMIS®) is one of the Roadmap’s key aspects. To create the next generation of patient-reported outcome measures, PROMIS utilizes item response theory (IRT) and computerized adaptive testing. In 2009, the NIH funded the second wave of PROMIS studies (PROMIS II). PROMIS II studies continue PROMIS’s agenda, but also include new features, including longitudinal analyses and more sociodemographically diverse samples. PROMIS II also includes increased emphasis on pediatric populations and evaluation of PROMIS item banks for clinical research and population science. These aspects bring new psychometric challenges. To address this, investigators associated with PROMIS gathered at the Third Psychometric Summit in September 2010 to identify, describe and discuss pressing psychometric issues and new developments in the field, as well as make analytic recommendations for PROMIS. The summit addressed five general themes: linking, differential item functioning, dimensionality, IRT models for longitudinal applications and new IRT software. In this article, we review the discussions and presentations that occurred at the Third PROMIS Psychometric Summit.

Next steps for use of item response theory in the assessment of health outcomes

We review the NCI/DIA conference, "Improving health outcomes assessment based on modern measurement theory and computerized adaptive testing," and suggest next steps in use of item response theory (IRT) to assess health outcomes. In recent years the level of interest and use of IRT methods has increased dramatically among health outcomes researchers. The NCI/DIA conference on June 24-25, 2004, was one of the first systematic opportunities to examine many challenging issues in applying IRT to the health outcomes field. Based on the conference presentations, we identified five issues important to future applications of IRT to health outcomes. The five key issues are as follows: (1) collaboration between academia, government and industry; (2) common versus unique item banks; (3) educating and establishing standards for use and reporting of IRT; (4) demonstrating the value of IRT; and (5) continuing efforts to improve the user friendliness of IRT software. Moving forward will require a collaborative effort between academia, government agencies, and industry to design and conduct IRT research. A common item bank developed with collaboration from investigators from multiple institutions could be very valuable to the field. The establishment of consensus standards for use and reporting of IRT results would help users and consumers of the methodology. Clear documentation of how IRT can lead to better patient-reported outcome measures and more accurate understanding of substantive issues is essential. Academia, government and industry should continue current work to enhance the user-friendliness of the IRT software.