Item Characteristic Curves Research Articles

An Item-Response Theory Analysis of DSM-IV Alcohol-Use Disorder Criteria and “Binge” Drinking in Undergraduates

This is the first study to examine the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV), criteria for alcohol-use disorders and heavy episodic (or "binge") drinking in a college sample using item-response theory (IRT) analysis. IRT facilitates assessment of the severity of the criteria, their ability to distinguish between those at greatest and lowest risk, and the value of adding a "binge" drinking criterion. In a sample of undergraduate drinkers (n = 353), we conducted factor analyses to determine whether the criteria best fit a one- or two-factor structure. We then conducted IRT analyses to obtain item-characteristic curves indicating the probability of endorsing a criterion at increasing levels of alcohol-use-disorder risk. These analyses were first conducted including current (i.e., past-year) DSM-IV alcohol-use-disorder criteria only and then rerun adding weekly "binge" drinking. A single-factor model of the DSM-IV criteria did not differ significantly from a two-factor model. After including "binge" drinking, two factors fit the data slightly better than one factor but with a dominant single factor. Withdrawal was the most severe criterion, whereas tolerance and "larger/longer" were the least severe. Time spent drinking and a combined social/legal difficulties criterion had the best ability to discriminate those at greatest and lowest risk for an alcohol-use disorder. "Binge" drinking showed both low discrimination and low severity. To our knowledge, this is the first study to examine DSM-IV criteria in an undergraduate sample using IRT. In this sample, abuse and dependence were intermixed on a continuum of severity, and "binge" drinking was in the least severe region.

Asymmetric item characteristic curve in item response theory: model construction and application

Most of the models used in the Item Response Theory assume point-symmetry of the Item Characteristic Curve. Although the convenience of such models is obvious a priori assumption of symmetry does not always correspond to an experimental data. In many cases we deal with obviously skewed distribution and the usage of symmetric models involves inevitable errors both in experimental studies and in test constructing. The applicability of the asymmetric model for the analysis of test items is discussed in the present study. Data obtained from Raven's Advanced Progressive Matrices (N=365) are used to illustrate the analysis.

Questioning the measurement precision of psychotherapy research

Recent developments in psychometric theory have important implications for the development and validation of the instruments used in psychotherapy research. This article illustrates how the advances in latent trait and item response theory (IRT) have distinct advantages over classical test theory. The basic assumptions of IRT (unidimensionality, local independence, and monotonicity) and the concepts common to all four of the basic IRT models (item characteristic curve, item information function, and test information function) are reviewed. IRT methods are illustrated using responses to a psychotherapy outcome instrument from a large data set of a commercial health plan.

Using the Rasch Model to Validate and Enhance the Interpretation of the Functional Assessment of Cancer Therapy–Kidney Symptom Index—Disease-Related Symptoms Scale

ObjectivesThe Functional Assessment of Cancer Therapy–Kidney Symptom Index—Disease-Related Symptoms (FKSI-DRS) was developed to assess patients' kidney-cancer-related symptoms. The Rasch rating scale, a one-parameter logistic item response model, may enhance FKSI-DRS interpretation and validate its measurement properties. MethodsWe applied the Rasch model to FKSI-DRS data from a randomized phase 3 trial in which first-line sunitinib therapy showed superiority to interferon-alfa in patients with metastatic renal cell carcinoma. Of 750 enrolled patients, 668 patients completed the questionnaire on cycle 1, day 28 and were evaluated in the current study. The nine FKSI-DRS items were analyzed to enhance interpretation of the summary score by using an item characteristic curve that related score to probability of reporting specific symptoms. ResultsThe Rasch model fitted the FKSI-DRS well: 8 of 9 items had acceptable infit and outfit statistics (<1.5, >0.5); item difficulty spanned a wide range (−3.23 to 1.64 logits); and the five response categories performed adequately. The item characteristic curve offered enhanced interpretation of FKSI-DRS: For example, an FKSI-DRS score of 27 (mean baseline score for total sample) indicated a 47% chance of reporting “no” to “lack of energy,” although a two-point difference between sunitinib and interferon-alfa, averaged across all assessments (29 vs. 27), corresponded to sunitinib achieving a 28% increase (13% absolute difference) in the probability of reporting “no” to “lack of energy” (60% vs. 47%). ConclusionsData suggest that the FKSI-DRS is an adequate measure of symptom status in patients with metastatic renal cell carcinoma. The Rasch model supports its validation and enhances its interpretation.

A Monte Carlo Study of the Effect of Item Characteristic Curve Estimation on the Accuracy of Three Person-Fit Statistics

To date, there have been no studies comparing parametric and nonparametric Item Characteristic Curve (ICC) estimation methods on the effectiveness of Person-Fit Statistics (PFS). The primary aim of this study was to determine if the use of ICCs estimated by nonparametric methods would increase the accuracy of item response theory—based PFS for small sample sizes. Using three recognized PFS (lz, ECI2 z, and ECI4z), four estimation methods were compared: two parametric methods (the two-parameter logistic model and three-parameter logistic model) and two nonparametric methods (Nadaraya-Watson's regression and the local logistic regression). Finally, matrices of 100- and 1,000-answer vectors were generated for this Monte Carlo study. For both large and small sample sizes, the accuracy of the PFS was greater when used with the parametric models.

Innovations in Measuring Rater Accuracy in Standard Setting: Assessing “Fit” to Item Characteristic Curves

Standard setting methods such as the Angoff method rely on judgments of item characteristics; item response theory empirically estimates item characteristics and displays them in item characteristic curves (ICCs). This study evaluated several indexes of rater fit to ICCs as a method for judging rater accuracy in their estimates of expected item performance for target groups of test-takers. Simulated data were used to compare adequately fitting ratings to poorly fitting ratings at various target competence levels in a simulated two stage standard setting study. The indexes were then applied to a set of real ratings on 66 items evaluated at 4 competence thresholds to demonstrate their relative usefulness for gaining insight into rater “fit.” Based on analysis of both the simulated and real data, it is recommended that fit indexes based on the absolute deviations of ratings from the ICCs be used, and those based on the standard errors of ratings should be avoided. Suggestions are provided for using these indexes in future research and practice.

A Bayesian Semiparametric Item Response Model with Dirichlet Process Priors

In Item Response Theory (IRT), item characteristic curves (ICCs) are illustrated through logistic models or normal ogive models, and the probability that examinees give the correct answer is usually a monotonically increasing function of their ability parameters. However, since only limited patterns of shapes can be obtained from logistic models or normal ogive models, there is a possibility that the model applied does not fit the data. As a result, the existing method can be rejected because it cannot deal with various item response patterns. To overcome these problems, we propose a new semiparametric IRT model using a Dirichlet process mixture logistic distribution. Our method does not rely on assumptions but only requires that the ICCs be a monotonically nondecreasing function; that is, our method can deal with more types of item response patterns than the existing methods, such as the one-parameter normal ogive models or the two- or three-parameter logistic models. We conducted two simulation studies whose results indicate that the proposed method can express more patterns of shapes for ICCs and can estimate the ability parameters more accurately than the existing parametric and nonparametric methods. The proposed method has also been applied to Facial Expression Recognition data with noteworthy results.

Cross-cultural comparisons of the Mini-mental State Examination between Japanese and U.S. cohorts

The Mini-mental State Examination (MMSE) is widely used in Japan and the U.S.A. for cognitive screening in the clinical setting and in epidemiological studies. A previous Japanese community study reported distributions of the MMSE total score very similar to that of the U.S.A. Data were obtained from the Monongahela Valley Independent Elder's Study (MoVIES), a representative sample of community-dwelling elderly people aged 65 and older living near Pittsburgh, U.S.A., and from the Tajiri Project, with similar aims in Tajiri, Japan. We examined item-by-item distributions of the MMSE between two cohorts, comparing (1) percentage of correct answers for each item within each cohort, and (2) relative difficulty of each item measured by Item Characteristic Curve analysis (ICC), which estimates log odds of obtaining a correct answer adjusted for the remaining MMSE items, demographic variables (age, gender, education) and interactions of demographic variables and cohort. Median MMSE scores were very similar between the two samples within the same education groups. However, the relative difficulty of each item differed substantially between the two cohorts. Specifically, recall and auditory comprehension were easier for the Tajiri group, but reading comprehension and sentence construction were easier for the MoVIES group. Our results reaffirm the importance of validation and examination of thresholds in each cohort to be studied when a common instrument is used as a dementia screening tool or for defining cognitive impairment.

Item Analysis of C, D and E Series from Raven’s Standard Progressive Matrices with Item Response Theory Two-Parameter Logistic Model

The present report is focused on the Item Response Theory research methodology and descriptive potential. Its purpose is to present the item analysis of C, D and E Series from Raven’s Standard Progressive Matrices which were given to 506 Bulgarian high school students. The basic concepts and underlying assumptions of IRT are briefly reviewed. The latent variable is defined after a short check of some intelligence theories and a detailed examination of the used items. After the verification of IRT assumptions, the Two-parameter logistic model is selected for the analysis. The estimated item parameters are interpreted in accordance to suggested guidelines. Item Characteristic Curves and Item Information Functions are plotted and their features are also discussed.

On the Use of Nonparametric Item Characteristic Curve Estimation Techniques for Checking Parametric Model Fit

The purpose of this study was to assess the model fit of a 2PL through comparison with the nonparametric item characteristic curve (ICC) estimation procedures. Results indicate that three nonparametric procedures implemented produced ICCs that are similar to that of the 2PL for items simulated to fit the 2PL. However for misfitting items, especially nonmonotone items, the greatest difference is between the 2PL and kernel smoothing procedures. In general, the differences between ICCs from the nonparametric procedures and the 2PL are reduced as both sample size and test length increase. The false positive rate of the test for model fit is promising for nonparametric ICC estimation methods. Power to detect misfitting items simulated with 4PL is low. Power to detect nonmonotone items is generally much higher. Power is best for kernel smoothing but also good for isotonic regression in the medium to large sample sizes and longer test length conditions. Power for the smoothed isotonic regression is uniformly low.

Modified Likelihood-Based Item Fit Statistics for the Generalized Graded Unfolding Model

Orlando and Thissen (2000) developed an item fit statistic for binary item response theory (IRT) models known as S - X2. This article generalizes their statistic to polytomous unfolding models. Four alternative formulations of S - X2 are developed for the generalized graded unfolding model (GGUM). The GGUM is a unidimensional IRT model for unfolding polytomous responses. It yields single-peaked, nonmonotonic item characteristic curves that predict a higher item score to the extent that an individual is located close to an item on the underlying latent continuum. A simulation was performed to examine the characteristics of these new item fit indices under the GGUM, as well as a traditional likelihood ratio χ2 test ( G2). All variants of S - X2 exhibited reasonable Type I error rates, but that for G2 was more erratic. The new indices exhibited variable power to detect misfit. Two new item fit tests are recommended for use based on simulation results.

Conversion of Proportion-Correct Standard-Setting Judgments to Cutoff Scores on the Item Response Theory θ Scale

This study compares the efficacy of different strategies for translating item-level, proportion-correct standard-setting judgments into a y-metric test cutoff score for use with item response theory (IRT) scoring, using Monte Carlo methods. Simulated Angoff-type ratings, consisting of 1,000 independent 75 Item × 13 Rater matrices, were generated at five points along the y continuum, at three levels of rater fit to the item characteristics curves, yielding 14,625,000 ratings as the basis of the analyses. These simulated proportion-correct ratings were converted to the IRT y scale using test-level and item-level methods explicated by Kane (1987). Kane's optimally weighted, item-level conversion method initially produced anomalous results; however, it was discovered that imposing a restriction on the weights avoided these anomalies and rendered the optimally weighted method the most statistically efficient. Six areas for future research are outlined for advancing the integration of these classical standard-setting ratings into IRT methodology.

Nonparametric Bayesian modelling for item response

Item response theory is widely used in standardized testing to model the relationship between test takers’ unobserved ability levels and their responses to items on the test. Item characteristic curves give the probability of a correct response to an item as a function of ability and are most often modelled with logistic curves. In this paper we demonstrate how to model the item characteristic curve with nonparametric Bayesian methods through the use of Dirichlet process priors and present a complementary model in which the ability distribution is modelled nonparametrically while the item characteristic curves are logistic. We compare the nonparametric models with the two-parameter logistic Bayesian model on data from an exam in an introductory statistics course. We find that the nonparametric curve model produces significantly different item characteristic curves for a few of the items and that the corresponding ability estimates also change substantially for some individuals.

Linking Parameters Estimated With the Generalized Graded Unfolding Model: A Comparison of the Accuracy of Characteristic Curve Methods

Methods for linking item response theory (IRT) parameters are developed for attitude questionnaire responses calibrated with the generalized graded unfolding model (GGUM). One class of IRT linking methods derives the linking coefficients by comparing characteristic curves, and three of these methods—test characteristic curve (TCC), item characteristic curve (ICC), and operating characteristic curve (OCC)—are extended for use with the GGUM. Alternate versions of these methods are developed to produce both symmetric and asymmetric solutions. The adequacy of these methods for use with the GGUM is evaluated in a simulation that varied the distributions of θ in the groups to be linked and the number of individuals and anchor items used for linking. All methods performed well under the conditions studied. Statistically significant effects were found for sample size and number of anchor items. It is recommended that choice of linking method should consider the context and intent of the linking.

Profiling User Behavior for Intrusion Detection Using Item Response Modeling

AbstractItem response theory (IRT) is a modern test measurement theory that has been widely used in many research areas over the last decade. This paper presents an IRT modeling approach that fits network traffic to a “test” (normal or abnormal) model and estimates an expected test score of being anomaly-free to profile user behavior. With four anomaly-free associated variables identified from previous studies, the findings demonstrate that there is a remarkable difference in item characteristic curves between the user behavior patterns with anomalies and those that are anomaly-free, and such a difference can be quantitatively measured with the expected test score ranging from 0 to 100 where a high score is more likely to be associate with an anomaly-free pattern. More specifically, there are approximately 25 (SD = 4.0) points’ differences between a pattern with anomalies and one without. Our study demonstrates the potential feasibility and achievability of applying IRT for modern network security.

PlotIRT: A Collection of R Functions to Plot Curves Associated With Item Response Theory

The PlotIRT set of R functions facilitates plotting a variety of curves associated with item response theory (IRT). These functions were developed so that researchers using IRT could have a simple, yet flexible method for creating graphics associated with item parameters. These graphics can be customized in numerous ways to produce attractive figures that can be used in manuscripts or presentations. The R software environment was chosen so that users could produce IRT graphics via free software. The plots that are available in PlotIRT are item and test characteristic curves, item and test information curves, expected item score functions, and posterior density distributions. The R functions are applicable to the two-parameter logistic, three-parameter logistic, graded, nominal, and multiple-choice models. The functions work with both individual items or with groups of items, which do not need to be based on the same IRT model. If the user chooses, the graphics can be customized with a number of options. These options include varying the color of the lines by item, response category, or test; making the lines a solid, dashed, or other pattern; labeling the response category curves; specifying the number of items per plot and the number of plots per window; labeling the plot and axes; and adjusting the font and its size. The posterior distribution plot requires the user to specify a response pattern for the items, and it may be produced along with the item characteristic curves and the underlying y distribution. Basic R functions can be used to read in an assortment of item parameter files, or item parameters can be typed directly into R arrays. As the function is submitted, plots appear in windows within the R interface. An option is included in the function to save the plots as any file type supported by R, including postscript and portable document format.

Generating Dichotomous Item Scores with the Four‐Parameter Beta Compound Binomial Model

A Monte Carlo simulation technique for generating dichotomous item scores is presented that implements (a) a psychometric model with different explicit assumptions than traditional parametric item response theory (IRT) models, and (b) item characteristic curves without restrictive assumptions concerning mathematical form. The four‐parameter beta compound‐binomial (4PBCB) strong true score model (with two‐term approximation to the compound binomial) is used to estimate and generate the true score distribution. The nonparametric item‐true score step functions are estimated by classical item difficulties conditional on proportion‐correct total score. The technique performed very well in replicating inter‐item correlations, item statistics (point‐biserial correlation coefficients and item proportion‐correct difficulties), first four moments of total score distribution, and coefficient alpha of three real data sets consisting of educational achievement test scores. The technique replicated real data (including subsamples of differing proficiency) as well as the three‐parameter logistic (3PL) IRT model (and much better than the 1PL model) and is therefore a promising alternative simulation technique. This 4PBCB technique may be particularly useful as a more neutral simulation procedure for comparing methods that use different IRT models.

Latent Trait Modeling with the Common beliefs Survey III: Using Item Response Theory to Evaluate an Irrational Beliefs Inventory

Assessment of irrational beliefs by such measures as the Common Beliefs Survey III (CBS) has traditionally relied upon classical test theory assumptions, in which the properties of specific test items are less important than the total test score as the aggregate of all item responses. An alternative approach using item response theory (IRT) methodology allows one to specify the parameters of difficulty and discrimination for each test item. Difficulty levels of CBS items range along a continuum of irrationality, the implied latent trait measured by responses to the questionnaire as a whole. We evaluated the CBS responses of 605 individuals from clinical and college settings, drawing from current and archival data. The original Likert scale ratings were recoded into dichotomous scores. Fourteen of the 54 items were highly or very highly discriminating in distinguishing respondents with high and low irrationality levels. However, discriminating items exhibited a very narrow range of difficulty; most functioned at a point a little above the halfway mark on the continuum of irrationality. Item characteristic curves and test information curves were very similar for female (n = 424) and male (n = 179) respondents. We derived a 4-item screening test for irrationality from our IRT analyses of the 54 CBS items. Further test development, focused on the selection and scaling of items with a much broader range of difficulty, would facilitate evaluation of the hierarchical structure of irrational beliefs.

A Comparison of Methods for Nonparametric Estimation of Item Characteristic Curves for Binary Items

This study compares the performance of three nonparametric item characteristic curve (ICC) estimation procedures: isotonic regression, smoothed isotonic regression, and kernel smoothing. Smoothed isotonic regression, employed along with an appropriate kernel function, provides better estimates and also satisfies the assumption of strict monotonicity. As the number of items and the sample size increase, the kernel smoothing and smoothed isotonic regression ICC estimation procedures yield similar results across all conditions.

A Pearson-Type-VII item response model for assessing person fluctuation

Using Lumsden’s Thurstonian fluctuation model as a starting point, this paper attempts to develop a unidimensional item response theory model intended for binary personality items. Under some additional assumptions, a new model is obtained in which the item characteristic curves are defined by a cumulative Pearson-Type-VII distribution, and the person response curves are two-parameter normal ogives. Procedures for fitting the new model are proposed. Furthermore, the relations between individual fluctuation and scalability are discussed, and a scalability index based on the new model is proposed. All the developments in this paper are illustrated using two empirical examples.