Logistic Item Response Theory Model Research Articles

Alternative Matching Scores to Control Type I Error of the Mantel–Haenszel Procedure for DIF in Dichotomously Scored Items Conforming to 3PL IRT and Nonparametric 4PBCB Models

When the matching score is either less than perfectly reliable or not a sufficient statistic for determining latent proficiency in data conforming to item response theory (IRT) models, Type I error (TIE) inflation may occur for the Mantel—Haenszel (MH) procedure or any differential item functioning (DIF) procedure that matches on summed-item score, but primarily on short tests. Alternative matching scores were developed based on sufficient statistics, reliability, and explicit corrections for measurement error. Manipulated factors were tests (20, 24, 26 items), reference/focal sample sizes (1,000/1,000, 800/200), proficiency distributions (identical, means differed, variances differed, means and variances differed), and simulation technique (three-parameter logistic IRT model and four-parameter beta compound-binomial model with nonparametric nonmonotonic item-true score step functions). Outcomes were as follows: TIE of MH chi-square test at the .05 nominal level; and the bias, standard error, and root mean square error of the MH delta-DIF statistic under null-DIF conditions. Of eight categorized alternative matching scores, four scores controlled TIE as well as or better than traditional summed-item score in almost all items for all conditions: (a) estimated latent proficiency from a 3PL IRT model, (b) the sum of weighted item scores where the weight was the item— total score biserial correlation coefficient excluding the item from total score, (c) the sum of weighted item scores where the weight was the item loading on the single common factor from factor analysis of tetrachoric correlation coefficients, and (d) Kelley’s linear regressed true score estimate.

Nonlinear sequential designs for logistic item response theory models with applications to computerized adaptive tests

Computerized adaptive testing is becoming increasingly popular due to advancement of modern computer technology. It differs from the conventional standardized testing in that the selection of test items is tailored to individual examinee’s ability level. Arising from this selection strategy is a nonlinear sequential design problem. We study, in this paper, the sequential design problem in the context of the logistic item response theory models. We show that the adaptive design obtained by maximizing the item information leads to a consistent and asymptotically normal ability estimator in the case of the Rasch model. Modifications to the maximum information approach are proposed for the two- and three-parameter logistic models. Similar asymptotic properties are established for the modified designs and the resulting estimator. Examples are also given in the case of the two-parameter logistic model to show that without such modifications, the maximum likelihood estimator of the ability parameter may not be consistent.

When less is more: How fewer diagnostic criteria can indicate greater severity.

For diagnosing many mental disorders, the current Diagnostic and Statistical Manual of Mental Disorders (DSM) system weights each diagnostic criterion equally--each criterion counts the same toward meeting the diagnostic threshold. Research on the diagnostic efficiency of criteria, however, reveals that some diagnostic criteria are more useful than others for identifying their associated mental disorders. That some criteria are more useful than others suggests that the criteria may indicate different levels of severity, but this has yet to be empirically tested. Using data from a large epidemiological study (N = 41,227) and two-parameter logistic item response theory models, the level of latent severity associated with each diagnostic criterion for a particular DSM mental disorder was estimated. Maximum likelihood estimates for all possible response patterns to the criteria were then calculated, and results indicated that items and combinations of items identified varying levels of severity. Furthermore, different response patterns associated with the same raw score identified a range (or band) of latent severity. In many instances, these bands overlapped, revealing that some response patterns with fewer endorsed criteria had higher estimated latent severity than did response patterns with more endorsed criteria. Specifically, many response patterns associated with a raw score of 3 (below threshold for the analyzed disorder) indicated greater latent severity than did response patterns associated with a raw score of 4 (at threshold).

A general diagnostic model applied to language testing data

Probabilistic models with one or more latent variables are designed to report on a corresponding number of skills or cognitive attributes. Multidimensional skill profiles offer additional information beyond what a single test score can provide, if the reported skills can be identified and distinguished reliably. Many recent approaches to skill profile models are limited to dichotomous data and have made use of computationally intensive estimation methods such as Markov chain Monte Carlo, since standard maximum likelihood (ML) estimation techniques were deemed infeasible. This paper presents a general diagnostic model (GDM) that can be estimated with standard ML techniques and applies to polytomous response variables as well as to skills with two or more proficiency levels. The paper uses one member of a larger class of diagnostic models, a compensatory diagnostic model for dichotomous and partial credit data. Many well-known models, such as univariate and multivariate versions of the Rasch model and the two-parameter logistic item response theory model, the generalized partial credit model, as well as a variety of skill profile models, are special cases of this GDM. In addition to an introduction to this model, the paper presents a parameter recovery study using simulated data and an application to real data from the field test for TOEFL Internet-based testing.

Evaluation of diagnostic criteria for night eating syndrome using item response theory analysis

Uniform diagnostic criteria for the night eating syndrome (NES), a disorder characterized by a delay in the circadian pattern of eating, have not been established. Proposed criteria for NES were evaluated using item response theory (IRT) analysis. Six studies yielded 1,481 Night Eating Questionnaires which were coded to reflect the presence/absence of five night eating symptoms. Symptoms were evaluated based on the clinical usefulness of their diagnostic information and on the assumptions of IRT analysis (unidimensionality, monotonicity, local item independence, correct model specification), using a two parameter logistic (2PL) IRT model. Reports of (1) nocturnal eating and/or evening hyperphagia, (2) initial insomnia, and (3) night awakenings showed high precision in discriminating those with night eating problems, while morning anorexia and delayed morning meal provided little additional information. IRT is a useful tool for evaluating the diagnostic criteria of psychiatric disorders and can be used to evaluate potential diagnostic criteria of NES empirically. Behavioral factors were identified as useful discriminators of NES. Future work should also examine psychological factors in conjunction with those identified here.

Investigating Faking Using a Multilevel Logistic Regression Approach to Measuring Person Fit

This article describes how a multilevel logistic regression (MLR) approach to assessing person fit can be used to test hypotheses concerning faking on personality assessments. Item difficulty and person trait estimates obtained from a two-parameter logistic item response theory model are used to predict the probability of endorsing an item in a MLR equation. The regression slope for item difficulty reflects the extent to which the probability of endorsement decreases as item difficulty increases. Less negative slopes may indicate faking, and slope variance may be modeled with person-level variables using MLR. Two examples are presented. Example 1 models faking on a personality assessment with dichotomous items. Example 2 extends the approach to scales using polytomous items.

Aggregated measures of functional disability in a nationally representative sample of disabled people: analysis of dimensionality according to gender and severity of disability

To determine (i) the dimensional invariance of instrumental and basic activities of daily living (IADL/ADL) by gender subgroups, and (ii) the extent to which ADL dimensionality varies with the inclusion or exclusion of nondisabled people. Data were taken from the 1999 Spanish Survey on Disability, Impairment and State of Health. The analysis focussed on 6,522 people aged over 65 years who received help to perform or were unable to perform IADL/ADL items. Unidimensional and multidimensional item response theory (IRT) models were applied to this sample. In the female sample, IADL/ADL items formed a scale with sufficient unidimensionality to fit a two-parameter logistic IRT model. In the male sample, the structure was bidimensional: self-care and mobility, and household activities. When the sample was composed of IADL/ADL disabled people, ADL items formed a unidimensional scale; when it was composed only of ADL disabled people, they formed a bidimensional structure: self-care and mobility. IADL/ADL items can be combined in a single scale to measure severity of functional disability in females, but not in males. Separate aggregated scores must be considered for each subdomain, basic mobility and self-care, in order to measure the severity of ADL disability.

Generating Dichotomous Item Scores with the Four‐Parameter Beta Compound Binomial Model

A Monte Carlo simulation technique for generating dichotomous item scores is presented that implements (a) a psychometric model with different explicit assumptions than traditional parametric item response theory (IRT) models, and (b) item characteristic curves without restrictive assumptions concerning mathematical form. The four‐parameter beta compound‐binomial (4PBCB) strong true score model (with two‐term approximation to the compound binomial) is used to estimate and generate the true score distribution. The nonparametric item‐true score step functions are estimated by classical item difficulties conditional on proportion‐correct total score. The technique performed very well in replicating inter‐item correlations, item statistics (point‐biserial correlation coefficients and item proportion‐correct difficulties), first four moments of total score distribution, and coefficient alpha of three real data sets consisting of educational achievement test scores. The technique replicated real data (including subsamples of differing proficiency) as well as the three‐parameter logistic (3PL) IRT model (and much better than the 1PL model) and is therefore a promising alternative simulation technique. This 4PBCB technique may be particularly useful as a more neutral simulation procedure for comparing methods that use different IRT models.

Assessing the physical violence component of the Revised Conflict Tactics Scales when used in heterosexual couples: an item response theory analysis

Although there are psychometric evaluations of the Revised Conflict Tactics Scales (CTS2) when applied to heterosexual relationships, none has used item response theory (IRT). To address this gap, the present paper assesses the instrument's physical violence subscale. The CTS2 was applied to 764 women who also responded for their partners. Single dimensionality assumption was corroborated. A 2-parameter logistic IRT model was used for estimating location and discriminating power of each item. Differential item functioning and item information pattern along the violence continuum were assessed. Gender differences were detected in 3 out of 12 items. Item coverage of the latent trait spectrum indicated little information at the lower ends, while plenty in the middle and upper ranges. Still, depending on gender, some item overlaps and regions with gaps could be detected. Despite some unresolved problems, the analysis shows that the items form a theoretically coherent information set across the continuum. Provided the user is aware of possible drawbacks, using the physical violence subscale of the CTS2 in heterosexual couples is still a sensible option.

Limited‐information goodness‐of‐fit testing of item response theory models for sparse 2P tables

Bartholomew and Leung proposed a limited-information goodness-of-fit test statistic (Y) for models fitted to sparse 2(P ) contingency tables. The null distribution of Y was approximated using a chi-squared distribution by matching moments. The moments were derived under the assumption that the model parameters were known in advance and it was conjectured that the approximation would also be appropriate when the parameters were to be estimated. Using maximum likelihood estimation of the two-parameter logistic item response theory model, we show that the effect of parameter estimation on the distribution of Y is too large to be ignored. Consequently, we derive the asymptotic moments of Y for maximum likelihood estimation. We show using a simulation study that when the null distribution of Y is approximated using moments that take into account the effect of estimation, Y becomes a very useful statistic to assess the overall goodness of fit of models fitted to sparse 2(P) tables.

Toward an alcohol use disorder continuum using item response theory: results from the National Epidemiologic Survey on Alcohol and Related Conditions

Item response theory (IRT) was used to determine whether the DSM-IV diagnostic criteria for alcohol abuse and dependence are arrayed along a continuum of severity. Data came from a large nationally representative sample of the US population, 18 years and older. A two-parameter logistic IRT model was used to determine the severity and discrimination of each DSM-IV criterion. Differential criterion functioning (DCF) was also assessed across subgroups of the population defined by sex, age and race-ethnicity. All DSM-IV alcohol abuse and dependence criteria, except alcohol-related legal problems, formed a continuum of alcohol use disorder severity. Abuse and dependence criteria did not consistently tap the mildest or more severe end of the continuum respectively, and several criteria were identified as potentially redundant. The drinking in larger amounts or for longer than intended dependence criterion had the greatest discrimination and lowest severity than any other criterion. Although several criteria were found to function differentially between subgroups defined in terms of sex and age, there was evidence that the generalizability and validity of the criterion forming the continuum remained intact at the test score level. DSM-IV diagnostic criteria for alcohol abuse and dependence form a continuum of severity, calling into question the abuse-dependence distinction in the DSM-IV and the interpretation of abuse as a milder disorder than dependence. The criteria tapped the more severe end of the alcohol use disorder continuum, highlighting the need to identify other criteria capturing the mild to intermediate range of the severity. The drinking larger amounts or longer than intended dependence criterion may be a bridging criterion between drinking patterns that incur risk of alcohol use disorder at the milder end of the continuum, with tolerance, withdrawal, impaired control and serious social and occupational dysfunction at the more severe end of the alcohol use disorder continuum. Future IRT and other dimensional analyses hold great promise in informing revisions to categorical classifications and constructing new dimensional classifications of alcohol use disorders based on the DSM and the ICD.

Development of a new critical thinking test using item response theory.

The authors describe the initial development of the Wagner Assessment Test (WAT), an instrument designed to assess critical thinking, using the 5-faceted view popularized by the Watson-Glaser Critical Thinking Appraisal (WGCTA; G. B. Watson & E. M. Glaser, 1980). The WAT was designed to reduce the degree of successful guessing relative to the WGCTA by increasing the number of response alternatives (i.e., 80% of WGCTA items are 2-alternative, multiple-choice), a change that was hypothesized to result in more desirable test information and standard-error functions. Analyses using the 3-parameter logistic item response theory (IRT) model in a sample of undergraduates (N = 407) supported this prediction, even when the WAT item pool was shortened to match the length of the WGCTA. Convergent validity between full-pool IRT score estimates was r = .69. Implications for subsequent research on IRT-based measurement of critical thinking are discussed.

A GENERAL DIAGNOSTIC MODEL APPLIED TO LANGUAGE TESTING DATA

Probabilistic models with more than one latent variable are designed to report profiles of skills or cognitive attributes. Testing programs want to offer additional information beyond what a single test score can provide using these skill profiles. Many recent approaches to skill profile models are limited to dichotomous data and have made use of computationally intensive estimation methods like the Markov chain Monte Carlo (MCMC), since standard maximum likelihood (ML) estimation techniques were deemed infeasible. This paper presents a class of general diagnostic models (GDMs) that can be estimated with customary ML techniques and applies to polytomous response variables as well as to skills with two or more proficiency levels. The model and the algorithm for estimating model parameters handles directly missing responses without the need of collapsing categories or recoding the data. Within the class of GDMs, compensatory as well as noncompensatory models may be specified. This report uses one member of this class of diagnostic models, a compensatory diagnostic model that is parameterized similar to the generalized partial credit model (GPCM). Many well-known models, such as uni- and multivariate versions of the Rasch model and the two parameter logistic item response theory (2PL-IRT) model, the GPCM, and the FACETS model, as well as a variety of skill profile models, are special cases of this member of the class of GDMs. This paper describes an algorithm that capitalizes on using tools from item response theory for scale linking, item fit, and parameter estimation. In addition to an introduction to the class of GDMs and to the partial credit instance of this class for dichotomous and polytomous skill profiles, this paper presents a parameter recovery study using simulated data and an application to real data from the field test for TOEFL® Internet-based testing (iBT).

Effect of Unequal Variances in Proficiency Distributions on Type‐I Error of the Mantel‐Haenszel Chi‐square Test for Differential Item Functioning

Empirical studies demonstrated Type‐I error (TIE) inflation (especially for highly discriminating easy items) of the Mantel‐Haenszel chi‐square test for differential item functioning (DIF), when data conformed to item response theory (IRT) models more complex than Rasch, and when IRT proficiency distributions differed only in means. However, no published study manipulated proficiency variance ratio (VR). Data were generated with the three‐parameter logistic (3PL) IRT model. Proficiency VRs were 1, 2, 3, and 4. The present study suggests inflation may be greater, and may affect all highly discriminating items (low, moderate, and high difficulty), when IRT proficiency distributions of reference and focal groups differ also in variances. Inflation was greatest on the 21‐item test (vs. 41) and 2,000 total sample size (vs. 1,000). Previous studies had not systematically examined sample size ratio. Sample size ratio of 1:1 produced greater TIE inflation than 3:1, but primarily for total sample size of 2,000.

Moderated Multiple Regression, Spurious Interaction Effects, and IRT

Two Monte Carlo studies were conducted to explore the Type I error rates in moderated multiple regression (MMR) of observed scores and estimated latent trait scores from a two-parameter logistic item response theory (IRT) model. The results of both studies showed that MMR Type I error rates were substantially higher than the nominal alpha levels when scale scores were composed of summed binary item responses (e.g., true/false, yes/no, disagree/agree items). Performing the regression analyses on estimated trait scores (Θ̂) from a two-parameter logistic model improved the error detection rates considerably. That is, the Type I error rates for spurious interaction effects were similar to the nominal alpha levels under most conditions. These findings suggest that IRT provides a viable means of controlling an important source of spurious interactions in data sets that are well characterized by IRT models.

DIF for examinees with different academic backgrounds

The purpose of the present study was to investigate differential item function (DIF) on the English subtest of the 1998 Korean National Entrance Exam for Colleges and Universities for examinees with different academic backgrounds (i.e., Humanities vs. Sciences) using the Item Response Theory (IRT) Likelihood Ratio (LR) approach. The English subtest consisted of a total of 55 items, which comprised the subscales of Listening Comprehension and Reading Comprehension. Data were fitted with a modified three parameter logistic IRT model, and DIF was detected using both the Mantel-Haenszel procedure and the IRT likelihood ratio approach. To evaluate a baseline estimate of DIF due to chance, the entire DIF detection procedure was repeated, with the sample randomly divided in half. The results of the study identified 18 DIF items with 28 DIF parameters. DIF magnitude for each studied item is also provided. Results of a preliminary content analysis are discussed in terms of the relationships between DIF directions and item content. Implications and recommendations for future research are also addressed.

Power analysis in randomized clinical trials based on item response theory

Patient relevant outcomes, measured using questionnaires, are becoming increasingly popular endpoints in randomized clinical trials (RCTs). Recently, interest in the use of item response theory (IRT) to analyze the responses to such questionnaires has increased. In this paper, we used a simulation study to examine the small sample behavior of a test statistic designed to examine the difference in average latent trait level between two groups when the two-parameter logistic IRT model for binary data is used. The simulation study was extended to examine the relationship between the number of patients required in each arm of an RCT, the number of items used to assess them, and the power to detect minimal, moderate, and substantial treatment effects. The results show that the number of patients required in each arm of an RCT varies with the number of items used to assess the patients. However, as long as at least 20 items are used, the number of items barely affects the number of patients required in each arm of an RCT to detect effect sizes of 0.5 and 0.8 with a power of 80%. In addition, the number of items used has more effect on the number of patients required to detect an effect size of 0.2 with a power of 80%. For instance, if only five randomly selected items are used, it is necessary to include 950 patients in each arm, but if 50 items are used, only 450 are required in each arm. These results indicate that if an RCT is to be designed to detect small effects, it is inadvisable to use very short instruments analyzed using IRT. Finally, the SF-36, SF-12, and SF-8 instruments were considered in the same framework. Since these instruments consist of items scored in more than two categories, slightly different results were obtained.

Sequential Computerized Mastery Tests—Three Simulation Studies

A simulation study of a sequential computerized mastery test is carried out with items modeled with the 3 parameter logistic item response theory model. The examinees' responses are either identically distributed, not identically distributed, or not identically distributed together with estimation errors in the item characteristics. The simulations indicated that the observed results from the operating characteristic function differ significantly from the theoretical results, which is probably due to the use of an approximation formula. The mean number of items in a test, the distribution of test length, and the variance depend highly on how well we know the true values of the item characteristics and whether they are identically distributed or not.

Estimating Consistency and Accuracy Indices for Multiple Classifications

This article describes procedures for estimating various indices of classification consistency and accuracy for multiple category classifications using data from a single test administration. The estimates of the classification consistency and accuracy indices are compared under three different psychometric models: the two-parameter beta binomial, four-parameter beta binomial, and three-parameter logistic IRT (item response theory) models. Using real data sets, the estimation procedures are illustrated, and the characteristics of the estimated classification indices are examined. This article also examines the behavior of the estimated classification indices as a function of the latent variable. All three components of the models (i.e., the estimated true score distributions, fitted observed score distributions, and estimated conditional error variances) appear to have considerable influence on the magnitudes of the estimated classification indices. Choosing a model in practice should be based on various considerations including the degree of model fit to the data, suitability of the model assumptions, and the computational feasibility.

Minimizing the Influence of Item Parameter Estimation Errors in Test Development: A Comparison of Three Selection Procedures

In test development, item response theory (IRT) is a method to determine the amount of information that each item (i.e., item information function) and combination of items (i.e., test information function) provide in the estimation of an examinee's ability. Studies investigating the effects of item parameter estimation errors over a range of ability have demonstrated an overestimation of information when the most discriminating items are selected (i.e., item selection based on maximum information). In the present study, the authors examined the influence of item parameter estimation errors across 3 item selection methods—maximum no target, maximum target, and theta maximum—using the 2- and 3-parameter logistic IRT models. Tests created with the maximum no target and maximum target item selection procedures consistently overestimated the test information function. Conversely, tests created using the theta maximum item selection procedure yielded more consistent estimates of the test information function and, at times, underestimated the test information function. Implications for test development are discussed.