Logistic Item Response Theory Model Research Articles

Item response theory-based continuous test norming.

In norm-referenced psychological testing, an individual's performance is expressed in relation to a reference population using a standardized score, like an intelligence quotient score. The reference population can depend on a continuous variable, like age. Current continuous norming methods transform the raw score into an age-dependent standardized score. Such methods have the shortcoming to solely rely on the raw test scores, ignoring valuable information from individual item responses. Instead of modeling the raw test scores, we propose modeling the item scores with a Bayesian two-parameter logistic (2PL) item response theory model with age-dependent mean and variance of the latent trait distribution, 2PL-norm for short. Norms are then derived using the estimated latent trait score and the age-dependent distribution parameters. Simulations show that 2PL-norms are overall more accurate than those from the most popular raw score-based norming methods cNORM and generalized additive models for location, scale, and shape (GAMLSS). Furthermore, the credible intervals of 2PL-norm exhibit clearly superior coverage over the confidence intervals of the raw score-based methods. The only issue of 2PL-norm is its slightly lower performance at the tails of the norms. Among the raw score-based norming methods, GAMLSS outperforms cNORM. For empirical practice this suggests the use of 2PL-norm, if the model assumptions hold. If not, or the interest is solely in the point estimates of the extreme trait positions, GAMLSS-based norming is a better alternative. The use of the 2PL-norm is illustrated and compared with GAMLSS and cNORM using empirical data, and code is provided, so that users can readily apply 2PL-norm to their normative data. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Bias and Linking Error in Fixed Item Parameter Calibration

The two-parameter logistic (2PL) item response theory (IRT) model is frequently applied to analyze group differences for multivariate binary random variables. The item parameters in the 2PL model are frequently fixed when estimating the mean and the standard deviation for a group of interest. This method is also called fixed item parameter calibration (FIPC). In this article, the bias and the linking error of the FIPC approach are analytically derived in the presence of random uniform differential item functioning (DIF). The adequacy of the analytical findings was validated in a simulation study. It turned out that the extent of the bias and the variance in distribution parameters increases with increasing variance of random DIF effects.

Development and validation of the MultiScent-20 digital odour identification test using item response theory

Although validated and reliable psychophysical tests of olfactory function are available, an easy-to-use and feasible test has yet to be developed. This study aimed to design a digital odour identification test, evaluate its validity, assess its reliability, establish a normative curve, and explore the impact of demographic factors. The odour identification test was presented with the Multiscent-20, a hand-held, tablet-like digital scent device that features an integrated odour digital delivery system. The identification performance on the 20 odours was assessed using item response theory (IRT). The normative curve was established by administering the test to a large sample of participants (n = 1299). The mean identification score was 17.5 (SD = 2.1). The two-parameter logistic IRT model provided the best fit, revealing variation in item discrimination and difficulty parameters. Educational attainment influenced performance, with primary education associated with lower scores. Additionally, sex was not found to be associated with performance. This study provides initial evidence supporting the validity and reliability of use of the Multiscent-20 as a digital odour identification test. The test’s automation and portability enable the standardized delivery of olfactory stimuli and efficient automatic recording and scoring of responses.

Estimation of Standard Error, Linking Error, and Total Error for Robust and Nonrobust Linking Methods in the Two-Parameter Logistic Model

The two-parameter logistic (2PL) item response theory model is a statistical model for analyzing multivariate binary data. In this article, two groups are brought onto a common metric using the 2PL model using linking methods. The linking methods of mean–mean linking, mean–geometric–mean linking, and Haebara linking are investigated in nonrobust and robust specifications in the presence of differential item functioning (DIF). M-estimation theory is applied to derive linking errors for the studied linking methods. However, estimated linking errors are prone to sampling error in estimated item parameters, thus resulting in artificially increased the linking error estimates in finite samples. For this reason, a bias-corrected linking error estimate is proposed. The usefulness of the modified linking error estimate is demonstrated in a simulation study. It is shown that a simultaneous assessment of the standard error and linking error in a total error must be conducted to obtain valid statistical inference. In the computation of the total error, using the bias-corrected linking error estimate instead of the usually employed linking error provides more accurate coverage rates.

The Rank-2PL IRT Models for Forced-Choice Questionnaires: Maximum Marginal Likelihood Estimation with an EM Algorithm

The rank two-parameter logistic (Rank-2PL) item response theory models refer to a set of models applying the 2PL model in a sequential ranking process that occurs in forced-choice questionnaires. The multi-unidimensional pairwise preference with 2PL model (MUPP-2PL) is a Rank-2PL model for items with two statements. Focusing on items with three statements, we develop a maximum marginal likelihood estimation with an expectation-maximization algorithm to estimate item parameters and their standard errors. A simulation study is conducted to check parameter recovery, and then the model is applied to a real dataset. Finally, the findings are summarized and discussed, and future research is suggested.

A One‐Parameter Diagnostic Classification Model with Familiar Measurement Properties

AbstractDiagnostic classification models (DCMs) are psychometric models designed to classify examinees according to their proficiency or nonproficiency of specified latent characteristics. These models are well suited for providing diagnostic and actionable feedback to support intermediate and formative assessment efforts. Several DCMs have been developed and applied in different settings. This study examines a DCM with functional form similar to the 1‐parameter logistic item response theory model. Using data from a large‐scale mathematics education research study, we demonstrate and prove that the proposed DCM has measurement properties akin to the Rasch and one‐parameter logistic item response theory models, including sum score sufficiency, item‐free and person‐free measurement, and invariant item and person ordering. We introduce some potential applications for this model, and discuss the implications and limitations of these developments, as well as directions for future research.

Smooth Information Criterion for Regularized Estimation of Item Response Models

Item response theory (IRT) models are frequently used to analyze multivariate categorical data from questionnaires or cognitive test data. In order to reduce the model complexity in item response models, regularized estimation is now widely applied, adding a nondifferentiable penalty function like the LASSO or the SCAD penalty to the log-likelihood function in the optimization function. In most applications, regularized estimation repeatedly estimates the IRT model on a grid of regularization parameters λ. The final model is selected for the parameter that minimizes the Akaike or Bayesian information criterion (AIC or BIC). In recent work, it has been proposed to directly minimize a smooth approximation of the AIC or the BIC for regularized estimation. This approach circumvents the repeated estimation of the IRT model. To this end, the computation time is substantially reduced. The adequacy of the new approach is demonstrated by three simulation studies focusing on regularized estimation for IRT models with differential item functioning, multidimensional IRT models with cross-loadings, and the mixed Rasch/two-parameter logistic IRT model. It was found from the simulation studies that the computationally less demanding direct optimization based on the smooth variants of AIC and BIC had comparable or improved performance compared to the ordinarily employed repeated regularized estimation based on AIC or BIC.

Identifying Individuals with Highest Social Risk in Adults with Type 2 Diabetes Using Item Response Theory.

The aim of this analysis was to create a parsimonious tool to screen for high social risk using item response theory to discriminate across social risk factors in adults with type 2 diabetes. Cross-sectional data of 615 adults with diabetes recruited from two primary care clinics were used. Participants completed assessments including validated scales on economic instability (financial hardship), neighborhood and built environment (crime, violence, neighborhood rating), education (highest education, health literacy), food environment (food insecurity), social and community context (social isolation), and psychological risk factors (perceived stress, depression, serious psychological distress, diabetes distress). Item response theory (IRT) models were used to understand the association between a participant's underlying level of a particular social risk factor and the probability of that response. A two-parameter logistic IRT model was used with each of the 12 social determinant factors being added as a separate parameter in the model. Higher values in item discrimination indicate better ability of a specific social risk factor in differentiating participants from each other. Rate of crime reported in a neighborhood (discrimination 3.13, SE 0.50; item difficulty - 0.68, SE 0.07) and neighborhood rating (discrimination 4.02, SE 0.87; item difficulty - 1.04, SE 0.08) had the highest discrimination. Based on these findings, crime and neighborhood rating discriminate best between individuals with type 2 diabetes who have high social risk and those with low social risk. These two questions can be used as a parsimonious social risk screening tool to identify high social risk.

Validating a performance assessment of computational thinking for early childhood using item response theory

Despite growing interest in early childhood computational thinking (CT), there is a lack of validated assessments for children who are emerging readers. This paper presents validity and reliability evidence of a performance-based assessment of CT using item response theory (IRT) from 272 children aged 4–8. Using a two-parameter logistic model IRT model (2PL IRT), we confirmed that model- and item-level fits are acceptable. Item analyses revealed a high discriminability (M = 2.26, SD = 1.12) and a moderate item difficulty (M = −0.21; SD = 0.86), on average, across 19 items. Reliability analysis demonstrated that the assessment was substantially reliable (marginal reliability: rxx = 0.87). Differential item functioning (DIF) analyses indicated that the assessment estimated children's item parameters fairly, regardless of their gender and age. However, we confirmed gaps in latent ability (θ) of CT by gender and age: boys showed higher latent ability of CT than girls, and old children (above 72 months) showed higher latent ability than young children (below 72 months). Findings suggest the assessment is a fair measure that can serve as a reliable and valid tool to assess CT for children who are emerging readers.

Proof of Reliability Convergence to 1 at Rate of Spearman–Brown Formula for Random Test Forms and Irrespective of Item Pool Dimensionality

It is shown that the psychometric test reliability, based on any true-score model with randomly sampled items and uncorrelated errors, converges to 1 as the test length goes to infinity, with probability 1, assuming some general regularity conditions. The asymptotic rate of convergence is given by the Spearman–Brown formula, and for this it is not needed that the items are parallel, or latent unidimensional, or even finite dimensional. Simulations with the 2-parameter logistic item response theory model reveal that the reliability of short multidimensional tests can be positively biased, meaning that applying the Spearman–Brown formula in these cases would lead to overprediction of the reliability that results from lengthening a test. However, test constructors of short tests generally aim for short tests that measure just one attribute, so that the bias problem may have little practical relevance. For short unidimensional tests under the 2-parameter logistic model reliability is almost unbiased, meaning that application of the Spearman–Brown formula in these cases of greater practical utility leads to predictions that are approximately unbiased.

Comparison of statistical models for individual’s ability index and ranking

ABSTRACT Economic efficiency demands accurate assessment of individual ability for selection purposes. This study investigates Classical Test Theory (CTT) and Item Response Theory (IRT) for estimating true ability and ranking individuals. Two Monte Carlo simulations and real data analyses were conducted. Results suggest a slight advantage for IRT, but ability estimates from both methods were highly correlated (r=0.95), indicating similar outcomes. The Logistic two-parameter IRT model emerged as the most reliable and rigorous approach.

How well do DSM-5 criteria measure alcohol use disorder in the general population of older Swedish adolescents? An item response theory analysis

BackgroundThis study assesses the psychometric properties of DSM-5 criteria of AUD in older Swedish adolescents using item response theory models, focusing specifically on the precision of the scale at the cut-offs for mild, moderate, and severe AUD. MethodsData from the second wave of Futura01 was used. Futura01 is a nationally representative cohort study of Swedish people born 2001 and data for the second wave was collected when participants were 17/18 years old. This study included only participants who had consumed alcohol during the past 12 months (n = 2648). AUD was measured with 11 binary items. A 2-parameter logistic item response theory model (2PL) estimated the items’ difficulty and discrimination parameters. Results31.8% of the participants met criteria for AUD. Among these, 75.6% had mild AUD, 18.3% had moderate, and 6.1% had severe AUD. A unidimensional AUD model had a good fit and 2PL models showed that the scale measured AUD over all three cut-offs for AUD severity. Although discrimination parameters ranged from moderate (1.24) to very high (2.38), the more commonly endorsed items discriminated less well than the more difficult items, as also reflected in less precision of the estimates at lower levels of AUD severity. The diagnostic uncertainty was pronounced at the cut-off for mild AUD. ConclusionDSM-5 criteria measure AUD with better precision at higher levels of AUD severity than at lower levels. As most older adolescents who fulfil an AUD diagnosis are in the mild category, notable uncertainties are involved when an AUD diagnosis is set in this group.

Benefits of the Curious Behavior of Bayesian Hierarchical Item Response Theory Models-An in-Depth Investigation and Bias Correction.

When using Bayesian hierarchical modeling, a popular approach for Item Response Theory (IRT) models, researchers typically face a tradeoff between the precision and accuracy of the item parameter estimates. Given the pooling principle and variance-dependent shrinkage, the expected behavior of Bayesian hierarchical IRT models is to deliver more precise but biased item parameter estimates, compared to those obtained in nonhierarchical models. Previous research, however, points out the possibility that, in the context of the two-parameter logistic IRT model, the aforementioned tradeoff has not to be made. With a comprehensive simulation study, we provide an in-depth investigation into this possibility. The results show a superior performance, in terms of bias, RMSE and precision, of the hierarchical specifications compared to the nonhierarchical counterpart. Under certain conditions, the bias in the item parameter estimates is independent of the bias in the variance components. Moreover, we provide a bias correction procedure for item discrimination parameter estimates. In sum, we show that IRT models create a unique situation where the Bayesian hierarchical approach indeed yields parameter estimates that are not only more precise, but also more accurate, compared to nonhierarchical approaches. We discuss this beneficial behavior from both theoretical and applied point of views.

Revealing the characteristics of Indonesian language test used in the national-standardized school examinations

This study aimed to determine the characteristics of Indonesian language test used in the national standardized school examinations (Ujian Sekolah Berstandar Nasional, USBN). We used the response data of 218 students from a public senior high school in the Special Region of Yogyakarta, Indonesia, to the test consisting of two packages, A and B, in the 2018/2019 academic year to investigate the characteristics of the test and its items. Quantitative analysis using classical test theory (CTT) and one-parameter logistic item response theory (1-PL IRT) model was conducted to investigate the characteristics of the test and its items based on difficulty and discriminating power. The results of the study under the CTT showed that most test items in both package A and package B have difficulty in the easy category and the portion of items in the difficult category is no more than 10%. In addition, while the majority of test items in both test packages demonstrated good discriminating power, package B contained a high number of items with poor discriminating power (47.5%). Under 1-PL IRT, our study results indicated the dominance of items with difficulty level in the moderate category in both test packages. In addition to revealing the difficulty level of the test items, our study showed that the items in the difficult category under CTT and IRT were related to the topics of types of conjunction; spelling, grammar, and sentence structure; job application letter; and observation report text and its structure. The results of this study are expected to contribute to improving the quality of the Indonesian language test and the quality of learning on topics where students have difficulty based on test items in the difficult category.

Interchangeability between factor analysis, logistic IRT, and normal ogive IRT.

In existing studies, it has been argued that factor analysis (FA) is equivalent to item response theory (IRT) and that IRT models that use different functions (i.e., logistic and normal ogive) are also interchangeable. However, these arguments have weak links. The proof of equivalence between FA and normal ogive IRT assumes a normal distribution. The interchangeability between the logistic and normal ogive IRT models depends on a scaling constant, but few scholars have examined whether the usual values of 1.7 or 1.702 maximize interchangeability. This study addresses these issues through Monte Carlo simulations. First, the FA model produces almost identical results to those of the normal ogive model even under severe nonnormality. Second, no single scaling constant maximizes the interchangeability between logistic and normal ogive models. Instead, users should choose different scaling constants depending on their purpose in using a model and the number of response categories (i.e., dichotomous or polytomous). Third, the interchangeability between logistic and normal ogive models is determined by several conditions. The interchangeability is high if the data are dichotomous or if the latent variables follow a symmetric distribution, and vice versa. In summary, the interchangeability between FA and normal ogive models is greater than expected, but that between logistic and normal ogive models is not.

An Introduction to Bayesian Knowledge Tracing with pyBKT

This study aims to introduce Bayesian Knowledge Tracing (BKT), a probabilistic model used in educational data mining to estimate learners’ knowledge states over time. It also provides a practical guide to estimating BKT models using the pyBKT library available in Python. The first section presents an overview of BKT by explaining its theoretical foundations and advantages in modeling individual learning processes. In the second section, we describe different variants of the standard BKT model based on item response theory (IRT). Next, we demonstrate the estimation of BKT with the pyBKT library in Python, outlining data pre-processing steps, parameter estimation, and model evaluation. Different cases of knowledge tracing tasks illustrate how BKT estimates learners’ knowledge states and evaluates prediction accuracy. The results highlight the utility of BKT in capturing learners’ knowledge states dynamically. We also show that the model parameters of BKT resemble the parameters from logistic IRT models.

Computerized adaptive test strategies for the matrix reasoning subtest of the Wechsler Adult Intelligence Scale, 4th Edition (WAIS-IV).

Most neuropsychological tests were developed without the benefit of modern psychometric theory. We used item response theory (IRT) methods to determine whether a widely used test - the 26-item Matrix Reasoning subtest of the WAIS-IV - might be used more efficiently if it were administered using computerized adaptive testing (CAT). Data on the Matrix Reasoning subtest from 2197 participants enrolled in the National Neuropsychology Network (NNN) were analyzed using a two-parameter logistic (2PL) IRT model. Simulated CAT results were generated to examine optimal short forms using fixed-length CATs of 3, 6, and 12 items and scores were compared to the original full subtest score. CAT models further explored how many items were needed to achieve a selected precision of measurement (standard error ≤ .40). The fixed-length CATs of 3, 6, and 12 items correlated well with full-length test results (with r = .90, .97 and .99, respectively). To achieve a standard error of .40 (approximate reliability = .84) only 3-7 items had to be administered for a large percentage of individuals. This proof-of-concept investigation suggests that the widely used Matrix Reasoning subtest of the WAIS-IV might be shortened by more than 70% in most examinees while maintaining acceptable measurement precision. If similar savings could be realized in other tests, the accessibility of neuropsychological assessment might be markedly enhanced, and more efficient time use could lead to broader subdomain assessment.

Comparing different trend estimation approaches in country means and standard deviations in international large-scale assessment studies

One major aim of international large-scale assessments (ILSA) like PISA is to monitor changes in student performance over time. To accomplish this task, a set of common items (i.e., link items) is repeatedly administered in each assessment. Linking methods based on item response theory (IRT) models are used to align the results from the different assessments on a common scale. This work employs the one-parameter logistic (1PL) and the two-parameter logistic (2PL) IRT models as scaling models for dichotomous item response data. The present article discusses different types of trend estimates in country means and standard deviations for countries in ILSA. These types differ in three aspects. First, the trend can be assessed by an indirect or direct linking approach for linking a country’s performance at an international metric. Second, the linking for the trend estimation can rely on either all items or only the link items. Third, item parameters can be assumed to be invariant or noninvariant across countries. It is shown that the most often employed trend estimation methods of original trends and marginal trends can be conceived as particular cases of indirect and direct linking approaches, respectively. Through a simulation study and analytical derivations, it is demonstrated that trend estimates using a direct linking approach and those that rely on only link items outperformed alternatives for the 1PL model with uniform country differential item functioning (DIF) and the 2PL model with uniform and nonuniform country DIF. We also illustrated the performance of the different scaling models for assessing the PISA trend from PISA 2006 to PISA 2009 in the cognitive domains of reading, mathematics, and science. In this empirical application, linking errors based on jackknifing testlets were utilized that adequately quantify DIF effects in the uncertainty of trend estimates.

Revisiting the Samejima–Bolfarine–Bazán IRT models: New features and extensions

In 2010, the Samejima–Bolfarine–Bazán (SBB) Item Response Theory (IRT) models were introduced by (Journal of Educational and Behavioral Statistics 35 (2010) 693–713) under a Bayesian approach. These models extend the regular Bayesian One and Two Parameter Logistic IRT models by incorporating a parameter accounting for asymmetry of the Item Characteristic Curve (ICC) which is named the complexity of the item. It includes the Logistic Positive Exponent (LPE) IRT model formulated initially by (Psychometrika 65 (2000) 319–335) and the Reflection of the LPE (RLPE). In the present work, new properties of the SBB models are developed including a random effect for testlet structures with a Bayesian inference through a Markov chain Monte Carlo (MCMC) algorithm which includes the parameter estimation and model comparison. The asymmetric behavior of the Item Characteristic Curve (ICC) is detected using a marginal item information function. Two simulation studies are developed to analyze the sensitiveness of the penalized parameter in the asymmetric behavior of the ICC and to evaluate the parameter recovery of the proposed model. A real data set, with a testlet structure and empirical evidence of asymmetric behavior of the ICCs, is used to apply the models.

Reducing the Bias of Norm Scores in Non-Representative Samples: Weighting as an Adjunct to Continuous Norming Methods.

We investigated whether the accuracy of normed test scores derived from non-demographically representative samples can be improved by combining continuous norming methods with compensatory weighting of test results. To this end, we introduce Raking, a method from social sciences, to psychometrics. In a simulated reference population, we modeled a latent cognitive ability with a typical developmental gradient, along with three demographic variables that were correlated to varying degrees with the latent ability. We simulated five additional populations representing patterns of non-representativeness that might be encountered in the real world. We subsequently drew smaller normative samples from each population and used an one-parameter logistic Item Response Theory (IRT) model to generate simulated test results for each individual. Using these simulated data, we applied norming techniques, both with and without compensatory weighting. Weighting reduced the bias of the norm scores when the degree of non-representativeness was moderate, with only a small risk of generating new biases.