Bayesian Item Response Theory Model Research Articles

A Bayesian nonparametric approach for handling item and examinee heterogeneity in assessment data.

We propose a novel nonparametric Bayesian item response theory model that estimates clusters at the question level, while simultaneously allowing for heterogeneity at the examinee level under each question cluster, characterized by a mixture of binomial distributions. The main contribution of this work is threefold. First, we present our new model and demonstrate that it is identifiable under a set of conditions. Second, we show that our model can correctly identify question-level clusters asymptotically, and the parameters of interest that measure the proficiency of examinees in solving certain questions can be estimated at a rate (up to a log term). Third, we present a tractable sampling algorithm to obtain valid posterior samples from our proposed model. Compared to the existing methods, our model manages to reveal the multi-dimensionality of the examinees' proficiency level in handling different types of questions parsimoniously by imposing a nested clustering structure. The proposed model is evaluated via a series of simulations as well as apply it to an English proficiency assessment data set. This data analysis example nicely illustrates how our model can be used by test makers to distinguish different types of students and aid in the design of future tests.

Examining Measurement Invariance in Bayesian Item Response Theory Models: A Simulation Study

The aim of the study is to determine a measurement invariance cut-off point based on item parameter differences in Bayesian Item Response Theory Models. Within this scope, the Bayes factor is estimated for testing measurement invariance. For this purpose, a simulation study is conducted. The data were generated in the R software for each simulation condition under the one-parameter logistic model for 10 binary (1-0 scored) items. The invariance test was performed for various group sizes (n=500, 1000, 1500 and 2000) and difficulty parameters (dk=0, dk=0.1, dk=0.3, dk=0.5 and dk=0.7). The Bayesian analyzes were performed on the WINBUGS using the codes written in the R. A Bayes factor that provides evidence for measurement invariance was calculated depending on the parameter differences. The Savage–Dickey density ratio, one of the MCMC sampling schemas, was used to calculate the Bayes factor. As a result, if the item parameter difference is dk=0.3 and group sizes are 1500 or larger, the measurement invariance cannot be achieved. However, for small sample sizes (n=1000 or less) measurement invariance interpretation should be done carefully. When the dk=0.5, there are invariant items only in n=500. According to Bayes factor test results, evidence has been produced when dk=0.7 that measurement invariance cannot be achieved.

Household resilience realized by photovoltaic and battery energy system in natural hazard-triggered blackouts: Evidence from Japan

Photovoltaic (PV) panels, increasingly popular due to decarbonization, are expected to increase household resilience during natural hazard-triggered blackouts. Their effectiveness is also expected to improve with the addition of storage batteries. However, PV ownership does not necessarily lead to electricity use in blackouts; even when used, the actual use of electrical appliances is unclear. We examined households that owned PV panels and experienced a natural hazard-triggered blackout to determine what barriers and facilitating factors existed for using PV electricity, which electrical appliances could be used with PV, and whether the benefits of PV changed with storage batteries. A questionnaire survey targeted households in Japan that experienced blackouts caused by the 2018 Hokkaido Eastern Iburi earthquake and the 2019 Typhoon Faxai, respectively, and their responses (n = 282 and 259, respectively) were analyzed. Descriptive statistics showed that while it was sometimes necessary to switch PV systems to stand-alone operation mode during blackouts, most respondents who did not use PV systems did not know how to operate them, which was a barrier to their use. The main facilitating factors were preparation like advance explanations from the sales staff and the regular manual check for use in case of blackouts. Bayesian item response theory models demonstrated that PV electricity enabled households to use a variety of electrical appliances, including those related to food and communication (i.e., refrigerators and cell phone chargers), whose needs increased during the blackouts. The probability of each appliance being used increased in households with batteries. The above results were commonly confirmed regardless of hazard type. These findings will improve the disaster resilience of households equipped with PV panels and batteries, as well as spread the installation of such equipment by highlighting their benefits.

Measuring accountability in interlocal agreements between Indigenous and local governments

AbstractInterlocal agreements are becoming a popular policy tool for facilitating intergovernmental coordination and cooperation in Canada and the United States. Indigenous and local governments are also turning to these agreements despite long histories of colonialism, exploitation and dispossession by the settler State toward Indigenous communities. To what extent do interlocal agreements between Indigenous and municipal governments require stringent accountability measures to facilitate intergovernmental coordination? Using a hierarchical Bayesian item response theory model, we explore this question by analyzing 317 interlocal agreements between Indigenous and municipal communities in Canada. We find that accountability strength varies significantly across agreements, contrary to our expectation that accountability requirements would be strong across agreements due to the long history of colonialism. We also find that some of the variation may be a function of the policy area addressed by each agreement, although this finding is likely the result of measurement uncertainty in our estimates.

Robustness of the performance of the optimized hierarchical two-parameter logistic IRT model for small-sample item calibration

Hierarchical Bayesian modeling is beneficial when complex models with many parameters of the same type, such as item response theory (IRT) models, are to be estimated with sparse data. Recently, Koenig et al. (Applied Psychological Measurement, 44, 311–326, 2020) illustrated in an optimized hierarchical Bayesian two-parameter logistic model (OH2PL) how to avoid bias due to unintended shrinkage or degeneracies of the posterior, and how to benefit from this approach in small samples. The generalizability of their findings, however, is limited because they investigated only a single specification of the hyperprior structure. Consequently, in a comprehensive simulation study, we investigated the robustness of the performance of the novel OH2PL in several specifications of their hyperpriors under a broad range of data conditions. We show that the novel OH2PL in the half-Cauchy or Exponential configuration yields unbiased (in terms of bias) model parameter estimates in small samples of N = 50. Moreover, it outperforms (especially in terms of the RMSE of the item discrimination parameters) marginal maximum likelihood (MML) estimation and its nonhierarchical counterpart. This further corroborates the possibility that hierarchical Bayesian IRT models behave differently than general hierarchical Bayesian models. We discuss these results regarding the applicability of complex IRT models in small-scale situations typical in psychological research, and illustrate the extended applicability of the 2PL IRT model with an empirical example.

Flexible Bayesian modelling in dichotomous item response theory using mixtures of skewed item curves.

Most item response theory (IRT) models for dichotomous responses are based on probit or logit link functions which assume a symmetric relationship between the probability of a correct response and the latent traits of individuals taking a test. This assumption restricts the use of those models to the case in which all items behave symmetrically. On the other hand, asymmetric models proposed in the literature impose that all the items in a test behave asymmetrically. This assumption is inappropriate for great majority of tests which are, in general, composed of both symmetric and asymmetric items. Furthermore, a straightforward extension of the existing models in the literature would require a prior selection of the items' symmetry/asymmetry status. This paper proposes a Bayesian IRT model that accounts for symmetric and asymmetric items in a flexible but parsimonious way. That is achieved by assigning a finite mixture prior to the skewness parameter, with one of the mixture components being a point mass at zero. This allows for analyses under both model selection and model averaging approaches. Asymmetric item curves are designed through the centred skew normal distribution, which has a particularly appealing parametrization in terms of parameter interpretation and computational efficiency. An efficient Markov chain Monte Carlo algorithm is proposed to perform Bayesian inference and its performance is investigated in some simulated examples. Finally, the proposed methodology is applied to a data set from a large-scale educational exam in Brazil.

Measuring public support for European integration using a Bayesian item response theory model

This study proposes the use of Bayesian item response theory models to measure aggregate public support for European integration. This approach addresses the limitations of other indicators and produces valid estimates of public attitudes over long time periods, even when available indicators change over time or present interruptions. I compare Bayesian item response theory models with alternative approaches used in the study of support for European integration, and demonstrate that they produce more accurate estimates of latent public opinion. The estimates are validated by showing their association both to alternative public opinion measures and to the vote share of Eurosceptic parties across Europe. I show that Bayesian models solve unaddressed issues like ensuring cross-country comparability of the estimates and modelling responses with multiple answer options.

A Bayesian Measurement of Political Connection and Entrepreneur Preference on Trade and Competition

This article investigates how political connection shapes firm owner preference on economic openness and international competition in China, a topic that is getting increasingly relevant today amid China’s trade disputes with its trading partners over the nature of the country’s business-government relations. Politically connected entrepreneurs and their enterprises usually exploit and benefit from their political resources, but this can lead to both supporting and opposing views on expanding trade liberalization. To solve this puzzle, this article proposes a theory that focuses on (1) a selection effect of political connection on firm productivity and (2) trade-related institutional development in China. With survey data on China joining the world trade organization, I develop a modified Bayesian item response theory model to measure political connection and find that Chinese politically connected entrepreneurs held a less supportive view before joining the world trade organization than their less connected counterparts. This suggests an anticipation that the imminent opening would neutralize the privileges of politically connected entrepreneurs. By looking at trade liberalization, this article offers a firm level analysis that political opposition to sustained economic reform may derive from the short-term winners, instead of the losers, in transitional societies.

Bayesian Item Response Modeling in R with brms and Stan

Item response theory (IRT) is widely applied in the human sciences to model persons' responses on a set of items measuring one or more latent constructs. While several R packages have been developed that implement IRT models, they tend to be restricted to respective pre-specified classes of models. Further, most implementations are frequentist while the availability of Bayesian methods remains comparably limited. I demonstrate how to use the R package brms together with the probabilistic programming language Stan to specify and fit a wide range of Bayesian IRT models using flexible and intuitive multilevel formula syntax. Further, item and person parameters can be related in both a linear or non-linear manner. Various distributions for categorical, ordinal, and continuous responses are supported. Users may even define their own custom response distribution for use in the presented framework. Common IRT model classes that can be specified natively in the presented framework include 1PL and 2PL logistic models optionally also containing guessing parameters, graded response and partial credit ordinal models, as well as drift diffusion models of response times coupled with binary decisions. Posterior distributions of item and person parameters can be conveniently extracted and postprocessed. Model fit can be evaluated and compared using Bayes factors and efficient cross-validation procedures.

Hot under the collar: A latent measure of interstate hostility

The majority of studies on international conflict escalation use a variety of measures of hostility including the use of force, reciprocity, and the number of fatalities. The use of different measures, however, leads to different empirical results and creates difficulties when testing existing theories of interstate conflict. Furthermore, hostility measures currently used in the conflict literature are ill suited to the task of identifying consistent predictors of international conflict escalation. This article presents a new dyadic latent measure of interstate hostility, created using a Bayesian item-response theory model and conflict data from the Militarized Interstate Dispute (MID) and Phoenix political event datasets. This model (1) provides a more granular, conceptually precise, and validated measure of hostility, which incorporates the uncertainty inherent in the latent variable; and (2) solves the problem of temporal variation in event data using a varying-intercept structure and human-coded data as a benchmark against which biases in machine-coded data are corrected. In addition, this measurement model allows for the systematic evaluation of how existing measures relate to the construct of hostility. The presented model will therefore enhance the ability of researchers to understand factors affecting conflict dynamics, including escalation and de-escalation processes.

Field evidence for two paths to cross-cultural competence: implications for cultural dynamics.

Interaction between members of culturally distinct (ethnic) groups is an important driver of the evolutionary dynamics of human culture, yet relevant mechanisms remain underexplored. For example, cultural loss resulting from integration with culturally distinct immigrants or colonial majority populations remains a topic whose political salience exceeds our understanding of mechanisms that may drive or impede it. For such dynamics, one mediating factor is the ability to interact successfully across cultural boundaries (cross-cultural competence). However, measurement difficulties often hinder its investigation. Here, simple field methods in a uniquely suited Amazonian population and Bayesian item-response theory models are used to derive the first experience-level measure of cross-cultural competence, as well as evidence for two developmental paths: cross-cultural competence may emerge as a side effect of adopting out-group cultural norms, or it may be acquired while maintaining in-group norms. Ethnographic evidence suggests that the path taken is a likely consequence of power differences in inter- vs intra-group interaction. The former path, paralleling language extinction, may lead to cultural loss; the latter to cultural sustainability. Recognition of such path-dependent effects is vital to theory of cultural dynamics in humans and perhaps other species, and to effective policy promoting cultural diversity and constructive inter-ethnic interaction.

A latent variable approach to measuring and explaining peace agreement strength

AbstractMuch of the peace agreement durability literature assumes that stronger peace agreements are more likely to survive the trials of the post-conflict environment. This work does an excellent job identifying which provisions indicate that agreements are more likely to endure. However, there is no widely accepted way to directly measure the strength of agreements, and existing measures suffer from a lack of nuance or reliance on subjective weighting. We use a Bayesian item response theory model to develop a principled measure of the latent strength of peace agreements in civil conflicts from 1975 to 2005. We illustrate the measure's utility by exploring how various international factors such as sanctions and mediation contribute to the strength or weakness of agreements.

Application of a Bayesian graded response model to characterize areas of disagreement between clinician and patient grading of symptomatic adverse events

BackgroundTraditional concordance metrics have shortcomings based on dataset characteristics (e.g., multiple attributes rated, missing data); therefore it is necessary to explore supplemental approaches to quantifying agreement between independent assessments. The purpose of this methodological paper is to apply an Item Response Theory (IRT) -based framework to an existing dataset that included unidimensional clinician and multiple attribute patient ratings of symptomatic adverse events (AEs), and explore the utility of this method in patient-reported outcome (PRO) and health-related quality of life (HRQOL) research.MethodsData were derived from a National Cancer Institute-sponsored study examining the validity of a measurement system (PRO-CTCAE) for patient self-reporting of AEs in cancer patients receiving treatment (N = 940). AEs included 13 multiple attribute patient-reported symptoms that had corresponding unidimensional clinician AE grades. A Bayesian IRT Model was fitted to calculate the latent grading thresholds between raters. The posterior mean values of the model-fitted item responses were calculated to represent model-based AE grades obtained from patients and clinicians.ResultsModel-based AE grades showed a general pattern of clinician underestimation relative to patient-graded AEs. However, the magnitude of clinician underestimation was associated with AE severity, such that clinicians’ underestimation was more pronounced for moderate/very severe model-estimated AEs, and less so with mild AEs.ConclusionsThe Bayesian IRT approach reconciles multiple symptom attributes and elaborates on the patterns of clinician-patient non-concordance beyond that provided by traditional metrics. This IRT-based technique may be used as a supplemental tool to detect and characterize nuanced differences in patient-, clinician-, and proxy-based ratings of HRQOL and patient-centered outcomes.Trial registrationClinicalTrials.gov NCT01031641. Registered 1 December 2009.

The Bayesian Multilevel Trifactor Item Response Theory Model.

Advancements in item response theory (IRT) have led to models for dual dependence, which control for cluster and method effects during a psychometric analysis. Currently, however, this class of models does not include one that controls for when the method effects stem from two method sources in which one source functions differently across the aspects of another source (i.e., a nested method-source interaction). For this study, then, a Bayesian IRT model is proposed, one that accounts for such interaction among method sources while controlling for the clustering of individuals within the sample. The proposed model accomplishes these tasks by specifying a multilevel trifactor structure for the latent trait space. Details of simulations are also reported. These simulations demonstrate that this model can identify when item response data represent a multilevel trifactor structure, and it does so in data from samples as small as 250 cases nested within 50 clusters. Additionally, the simulations show that misleading estimates for the item discriminations could arise when the trifactor structure reflected in the data is not correctly accounted for. The utility of the model is also illustrated through the analysis of empirical data.

Scoring Depression on a Common Metric: A Comparison of EAP Estimation, Plausible Value Imputation, and Full Bayesian IRT Modeling

There are a growing number of item response theory (IRT) studies that calibrate different patient-reported outcome (PRO) measures, such as anxiety, depression, physical function, and pain, on common, instrument-independent metrics. In the case of depression, it has been reported that there are considerable mean score differences when scoring on a common metric from different, previously linked instruments. Ideally, those estimates should be the same. We investigated to what extent those differences are influenced by different scoring methods that take into account several levels of uncertainty, such as measurement error (through plausible value imputation) and item parameter uncertainty (through full Bayesian IRT modeling). Depression estimates from different instruments were more similar, and their corresponding confidence/credible intervals were larger when plausible value imputation or Bayesian modeling was used, compared to the direct use of expected a posteriori (EAP) estimates. Furthermore, we explored the use of Bayesian IRT models to update item parameters based on newly collected data.

Southern realignment, party sorting, and the polarization of American primary electorates, 1958–2012

Many scholars have argued that primary elections are an important factor in the polarization of the American Congress. Yet little research measures change in the policy preferences of primary electorates to evaluate the connection directly. We create the first explicit measures of the preferences of primary voters over the last 60 years using a Bayesian item-response theory model. Although the overall distribution of population preferences has changed little, the preferences of primary voters are now much more related to the party of the primary that they attend. We show that liberals are much more likely to turn out in Democratic primaries and conservatives are much more likely to turn out in Republican primaries. We estimate that the divergence of primary from general electorates is six times larger in 2012 than in 1958 owing to this “primary sorting”. This trend began with the emergence of the Southern Republicans. As the Republican party became viable, conservative Southerners switched to Republican primaries leading to a leftward shift in Democratic primary electorates. Nationwide, primary sorting began sometime after it began in the South. We speculate that Southern realignment played a clarifying role that contributed to subsequent sorting of primary electorates nationwide.

Parallel Markov chain Monte Carlo for Bayesian dynamic item response models in educational testing

Bayesian dynamic item response models have been successfully used for educational testing data; these models are especially useful for individually varying and irregularly spaced longitudinal testing data. However, because of the complexity of the models and the large size of the data sets, computation time is excessive for carrying out full data analyses in practice. Here, we introduce a parallel Markov chain Monte Carlo method to speed the implementation of these Bayesian models. Using both simulation data and real educational testing data for reading ability, we demonstrate that computation time is greatly reduced for our parallel computing method versus full data analyses. The estimated error of our method is shown to be small, using common distance metrics. Our parallel computing approach can be used for other models in the Educational and Psychometric fields, including Bayesian item response theory models. Copyright © 2017 John Wiley & Sons, Ltd.

A Bayesian Robust IRT Outlier-Detection Model.

In psychometric practice, the parameter estimates of a standard item-response theory (IRT) model can become biased when item-response data, of persons' individual responses to test items, contain outliers relative to the model. Also, the manual removal of outliers can be a time-consuming and difficult task. Besides, removing outliers leads to data information loss in parameter estimation. To address these concerns, a Bayesian IRT model that includes person and latent item-response outlier parameters, in addition to person ability and item parameters, is proposed and illustrated, and is defined by item characteristic curves (ICCs) that are each specified by a robust, Student's t-distribution function. The outlier parameters and the robust ICCs enable the model to automatically identify item-response outliers, and to make estimates of the person ability and item parameters more robust to outliers. Hence, under this IRT model, it is unnecessary to remove outliers from the data analysis. Our IRT model is illustrated through the analysis of two data sets, involving dichotomous- and polytomous-response items, respectively.

A novel method for expediting the development of patient-reported outcome measures and an evaluation across several populations.

Item response theory (IRT) models provide an appropriate alternative to the classical ordinal confirmatory factor analysis (CFA) during the development of patient-reported outcome measures (PROMs). Current literature has identified the assessment of IRT model fit as both challenging and underdeveloped (Sinharay & Johnson, 2003; Sinharay, Johnson, & Stern, 2006). This study evaluates the performance of Ordinal Bayesian Instrument Development (OBID), a Bayesian IRT model with a probit link function approach, through applications in two breast cancer-related instrument development studies. The primary focus is to investigate an appropriate method for comparing Bayesian IRT models in PROMs development. An exact Bayesian leave-one-out cross-validation (LOO-CV) approach (Vehtari & Lampinen, 2002) is implemented to assess prior selection for the item discrimination parameter in the IRT model and subject content experts' bias (in a statistical sense and not to be confused with psychometric bias as in differential item functioning) toward the estimation of item-to-domain correlations. Results support the utilization of content subject experts' information in establishing evidence for construct validity when sample size is small. However, the incorporation of subject experts' content information in the OBID approach can be sensitive to the level of expertise of the recruited experts. More stringent efforts need to be invested in the appropriate selection of subject experts to efficiently use the OBID approach and reduce potential bias during PROMs development.

Measuring Foreign Policy Positions of Members of the US Congress

Studies have shown that a foreign policy position of a member of Congress is often distinct from a domestic one. Despite this, measures commonly used to determine the foreign policy positions of members of Congress are based on congressional votes on domestic as well as foreign policy matters. As foreign policy votes take up only a small portion of all congressional votes, these measures conflate a member’s foreign policy position with his or her domestic policy position. While there are other measures based exclusively on foreign policy votes, these are also problematic because they tend to use a small number of controversial votes and thus inflate extremism. To address these shortcomings, I present a new measure by applying a Bayesian item response theory model to all foreign policy votes. This paper demonstrates the similarities, differences, and advantages of this measure by comparing it with the existing measures in a series of analyses of foreign policy positions of political parties and individual legislators.