Application Of Item Response Theory Research Articles

The application of item response theory in developing and validating a shortened version of the Rotterdam Emotional Intelligence Scale

The application of item response theory in developing and validating a shortened version of the Rotterdam Emotional Intelligence Scale

Use of computerised adaptive testing to reduce the number of items in patient-reported hip and knee outcome scores: an analysis of the NHS England National Patient-Reported Outcome Measures programme

ObjectiveOver 160 000 participants per year complete the 12-item Oxford Hip and Knee Scores (OHS/OKS) as part of the NHS England Patient-Reported Outcome Measures (PROMs) programme. We used a modern...

Item Response Theory True Score Equating for the Bifactor Model Under the Common-Item Nonequivalent Groups Design.

Applying item response theory (IRT) true score equating to multidimensional IRT models is not straightforward due to the one-to-many relationship between a true score and latent variables. Under the common-item nonequivalent groups design, the purpose of the current study was to introduce two IRT true score equating procedures that adopted different dimension reduction strategies for the bifactor model. The first procedure, which was referred to as the integration procedure, linked the latent variable scales for the bifactor model and integrated out the specific factors from the item response function of the bifactor model. Then, IRT true score equating was applied to the marginalized bifactor model. The second procedure, which was referred to as the PIRT-based procedure, projected the specific dimensions onto the general dimension to obtain a locally dependent unidimensional IRT (UIRT) model and linked the scales of the UIRT model, followed by the application of IRT true score equating to the locally dependent UIRT model. Equating results obtained with the two equating procedures along with those obtained with the unidimensional three-parameter logistic (3PL) model were compared using both simulated and real data. In general, the integration and PIRT-based procedures provided equating results that were not practically different. Furthermore, the equating results produced by the two bifactor-based procedures became more accurate than the results returned by the 3PL model as tests became more multidimensional.

Affective components of gastronomy tourism: measurement scale development and validation

PurposeOur knowledge of what emotions are elicited explicitly from food consumption and gastronomy experiences in the travel destination and how these emotions establish a relationship with tourists’ behavior is limited. Thus, this study aims to enrich the current knowledge in the gastronomy tourism field from the affective experience perspective and develop a scale to measure tourists’ affective gastronomy experiences (TAGES).Design/methodology/approachBoth qualitative scale development and quantitative scale validation were conducted to ensure the psychometric properties of TAGES.FindingsWith the focus group’s contributions and experts’ validation, 12 gastronomy experience affects were identified in the first stage. In the second stage, a quantitative data collection involving 650 samples helped refine the scale. Finally, a reliable and valid scale with five items measuring TAGES was successfully developed.Originality/valueThis study provides a novel perspective by viewing tourists’ gastronomy experiences through an affective lens. Moreover, this study successfully provides evidence for the psychometric properties of the newly developed TAGES by systematically applying item response theory (IRT) and classical test theory (CTT). This study enriches the gastronomy tourism domain by developing the TAGES and presenting a rigorous and exhaustive investigation of its psychometric properties based on an integration of IRT and CTT. A valid and reliable scale that measures the TAGES fills the gastronomy literature gap and proposes an effective tool for future gastronomy experience studies.

The Adaptability of Passion Scale for Tertiary Students in Japanese Online Courses in Indonesia

This study aimed at examining the opportunities for adapting the passion scale to Japanese literature students who carried out the online learning process in Indonesia. This study used quantitative covariance structure design as the application of item response theory. From a sample of 148 undergraduate students, originally from three different courses. Data were obtained by filling out a virtual questionnaire which adapted from the original Dual Model Passion scale. Since this scale had been developed abroad, the data were analysed quantitatively using confirmatory factor analysis and regression, this study replicated the similar analysis while examined passion by using five criteria such as time, like, value, passion, identity as used in the previous development process. In addition, the criteria for student characteristics (gender and age) and conditions (first year vs. advanced year) in attending online lectures by observing the relationship between the two main dimensions of passion). The results showed that first year students tend to have a higher harmonious passion (HP), while second year students had a higher obsessive passion (OP). In conclusion, several things needed to be considered by researchers and lecturers at tertiary institutions such as course and student characteristics, so that students' passion remained good even though they had to study online during the outbreak of the Covid-19.

Application of Item Response Theory (IRT)-Graded Response Model (GRM) to Entrepreneurial Ecosystem Scale

The scale of entrepreneurial ecosystems (EE) assesses the perceptions about entrepreneurial ecosystem domains, finances, capital finances, support, support professions, policies, markets, human resources, and culture. The scales are always error-prone—these scales must possess properties that enable them it to provide maximum information and validity reliability. Convenient sampling data from (n = 474) founders, co-founders, and entrepreneurs were collected. The IRT-GRM model is used to validate and test the instrument-based on polytomous scales. IRT yields discriminating power—the level of difficulty of the items of the scale. The scale consists of 48 items. The item Pol5 (4.13) was found to have the highest discriminating value (4.13), the item mar5 had the lowest discriminating value (1.57), and all items had discriminating values greater than the threshold value of 0.60. The EE Scale showed good reliability based on McDonald’s omega and Cronbach’s alpha (0.80 and 0.88). The parallel and factor analysis showed good agreement of the one-dimesnionality of the scale. The model goodness of fit statistics based on the comparative fit index (CFI) and the Tucker–Lewis index, (TLI) and the standardized root mean square residual (SRMR) showed a satisfactory level of fit; however, the root mean square error of approximation (RMSE) showed a poor fit. The item characteristic curves showed that the all item responses were properly ordered. The items of the scale showed a satisfactory level of discrimination power and level of difficulty, and it was found to have three levels of agreement about entrepreneurial ecosystem scale. It is concluded that the EE scale possesses good psychometric properties and that it is reliable and valid instrument to measure the entrepreneurial ecosystem of the given region.

The Measurement Performance of the Parkinson's Disease Activities of Daily Living, Interference, and Dependence Instrument.

Parkinson's disease is a neurodegenerative disease that can be associated with motor fluctuations that result in substantial negative impact to an individual's activities of daily living. Understanding the patient's perspective about the impact of Parkinson's disease therapies is an important part of drug development and shared treatment decision-making. The objective of this research was to examine the structure, scoring, internal consistency, test-retest reliability, and concurrent and known groups validity of the Parkinson's Disease Activities of Daily Living, Interference and Dependence (PD-AID) instrument, a new, patient-reported outcomes instrument, developed to assess the clinical benefit of Parkinson's disease treatment from the patient's perspective. This was a non-interventional study among persons with mild-to-moderate Parkinson's disease currently using and responding to L-Dopa. The structure of the measure was confirmed applying item response theory to data from baseline, supporting 4 candidate scores. Baseline Patient Global Impression of Severity ratings facilitated known-groups analysis. Data from all participants were used to estimate test-retest reliability. Concurrent validity was assessed using correlations with related measures. Participants (n = 94) were mean age 69 years (mean time since diagnosis 6.9 years); 34 experienced L-Dopa-related dyskinesia. Psychometric models supported 4 candidate scoring regimes for the PD-AID. All exhibited adequate reliability and validity characteristics and strong internal consistency. Correlations with reference measures were in the expected direction and range of magnitude. Analyses supported the PD-AID as fit-for-purpose, producing psychometrically sound scores. Further research to confirm the measurement properties of the PD-AID in an expanded sample and to establish thresholds for meaningful score changes is recommended.

Development of empirically derived brief program evaluation measures in Pennsylvania first-episode psychosis coordinated specialty care programs.

The Pennsylvania first episode psychosis program evaluation (PA-FEP-PE) core assessment battery was developed as a standard and comprehensive clinical assessment and data collection tool in Pennsylvania coordinated specialty care programs (CSC). To reduce administrative time and maximize clinical utility by maintaining acceptable levels of precision, we aimed to generate a short form using item response theory (IRT)-based computer-adaptive test (CAT) simulation and analyse the implementation and acceptability of the short form among providers from PA-CSC. FEP participants (n=759; age 14-36) from nine coordinated specialty care programs completed 156 items drawn from the PA-FEP-PE battery. Multidimensional IRT-based CAT simulations were used to select the best PA-FEP-PE items for abbreviated forms. A 67-item PA-FEP-PE short form was developed to capture six factors: (1) positive affect and surgency (with negative loadings on Anxious-Misery items); (2) psychiatric services satisfaction; (3) antipsychotic side effect severity; (4) family turmoil and associated traumas; (5) trauma load; and (6) psychosis. The total number of items was reduced more than 50% in the PA-FEP-PE shortened forms. The short form demonstrated good psychometric properties, and it was well accepted by our providers in the implementation. The empirical derivation and implementation of abbreviated measures of key domains and constructs in FEP care have streamlined and facilitated PA-FEP program evaluation. Our work supports potential application of IRT based methods to empirically reduce core assessment battery measures in large-scale data collection efforts such as in the Early Psychosis Intervention Network.

Facilitating International Research on Career Indecision: Developing Career Indecision Profile-Short-5 in China and the U.S.

While five-factor model of career indecision proposes five cross-culturally important indecision factors, a psychologically sound brief measure of career indecision that corresponds to the five-factor model is lacking. Thus, using a Chinese college sample ( n = 588) and an U.S. college sample ( n = 762), this study developed and validated the Career Indecision Profile-Short-5 factor (CIP-Short-5) in China and the U.S. Applying item response theory, the CIP-Short-5 consists of five items each for the five overarching domains of career indecision. It showed desirable option occurrence patterns and minimal gender-oriented differential item functioning in both China and the U.S. Additionally, the scale supported the five-factor model over an alternative, four-factor model of career indecision in both China and the U.S. The convergent and discriminant patterns of the CIP-Short-5 with criteria were largely supported in China and the U.S. Last, the results supported the configural and metric invariance of the CIP-Short-5 but did not fully support the scalar invariance of the CIP-Short-5 across China and the U.S. Together, the results offer psychometric evidence for the CIP-Short-5, which has important implications for research and practice on career indecision in the international context in general and in China in specific.

Item Complexity: A Neglected Psychometric Feature of Test Items?

Despite its frequent consideration in test development, item complexity receives little attention in the psychometric modeling of item response data. In this address, I consider how variability in item complexity can be expected to emerge in the form of item characteristic curve (ICC) asymmetry, and how such effects may significantly influence applications of item response theory, especially those that assume interval-level properties of the latent proficiency metric and groups that vary substantially in mean proficiency. One application is the score gain deceleration phenomenon often observed in vertical scaling contexts, especially in subject areas like math or second language acquisition. It is demonstrated how the application of symmetric IRT models in the presence of complexity-induced positive ICC asymmetry can be a likely cause. A second application concerns the positive correlation between DIF and difficulty commonly seen in verbal proficiency (and other subject area) tests where problem-solving complexity is minimal and proficiency-related guessing effects are likely pronounced. Here we suggest negative ICC asymmetry as a probable cause and apply sensitivity analyses to demonstrate the ease with which such correlations disappear when allowing for negative ICC asymmetry. Unfortunately, the presence of systematic forms of ICC asymmetry is easily missed due to the considerable flexibility afforded by latent trait metrics in IRT. Speculation is provided regarding other applications for which attending to ICC asymmetry may prove useful.

Bayesian Approaches for Detecting Differential Item Functioning Using the Generalized Graded Unfolding Model.

Differential item functioning (DIF) analysis is one of the most important applications of item response theory (IRT) in psychological assessment. This study examined the performance of two Bayesian DIF methods, Bayes factor (BF) and deviance information criterion (DIC), with the generalized graded unfolding model (GGUM). The Type I error and power were investigated in a Monte Carlo simulation that manipulated sample size, DIF source, DIF size, DIF location, subpopulation trait distribution, and type of baseline model. We also examined the performance of two likelihood-based methods, the likelihood ratio (LR) test and Akaike information criterion (AIC), using marginal maximum likelihood (MML) estimation for comparison with past DIF research. The results indicated that the proposed BF and DIC methods provided well-controlled Type I error and high power using a free-baseline model implementation, their performance was superior to LR and AIC in terms of Type I error rates when the reference and focal group trait distributions differed. The implications and recommendations for applied research are discussed.

Language comprehension in toddlers with significant developmental delays: An IRT approach

PurposeLanguage comprehension, or the ability to understand spoken language, is critical to a variety of child outcomes. Effective early intervention relies on valid, reliable language comprehension assessment. The purpose of this study was to explore language comprehension in a sample of toddlers with significant developmental delays associated with varied medical conditions. MethodWe investigated language comprehension in a sample of 112 toddlers by applying Item Response Theory (IRT) methods to two measures; one standardized and one flexible. ResultsData from a standardized measure fit the unidimensional model, whereas the flexible measure did not. The overall pattern of results suggested that items related to early social/contextual comprehension are distinct from linguistic comprehension items. ConclusionOur findings inform clinical practice by underscoring the importance of comprehensive assessment of language comprehension and considering strengths and weaknesses across social/contextual and linguistic comprehension among toddlers with developmental delays.

The Creation of a Correctional Officer Trainee Self-Efficacy Index: An Application of Item Response Theory

Correctional staff turnover has been linked to burnout in current staff, risk of physical injury due to understaffing, and high replacement costs due to onboarding processes. To date, much research has been devoted to job characteristics and demographic profiles to understand turnover, but little research has examined highly dynamic factors, such as self-efficacy. Given that self-efficacy has been linked to absenteeism, turnover, and engagement across vocational contexts, the current study seeks to provide practitioners and researchers with the tools to examine self-efficacy through the Correctional Officer Trainee Self-Efficacy Index. Comprising each task that correctional officer training academies train in, the self-efficacy index can be used to identify areas improvement in curricula or in individual officer development throughout training. Using the graded response model of item response theory, we examine item-level properties of the index using a sample of 673 pre-service correctional officers across two Midwestern and one Southern state. Results indicated that the self-efficacy indices loaded onto two factors. Each item had high discrimination, and the range of difficulty parameters for the items was wide. Overall, the correctional officer trainee task self-efficacy index is effective for those with low to high self-efficacy for correctional officer tasks.

Investigating the Practical Utility of Proportion Agreement, Simulation-based Proportion Agreement, and Concordance Index for Item Fit Assessment in Item Response Theory

Item response theory (IRT) is a modern measurement technique and widely used in psychology and education. Popular IRT models for dichotomously scored item response data are the 2-parameter logistic model (2PLM) and the 3-parameter logistic model (3PLM). In applications of IRT, item fit investigation is typically conducted. To complement the existing well-known statistical significance test of item fit such as and, this study proposes and explores the utility of proportion agreement (PA), its refined version, PA2, and concordance index (CD), which are descriptive item fit measures equipped with efficient computation and easy understanding of the degree of item fit. The performance of PA, PA2, and CD was also compared with the existing item fit test proceduresQ1 and S-X2.

[RETRACTED ARTICLE] irtplay: An R Package for Unidimensional Item Response Theory Modeling

Item response theory (IRT) is a general framework in which mathematical models are formulated to explain the relationship between an examinee's observable response on an item and the latent ability measured by a test. The application of IRT models and related statistical methods are commonly found in educational and psychological research. An important step in applying IRT models to test data is estimating the IRT model parameters. Accordingly, the successful application of IRT rests on the satisfactory statistical techniques and software for accurately estimating the model parameters. The irtplay R package was developed to provide users with a user-friendly experience and convenience when analyzing test data using unidimensional IRT models. The package can be used to fit the IRT models to a mixture of dichotomous and polytomous item data using marginal maximum likelihood estimation via the expectation-maximization, calibrate pretest items, and estimate examinees' latent ability parameters. In addition, the package provides practical tools that conveniently enable users to conduct many analyses related to IRT such as evaluating IRT model-data fit, analyzing differential item functioning, computing asymptotic variance-covariance matrices of item parameter estimates, calculating the conditional probability distribution of observed scores using the Lord and Wingersky (1984) formula, and importing item and ability parameter estimates from the output of popular IRT software. The main features of the irtplay package are illustrated using three data examples.

Determining the Difficulty and Discrimination Parameters of a Mathematics Performance-Based Assessment

Performance-based assessment (PBA) is different from traditional testing methods in that PBA presents real-life problems for students to solve by integrating critical thinking with their content knowledge and skills. Implementing PBA regularly in mathematics classes is associated with improved student achievement and motivation to learn; however, there are concerns about the general lack of psychometric data to support the use of performance assessments. To address such concerns, this study applied item response theory to estimate the difficulty and discrimination indices of items that comprised a newly developed mathematics PBA. Data were collected by administering the PBA to 750 senior high school students in the Western Region of Ghana. The results indicated that the difficulty and discrimination levels of each item were satisfactory, which suggests that well-designed and properly vetted math PBA items would improve classroom assessments as well as high-stakes tests administered on a large scale. Additional recommendations are included at the end of this paper.

Development and validation of the Brief Parental Burnout Scale (BPBS).

Since parental burnout can have detrimental consequences for parents, couples, and children, easy-to-use, reliable, and valid practical tools for the early detection of parental burnout are required. We developed and validated a brief scale to identify burned-out parents and parents at burnout risk. In Study 1, we developed the 5-item Brief Parental Burnout scale (BPBs) based on the gold standard measure of parental burnout, the Parental Burnout Assessment (PBA). We applied Item Response Theory Graded Response Model (GRM) analyses on the data of 1,725 Finnish parents. Compared against the PBA total score, the five selected items showed both high sensitivity and specificity in screening parental burnout. As expected, burned-out parents and those at burnout risk according to the BPBs displayed higher depressive symptoms, lower self-esteem, and more frequent sleep disruptions than non-burned-out parents. In Study 2, we tested the sensitivity and specificity of the BPBs by using both the BPBs and the PBA as independent assessment tools in a sample of 1,088 Finnish parents. In Study 3, the high sensitivity and specificity of the BPBs found in Study 2 were replicated in an independent sample of 104 Belgian parents. We further demonstrated the concurrent validity of the BPBs test score interpretations by showing that burned-out parents, or those at burnout risk, reported higher child neglect and parental violence than non-burned-out parents. Because of its user-friendliness and the strong psychometric properties of its test score, the BPBs is a good candidate to use as a screening tool for parental burnout. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Quantifying the Pharmacodynamics of Morphine in the Treatment of Postoperative Pain in Preverbal Children.

While the pharmacokinetics of morphine in children have been studied extensively, little is known about the pharmacodynamics of morphine in this population. Here, we quantified the concentration‐effect relationship of morphine for postoperative pain in preverbal children between 0 and 3 years of age. For this, we applied item response theory modeling in the pharmacokinetic/pharmacodynamic analysis of COMFORT‐Behavior (COMFORT‐B) scale data from 2 previous clinical studies. In the model, we identified a sigmoid maximal efficacy model for the effect of morphine and found that in 26% of children, increasing morphine concentrations were not associated with lower pain scores (nonresponders to morphine up‐titration). In responders to morphine up‐titration, the COMFORT‐B score slowly decreases with increasing morphine concentrations at morphine concentrations >20 ng/mL. In nonresponding children, no decrease in COMFORT‐B score is expected. In general, lower baseline COMFORT‐B scores (2.1 points on average) in younger children (postnatal age <10.3 days) were found. Based on the model, we conclude that the percentage of children at a desirable COMFORT‐B score is maximized at a morphine concentration between 5 and 30 ng/mL for children aged <10 days, and between 5 and 40 ng/mL for children >10 days. These findings support a dosing regimen previously suggested by Krekels et al, which would put >95% of patients within this morphine target concentration range at steady state. Our modeling approach provides a promising platform for pharmacodynamic research of analgesics and sedatives in children.

An Item Response Theory Model of Inter-Regional Collaboration for Transportation Planning in the United States

Regional governments and regional intergovernmental organizations play an increasingly important role in land use and transportation planning in many countries. In the U.S., regional organizations such as metropolitan planning organizations provide regional forums and institutions to coordinate actions of local government necessary to overcome collective action problems that result from the fragmentation of local authority. Their regional scope allows them to directly address collaboration problems or broker collaborative arrangements among local governments within their boundaries. Nevertheless, the scale of regional problems often extends beyond the boundaries of these regional entities. Thus, collaboration across regional governance organizations is necessary to address problems that have multi-regional impacts, such as large transportation projects. Extant research generally measures regional collaboration based on counts of collaboration actions undertaken, but this does not account for the fact that some are symbolic, while others require resources and commitment. Drawing insights from the institutional collective action framework, we advance an explanation for how regional organizations overcome collaboration risks to participate in collaborative solutions to regional and multi-regional problems. The analysis employs a unique national survey of metropolitan planning organizations (MPOs) and adds a novel application of item response theory (IRT) to capture differences in risk or difficulty among collaborative actions. The IRT results offer support for our ICA-based explanation of collaboration commitments. The implications of the findings for theory development and empirical study of RIGOs are discussed in conclusion.

Can the application of item response theory improve the quality of survey items for assessing human fingerprints on climate change?

The objective of the study was to use the item response theory to improve the quality of survey items used in research on the human fingerprint of climate change in the Beitbridge district, Zimbabwe. Data were collected using questionnaires to 365 randomly selected participants and analysed using the 1; 2- and 3-parameter models. The 1-parameter model showed that concerning item difficulty level, all the survey items on knowledge of human fingerprint on climate change are in the “good” category. Meanwhile, results of the 2-parameter model show that in terms of item discrimination, all the tests were fairly “good”, with a relatively flat slope. The 3-parameter model shows that all the survey items were in the “good” class with a guessing factor of 0% as well as improved discrimination values on most of the items. Therefore, item response theory is an appropriate approach to improve survey-based research methodology and outputs.