Fit Of Item Response Theory Models Research Articles

Using ECHO program data to develop a brief measure of caregiver support and cognitive stimulation using the home observation for measurement of the environment-infant/toddler (HOME-IT).

Data from three NIH Environmental influences on Child Health Outcomes (ECHO) Program cohorts that collected the HOME-Infant-Toddler (HOME-IT age 0-3 years) version were used to examine the reliability of a brief scale of caregiver support and cognitive stimulation. Participants with HOME-IT data (N = 2518) were included in this analysis. Mean child age at HOME-IT assessment was 1.51 years, 48% of children were female, and 43% of children identified as Black. A four-stage analysis plan was used to evaluate item response theory assumptions, item response theory model fit, monotonicity, scalability, item fit, and differential item functioning. Results indicate the feasibility of developing a reliable 10-item scale (reliability >0.7) with particularly high precision for children with lower levels of cognitive stimulation.

Psychometric Assessment of an Item Bank for Adaptive Testing on Patient-Reported Experience of Care Environment for Severe Mental Illness: Validation Study.

The care environment significantly influences the experiences of patients with severe mental illness and the quality of their care. While a welcoming and stimulating environment enhances patient satisfaction and health outcomes, psychiatric facilities often prioritize staff workflow over patient needs. Addressing these challenges is crucial to improving patient experiences and outcomes in mental health care. This study is part of the Patient-Reported Experience Measure for Improving Quality of Care in Mental Health (PREMIUM) project and aims to establish an item bank (PREMIUM-CE) and to develop computerized adaptive tests (CATs) to measure the experience of the care environment of adult patients with schizophrenia, bipolar disorder, or major depressive disorder. We performed psychometric analyses including assessments of item response theory (IRT) model assumptions, IRT model fit, differential item functioning (DIF), item bank validity, and CAT simulations. In this multicenter cross-sectional study, 498 patients were recruited from outpatient and inpatient settings. The final PREMIUM-CE 13-item bank was sufficiently unidimensional (root mean square error of approximation=0.082, 95% CI 0.067-0.097; comparative fit index=0.974; Tucker-Lewis index=0.968) and showed an adequate fit to the IRT model (infit mean square statistic ranging between 0.7 and 1.0). DIF analysis revealed no item biases according to gender, health care settings, diagnosis, or mode of study participation. PREMIUM-CE scores correlated strongly with satisfaction measures (r=0.69-0.78; P<.001) and weakly with quality-of-life measures (r=0.11-0.21; P<.001). CAT simulations showed a strong correlation (r=0.98) between CAT scores and those of the full item bank, and around 79.5% (396/498) of the participants obtained a reliable score with the administration of an average of 7 items. The PREMIUM-CE item bank and its CAT version have shown excellent psychometric properties, making them reliable measures for evaluating the patient experience of the care environment among adults with severe mental illness in both outpatient and inpatient settings. These measures are a valuable addition to the existing landscape of patient experience assessment, capturing what truly matters to patients and enhancing the understanding of their care experiences. ClinicalTrials.gov NCT02491866; https://clinicaltrials.gov/study/NCT02491866.

Sociocognitive Processes and Item Response Models: A Didactic Example

AbstractThe usual interpretation of the person and task variables in between‐persons measurement models such as item response theory (IRT) is as attributes of persons and tasks, respectively. They can be viewed instead as ensemble descriptors of patterns of interactions among persons and situations that arise from sociocognitive complex adaptive system (CASs). This view offers insights for interpreting and using between‐persons measurement models and connecting with sociocognitive research. In this article, we use data generated from an agent‐based model to illustrate relations between “social” and “cognitive” features of a simple underlying CAS and the variables of an IRT model fit to resulting data. We note how the ideas connect to explanatory item response modeling and briefly comment on implications for score interpretations and uses in practice.

Psychometric evaluation of a national exam for clinical undergraduates.

As a high-stake national-level examination administrated before students' clerkship in China, the Standardized Competence Test for Clinical Medicine Undergraduates (SCTCMU) has received much attention from the relevant educational departments and society at large. Investigating SCTCMU's validity and reliability is critical to the national healthcare profession education. Raw responses from SCTCMU, answered by 44,332 examines of 4th-year undergraduate medical students on 300 multiple-choice items, were used to evaluate the quality of the exam via psychometric methods based on item response theory (IRT). The core assumptions and model-data fit of IRT models were evaluated, as well as the item properties and information functions. The IRT models were fitted to the observed assessment data, where all the required assumptions were met. The IRT analysis showed that most items had acceptable psychometric properties, and the passing score was located close to the lowest measurement error computed from the model outcomes. The proposed modern psychometric method provides a practical and informative approach to calibrating and analyzing medical education assessments. This work showcases a realistic depiction of the IRT analysis process and therefore facilitates the work of applied researchers wanting to conduct, interpret, and report IRT analyses on medical assessments.

Towards standardization of fatigue measurement: Psychometric properties and reference values of the PROMIS Fatigue item bank in the Dutch general population

Background There is little consensus on how to measure fatigue. Objectives To standardize the measurement of fatigue across populations, we aimed to assess the psychometric properties of the PROMIS Fatigue item bank in the Dutch general population and obtain reference values. Methods A sample of 1006 people participating in an internet panel completed the full v1.0 PROMIS Fatigue item bank (95 items). Structural validity (item response theory (IRT) assumptions and IRT model fit), measurement invariance/cross-cultural validity (absence of differential items functioning (DIF) for demographic variables and language, compared to data from US participants in PROMIS wave 1), and (internal) reliability (percentage of respondents with reliable estimates) were assessed. Results The IRT model assumptions were considered met (ECV 0.86, Omega-H 0.92), all items fitted the IRT model, no items showed DIF for demographic variables and seven for language, but with negligible impact on T-scores. Reliable fatigue T-scores were found for 98.3%, 69.8–82.6%, and 96.5% of the respondents with the full item bank, the standard short forms, and a simulated computerized adaptive test (CAT), respectively. The CAT administered on average only five items. A T-score of 49.1 represented the average score of the Dutch general population, T-scores &lt;55 are considered within normal limits, T-scores of 55–59 indicate mild fatigue, T-scores of 60–70 indicate moderate fatigue, and T-scores &gt;70 indicate severe fatigue. Conclusions The PROMIS Fatigue item bank showed sufficient structural validity, no measurement invariance for demographic characteristics, sufficient cross-cultural validity, and sufficient (internal) reliability in the Dutch general population.

Measurement Properties of the Lymphedema Life Impact Scale.

Background: The updated Lymphedema Life Impact Scale (LLIS, version 2) has been widely used to evaluate the effect of lymphedema from the patient's perspective. We sought to assess its ability to accurately and efficiently measure lymphedema-related impact using modern psychometric techniques. Methods and Results: We collected a total of 1054 patient-reported outcome measure scores from 285 patients with upper extremity lymphedema and 65 patients with lower extremity lymphedema between 2016 and 2020. We first evaluated the relationship between the LLIS score, L-Dex score, and limb volume difference (LVD), and used classical test and item response theories to assess its psychometric performance. The LLIS score was only very weakly associated with LVD (r = 0.17, p < 0.001) and L-Dex score (r = 0.22, p < 0.001). The LLIS had acceptable dimensionality. Items 7 (affects body image) and 16 (affects proper fit of clothing/shoes) were locally dependent (Yen's Q3 = 0.45). Eight of the 17 items was interpreted differently between upper and lower limb lymphedema patients (pseudo R2 ≥ 0.01). The scoring structure required correction for items 9 (affects intimate relations) and 12 (manages lymphedema). Removing items 18 (infection occurrence) and 7 resulted in substantially improved item response theory model fit (Tucker-Lewis index = 0.93, comparative fix index = 0.95, root mean square error of approximation = 0.07, and root means square of the residual = 0.06). The relationships between the LLIS and objective measures of lymphedema remained weak following modification (LVD; r = 0.13, p = 0.01, L-Dex; r = 0.26, p < 0.001). Conclusion: We were able to slightly improve the psychometric properties of the LLIS. However, these improvements did not rectify apparent issues with construct validity and both versions of the LLIS displayed a weak relationship with objective measures of lymphedema severity.

Establishing a common metric for self-reported anxiety in patients with prostate cancer: Linking the Memorial Anxiety Scale for Prostate Cancer with PROMIS Anxiety.

Anxiety is a common patient concern and clinical endpoint in prostate cancer outcomes research. It is measured using different self-report instruments that are not directly comparable, thereby making clinical trials, clinical performance measurement, and comparative effectiveness research challenging when anxiety is the outcome of interest. The objective of the current study was to enable a common reporting metric of anxiety so that scores on commonly used anxiety measures could be converted into Patient-Reported Outcomes Measurement Information System (PROMIS) scores for ease of application, interpretation, and comparability. Using an internet health panel, a total of 806 men with clinically localized prostate cancer completed items from the National Institutes of Health PROMIS Anxiety Short Form (version 7a) and the 18-item Memorial Anxiety Scale for Prostate Cancer (MAX-PC). A common metric was created using analyses based on item response theory, producing score crosswalk tables. The linking relationships were evaluated by resampling small subsets and estimating confidence intervals for the differences between the observed and linked PROMIS scores. Results of factor analysis and item response theory model fit supported the hypothesis that both scales measure essentially the same concept. Therefore, crosswalk tables appear to be justified and increasingly robust with increasing sample sizes. MAX-PC Anxiety results can be expressed on the PROMIS Anxiety metric for the purposes of clinical performance measurement, clinical trial outcomes, comparative effectiveness research, and other efforts to compare anxiety results across studies that use any one of these measures.

Psychometric properties of the Dutch-Flemish Patient-Reported Outcomes Measurement Information System Pain Behavior item bank in patients with musculoskeletal complaints

We studied the psychometric properties of the 39-item v1.1 Dutch-Flemish Patient-Reported Outcomes Measurement Information System Pain Behavior item bank in a sample of 1,602 patients with musculoskeletal complaints. We evaluated the assumptions of the underlying item response theory (IRT) model (unidimensionality and local dependency with confirmatory factor analyses), and monotonicity with scalability coefficients). We studied the IRT model fit of all items and estimated the item parameters of the IRT model. Differential item functioning (DIF) was studied for age and gender, and DIF for language was studied as a measure of cross-cultural validity. Confirmatory factor analyses showed suboptimal fit of a unidimensional model, but a bifactor model showed low risk of bias when a unidimensional model was assumed (Omega H = .92, explained common variance of .70). Fifteen item pairs (2%) were locally dependent. Five items showed poor scalability. All items fitted the IRT model; slope parameters ranged from .60 to 2.00, and threshold parameters from –2.05 to 6.80. One item showed DIF for age, 1 item DIF for gender, and 5 items showed DIF for language, but the impact on total scores was low. Our study shows limitations of the Dutch-Flemish Patient-Reported Outcomes Measurement Information System Pain Behavior item bank when used in a primary care population with musculoskeletal complaints. PerspectiveWe studied the psychometric properties of the Dutch-Flemish Patient-Reported Outcomes Measurement Information System Pain Behavior item bank in a large primary care population of patients with musculoskeletal complaints. It showed that the Pain Behavior item bank has limitations when used in this population.

Advancing and Evaluating IRT Model Data Fit Indices in Organizational Research

Item response theory (IRT) models have a number of advantages for developing and evaluating scales in organizational research. However, these advantages can be obtained only when the IRT model used to estimate the parameters fits the data well. Therefore, examining IRT model fit is important before drawing conclusions from the data. To test model fit, a wide range of indices are available in the IRT literature and have demonstrated utility in past research. Nevertheless, the performance of many of these indices for detecting misfit has not been directly compared in simulations. The current study evaluates a number of these indices to determine their utility for detecting various types of misfit in both dominance and ideal point IRT models. Results indicate that some indices are more effective than others but that none of the indices accurately detected misfit due to multidimensionality in the data. The implications of these results for future organizational research are discussed.

A novel method for expediting the development of patient-reported outcome measures and an evaluation across several populations.

Item response theory (IRT) models provide an appropriate alternative to the classical ordinal confirmatory factor analysis (CFA) during the development of patient-reported outcome measures (PROMs). Current literature has identified the assessment of IRT model fit as both challenging and underdeveloped (Sinharay & Johnson, 2003; Sinharay, Johnson, & Stern, 2006). This study evaluates the performance of Ordinal Bayesian Instrument Development (OBID), a Bayesian IRT model with a probit link function approach, through applications in two breast cancer-related instrument development studies. The primary focus is to investigate an appropriate method for comparing Bayesian IRT models in PROMs development. An exact Bayesian leave-one-out cross-validation (LOO-CV) approach (Vehtari & Lampinen, 2002) is implemented to assess prior selection for the item discrimination parameter in the IRT model and subject content experts' bias (in a statistical sense and not to be confused with psychometric bias as in differential item functioning) toward the estimation of item-to-domain correlations. Results support the utilization of content subject experts' information in establishing evidence for construct validity when sample size is small. However, the incorporation of subject experts' content information in the OBID approach can be sensitive to the level of expertise of the recruited experts. More stringent efforts need to be invested in the appropriate selection of subject experts to efficiently use the OBID approach and reduce potential bias during PROMs development.

An NCME Instructional Module on Item‐Fit Statistics for Item Response Theory Models

Drawing valid inferences from item response theory (IRT) models is contingent upon a good fit of the data to the model. Violations of model‐data fit have numerous consequences, limiting the usefulness and applicability of the model. This instructional module provides an overview of methods used for evaluating the fit of IRT models. Upon completing this module, the reader will have an understanding of traditional and Bayesian approaches for evaluating model‐data fit of IRT models, the relative advantages of each approach, and the software available to implement each method.

How Often Is the Misfit of Item Response Theory Models Practically Significant?

Standard 3.9 of the Standards for Educational and Psychological Testing () demands evidence of model fit when item response theory (IRT) models are employed to data from tests. Hambleton and Han () and Sinharay () recommended the assessment of practical significance of misfit of IRT models, but few examples of such assessment can be found in the literature concerning IRT model fit. In this article, practical significance of misfit of IRT models was assessed using data from several tests that employ IRT models to report scores. The IRT model did not fit any data set considered in this article. However, the extent of practical significance of misfit varied over the data sets.

A164: Development of Pediatric Item Banks to Measure Pain Behavior in the Patient Reported Outcomes Measurement Information System

Background/Purpose:The NIH Patient Reported Outcomes Measurement Information System (PROMIS) has created publicly available patient reported outcomes measures in several domains of physical, social and emotional health. Measurement of pediatric pain in PROMIS is currently limited to pain interference. Pain behaviors are observable actions orreactions that communicate pain including verbal, nonverbal and pain reducing behaviors. Numerous validated parent/provider‐rating scales of pain behavior in children exist but there are currently no validated self‐report measures of pain behavior in school‐age children and adolescents. Such measures could be useful in establishing targets for treatment and assessing outcomes. The aim of this study is to enhance PROMIS pediatric pain assessment by developing and testing pediatric pain behavior item banks for self‐ and proxy‐report.Methods:Candidate items were developed through a qualitative item review process, and were in the format, “In the past 7 days, when I was in pain …” Patients ages 8 to 17 years, or parents/guardians of children, with a chronic painful condition (fibromyalgia, juvenile idiopathic arthritis, sickle cell disease) were recruited through outpatient clinics at 3 centers. Child participants completed approximately 100 PROMIS items concerning their pain (including 47 candidate pain behavior items), physical function, fatigue, and psychosocial well/being. Proxies responded to sociodemographic and health history items and 51 new candidate proxy‐report pain behavior items were collected. A confirmatory factor analysis (CFA) was performed on the child and guardian pain behavior data, with model fit assessed by the comparative fit index (CFI) and root mean square error of approximation (RMSEA). Item response theory (IRT) analysis was performed on the pain behavior items based on the graded response model. Differential item functioning (DIF) was assessed by age group and disease group.Results:450 children (71% female, Mage 13.54), and 232 proxies participated. CFA indicated unidimensionality in the child (CFI = 0.962; RMSEA = 0.079) and proxy pain behavior responses (CFI = 0.970; RMSEA = 0.080). The responses for the child and proxy data had good IRT model fit and were free of local dependence. Slopes for the pediatric responses ranged from 1.81 (“rubbed body where hurt”) to 4.40 (“moved slower”), and thresholds ranged from −2.00 to 4.95. For the proxy data, slopes ranged from 1.51 (“think of something fun”) to 3.48 (“tried not to move”), and thresholds ranged from −3.02 to 2.43. Items performed well across disease groups and age. There was little DIF either by age group (8–12, 13–18) or by sample (child vs. proxy). Child and proxy scores were correlated at 0.70. Correlations between pain behavior and pain intensity were 0.60 in children and 0.65 in proxy sample.Conclusion:The PROMIS pediatric pain behavior item‐banks for self and proxy report are suitable for use in non‐adaptive format as short forms or in dynamic format as computerized adaptive tests in clinical research with the potential for adoption into clinical care.

Development and evaluation of the PI-G: a three-scale measure based on the German translation of the PROMIS® pain interference item bank

Study aim was to translate the PROMIS(®) pain interference (PI) item bank (41 items) into German, test its psychometric properties in patients with chronic low back pain and develop static subforms. We surveyed N = 262 patients undergoing rehabilitation who were asked to fill out questionnaires at the beginning and 2 weeks after the end of rehabilitation, applying the Oswestry Disability Index (ODI) and Pain Disability Index (PDI) in addition to the PROMIS(®) PI items. For psychometric testing, a 1-parameter item response theory (IRT) model was used. Exploratory and confirmatory factor analyses as well as reliability and construct validity analyses were conducted. The assumptions regarding IRT scaling of the translated PROMIS(®) PI item bank as a whole were not confirmed. However, we succeeded in devising three static subforms (PI-G scales: PI mental 13 items, PI functional 11 items, PI physical 4 items), revealing good psychometric properties. The PI-G scales in their static form can be recommended for use in German-speaking countries. Their strengths versus the ODI and PDI are that pain interference is assessed in a differentiated manner and that several psychometric values are somewhat better than those associated with the ODI and PDI (distribution properties, IRT model fit, reliability). To develop an IRT-scaled item bank of the German translations of the PROMIS(®) PI items, it would be useful to have additional studies (e.g., with larger sample sizes and using a 2-parameter IRT model).

The Use of the lz and lz* Person-Fit Statistics and Problems Derived From Model Misspecification

We extend a recent didactic by Magis, Raiche, and Beland on the use of the lz and lz* person-fit statistics. We discuss a number of possibly confusing details and show that it is important to first investigate item response theory model fit before assessing person fit. Furthermore, it is argued that appropriate distributions are only a first step for practical use of person-fit statistics.

The Nursing Student Self-Efficacy Scale

For nurse educators to optimize instruction that supports students' accurate estimates of their ability, a sound measure of nursing self-efficacy is needed. To provide evidence for an adequate item response theory model fit to scores measuring nursing students' self-efficacy to care for critically ill patients and to use the theory to provide evidence for reliable and valid interpretation of self-efficacy scores. The Nursing Student Self-Efficacy Scale was administered to 421 nursing students. Data regarding students' self-efficacy beliefs for psychomotor skill performance and communication skills were analyzed using the generalized partial credit item response theory model. All items showed acceptable model-data fit using the chi-square test with an alpha of .001. Review of item information functions showed that 40% of the items provided high information about self-efficacy and 20% provided moderately high information. Comparison of trait level distributions with reliability information given by the two subscales showed that the psychomotor subscale measured self-efficacy most accurately for second and third semester students and the communication subscale measured efficacy most accurately for first and second semester students. Significant mean differences in self-efficacy scores from both subscales were found between students in successive semesters of the program. Data analysis provided some evidence for reliable interpretation of scores indicating self-efficacy beliefs of nursing students. Comparison of scores from students in different semesters provided validity evidence; the instrument could be used to discriminate between beginning students and more advanced students possessing different levels of self-efficacy. When used for scale development, item response theory procedures can be more informative about item and test reliability than classical true score theory.

FIT OF ITEM RESPONSE THEORY MODELS: A SURVEY OF DATA FROM SEVERAL OPERATIONAL TESTS

ABSTRACTStandard 3.9 of the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, &amp; National Council for Measurement in Education, 1999) demands evidence of model fit when an item response theory (IRT) model is used to make inferences from a data set. We applied two recently suggested methods for assessing goodness of fit of IRT models—generalized residual analysis (Haberman, 2009) and residual analysis for assessing item fit (Bock &amp; Haberman, 2009)—to several operational data sets. We assessed the practical significance of misfit whenever possible. This report summarizes our findings. Though evidence of misfit of the IRT model was found for all the data sets, the misfit was not always practically significant.

Characterizing Sources of Uncertainty in Item Response Theory Scale Scores

Traditional estimators of item response theory (IRT) scale scores ignore uncertainty carried over from the item calibration process, which can lead to incorrect estimates of standard errors of measurement (SEM). Here, we review a variety of approaches that have been applied to this problem and compare them on the basis of their statistical methods and goals. We then elaborate on the particular flexibility and usefulness of a Multiple Imputation (MI) based approach, which can be easily applied to tests with mixed item types and multiple underlying dimensions. This proposed method obtains corrected estimates of individual scale scores, as well as their SEM. Furthermore, this approach enables a more complete characterization of the impact of parameter uncertainty by generating confidence envelopes (intervals) for item tracelines, test information functions, conditional SEM curves, and the marginal reliability coefficient. The MI based approach is illustrated through the analysis of an artificial data set, then applied to data from a large educational assessment. A simulation study was also conducted to examine the relative contribution of item parameter uncertainty to the variability in score estimates under various conditions. We found that the impact of item parameter uncertainty is generally quite small, though there are some conditions under which the uncertainty carried over from item calibration contributes substantially to variability in the scores. This may be the case when the calibration sample is small relative to the number of item parameters to be estimated, or when the IRT model fit to the data is multidimensional.

Screening for Elevated Levels of Fear-Avoidance Beliefs Regarding Work or Physical Activities in People Receiving Outpatient Therapy

Screening people for elevated levels of fear-avoidance beliefs is uncommon, but elevated levels of fear could worsen outcomes. Developing short screening tools might reduce the data collection burden and facilitate screening, which could prompt further testing or management strategy modifications to improve outcomes. The purpose of this study was to develop efficient yet accurate screening methods for identifying elevated levels of fear-avoidance beliefs regarding work or physical activities in people receiving outpatient rehabilitation. A secondary analysis of data collected prospectively from people with a variety of common neuromusculoskeletal diagnoses was conducted. Intake Fear-Avoidance Beliefs Questionnaire (FABQ) data were collected from 17,804 people who had common neuromusculoskeletal conditions and were receiving outpatient rehabilitation in 121 clinics in 26 states (in the United States). Item response theory (IRT) methods were used to analyze the FABQ data, with particular emphasis on differential item functioning among clinically logical groups of subjects, and to identify screening items. The accuracy of screening items for identifying subjects with elevated levels of fear was assessed with receiver operating characteristic analyses. Three items for fear of physical activities and 10 items for fear of work activities represented unidimensional scales with adequate IRT model fit. Differential item functioning was negligible for variables known to affect functional status outcomes: sex, age, symptom acuity, surgical history, pain intensity, condition severity, and impairment. Items that provided maximum information at the median for the FABQ scales were selected as screening items to dichotomize subjects by high versus low levels of fear. The accuracy of the screening items was supported for both scales. This study represents a retrospective analysis, which should be replicated using prospective designs. Future prospective studies should assess the reliability and validity of using one FABQ item to screen people for high levels of fear-avoidance beliefs. The lack of differential item functioning in the FABQ scales in the sample tested in this study suggested that FABQ screening could be useful in routine clinical practice and allowed the development of single-item screening for fear-avoidance beliefs that accurately identified subjects with elevated levels of fear. Because screening was accurate and efficient, single IRT-based FABQ screening items are recommended to facilitate improved evaluation and care of heterogeneous populations of people receiving outpatient rehabilitation.

Confirmatory factor analysis and item response theory: Two approaches for exploring measurement invariance.

This study investigated the utility of confirmatory factor analysis (CFA) and item response theory (IRT) models for testing the comparability of psychological measurements. Both procedures were used to investigate whether mood ratings collected in Minnesota and China were comparable. Several issues were addressed. The first issue was that of establishing a common measurement scale across groups, which involves full or partial measurement invariance of trait indicators. It is shown that using CFA or IRT models, test items that function differentially as trait indicators across groups need not interfere with comparing examinees on the same trait dimension. Second, the issue of model fit was addressed. It is proposed that person-fit statistics be used to judge the practical fit of IRT models. Finally, topics for future research are suggested.