Item Response Theory Calibration Research Articles

Modifying the social responsiveness scale for adaptive administration.

The social responsiveness scale (SRS) is frequently used to quantify the autism-related phenotype and is gaining use in health outcomes research. However, it has a high respondent burden (65 items) for large-scale studies. Further, most evaluations of it have focused on the school-age form, not the preschool form. More validity evidence of shortened forms is necessary in the general population to support the broader health outcomes context of use. We evaluated the psychometrics of the SRS in 7030 individuals from multiple predominantly neurotypical samples in order to shorten it based on non-autistic sample metrics. Analyses included item factor analysis, differential item functioning (DIF), and multiple-group item response theory (IRT) to place the SRS items on a comparable scale, which was then simulated via computer adaptive testing (CAT) administration. The SRS was broadly unidimensional with few methodological residual dependencies. On average, males had more autistic characteristics than females, and preschoolers had fewer characteristics than school-age children. The final IRT calibration included 45 items equated across forms, and each form had 11 with significant wording discrepancies and 9 items with near-identical wording that exhibited form-related DIF. The CAT simulation suggested a median of 14 items was sufficient to reach a reliable score, demonstrating its feasibility across the range of impairments. IRT allows practitioners the ability to get highly reliable scores with fewer items than the full-length SRS. This supports the future application of the SRS in a computer adaptive testing mode in both neurotypical and ASD samples.

A New Instrument for Assessing Work-Related Body Mechanics and Strain in the General Population

Clinical pain is often linked to poor body mechanics, with individuals sometimes presenting multiple painful disorders. Such disorders may be influenced by behaviors that affect the general resiliency and health of the musculoskeletal system. We aimed to develop a self-reported scale using the Malmö Diet and Cancer Study questions on work-related body mechanical exposures. An expert panel identified 41 variables having content validity for musculoskeletal problems. Exploratory factor analysis was conducted on a random selection of 50% of the cohort (n = 6,789 adults); the remaining was reserved for confirmatory factor analyses (CFA), item response theory (IRT) item calibration, and differential item functioning investigations. Supported by standard measure development methods and fit criteria, the final unidimensional item bank contains 13 items. Overall CFA statistics (root mean square error of approximation = .09; comparative fit index = .96; Tucker-Lewis index = .96; standardized root mean residuals = .05) indicated excellent single-factor model fit and appropriateness of IRT modeling and calibration. Expert review and item information values (score-precision) guided selection of an 8-item short form with acceptable score-level reliabilities (≥.70) for T-scores = 39–80+. This measure provides reliable assessment of body mechanics strain in adults and can be useful when evaluating different contributions to musculoskeletal problems affecting pain-treatment success in future clinical research. PerspectiveThis article presents the development and psychometric properties of a new measure, “Work-related Body Mechanics and Strain Scale (WR-BMSS).” The scale has 13-items or alternatively an 8-item short form. This measure could potentially help clinicians who seek to assess how musculoskeletal problems may contribute to patient pain and disability.

A novel Scale to Assess Palliative Care Patients' Experience of Feeling Heard and Understood

ContextPatient experience of palliative care serves as an important indicator of quality and patient-centeredness. ObjectivesTo develop a novel patient-reported scale measuring ambulatory palliative care patients’ experience of feeling heard and understood by their providers. MethodsWe used self-reported patient experience data collected via mixed-mode survey administration. We conducted an exploratory factor analysis (EFA) and an expert panel ranking exercise to reduce the 10-item set based on underlying dimensionality. We then used item response theory (IRT) to calibrate remaining items based on psychometric properties and test information and precision. We considered item-level fit and examined the standardized local dependence chi-square statistics. We evaluated candidate items for differential item functioning by survey mode. We evaluated the test-retest reliability and validity of the final scale. ResultsThe EFA yielded a single factor (9/10 items had loadings > 0.80 on the single factor). We removed two items with the lowest factor loadings and ranked by the expert panel as being least reflective of the overall construct. IRT calibration of the remaining eight items showed high slopes (range 2.66 – 5.18); location parameters were all negative (range -0.90 - -0.36). We removed two more items based on local dependence indices and item-level fit. Combining psychometric information with the expert ratings we established the final 4-item scale, which was reliable (Cronbach's alpha = 0.84; polychoric correlation coefficient = 0.72) and had good convergent validity. ConclusionsThis novel multi-item Feeling Heard and Understood scale can be used to measure and improve ambulatory palliative care patient experience.

Accuracy of test battery harmonization: Simulation results using the HRS and MHAS studies

AbstractBackgroundStandardized measures of cognitive function are often used in research to understand predictors of heterogeneity in cognitive aging. Increasingly, researchers seek to pool data across studies. When assessment protocols contain overlapping items, item response theory and associated analytical techniques can be used to combine test items across studies into harmonized cognitive outcome(s). To better understand how characteristics of test item overlap impact a harmonized outcome, this study uses psychometric data from two cognitive aging studies to simulate and compare the accuracy of harmonization across six total items/overlapping items scenarios.MethodItem‐level cognitive data from 19 unique items across the Health and Retirement Study (HRS; 12 items) and the Mexican Health and Aging Study (MHAS; 10 items) were used (Table 1). Item response theory (IRT) calibration of a unidimensional model using data from the combined sample yielded item parameter and ability (theta) estimates that were then used to simulate six scenarios exploring item count and overlap combinations (19/19, 12/12, 10/10, 19/4, 19/3, and 19/1 for total/overlapping (akak “linking”) items, respectively). Briefly, simulation methods were: generate 500 simulated item‐level data sets with two groups (n = 500/group), each group having mean(sd) true ability set equal to HRS and MHAS estimates, respectively; apply IRT calibration to each simulated dataset, generating factor scores (using expected a posteriori (EAP) method); accuracy of harmonization was quantified by the difference between the IRT‐estimated factor scores and thetas for each simulated dataset using root mean square error (RMSE).ResultRMSE generally improved (decreased error) with increasing numbers of overlapping tests and linking items (Fig1). Differences between group means were poorly reproduced with <4 linking items (Fig2). The scenario representing the true overlap of three items between MHAS and HRS items (19/3) did not reproduce means(sd) well (suggesting that, without at least one additional linking item with desirable measurement properties (scenario 5), these tests may not be harmonized accurately.ConclusionFuture simulations should attempt to disentangle the effects of linking item quantity and characteristics (e.g. range of difficulty) from the effects of differences in group means and standard deviations on RMSE.

Development of the PROMIS Pediatric Physical Activity Item Banks.

The purpose of this study was to develop self-report and parent-proxy measures of children's physical activity for clinical research and practice and to demonstrate a valid and reliable instrument of children's lived experience of physical activity as reported by the children themselves or their parent proxies. This study involved qualitative development of item pools followed by 2 cross-sectional validity and reliability studies. The National Institutes of Health Patient Reported Outcome Measurement Information System (PROMIS) instrument development standards were applied to create child self-report and parent-proxy physical activity instruments from previously developed, content-valid pools of physical activity items. Each item used a 7-day recall period and had 5 response options. Item bank calibration was based on national samples totaling 3033 children aged 8 to 17years and 2336 parents of children aged 5 to 17years. Quantitative analyses included reliability assessments, factor analyses, item response theory calibration, differential item functioning, and construct validation. The final item banks comprised 10 items each. The items were selected based on content and psychometric properties. The item banks appeared to be unidimensional and free from differential item functioning. They showed excellent reliability and a high degree of precision across the range of the latent variable. Child-report and parent-proxy 4- and 8-item fixed-length instruments were specified. The instruments showed moderate correlation with existing self-report measures of physical activity. The PROMIS Pediatric Physical Activity instruments provide precise and valid measurement of children's lived experiences of physical activity. The availability of the PROMIS Pediatric Physical Activity instruments will support advances in clinical practice and research that require measurement of pediatric physical activity by self- and parent-proxy report.

Estimation of equable scale scores and treatment outcomes from patient- and clinician-reported PTSD measures using item response theory calibration.

Across multiple RCTs, discrepancies between patient and clinician reports of PTSD symptoms are at least a partial contributing factor to large discrepancies between treatment outcome effect sizes from self-report and clinician reports within the same patients. Using secondary data from the NIDA-funded Women and Trauma Study, we demonstrated Common Persons Item Response Theory (IRT) Calibration for calibrating self-reported and clinician-reported PTSD severity scores in a manner similar to the process used to produce equated scores across multiple forms of standardized tests (e.g., SAT, GRE). Under IRT calibration, treatment effect sizes between the CAPS and MPSS-SR did not differ, while with the noncalibrated measures, the CAPS effect size was 85% larger than the MPSS-SR. Further, across the range of a combined CAPS/MPSS-SR gold standard, IRT-calibrated CAPS and MPSS-SR individual scores did not differ; for uncalibrated individual scores, MPSS scores were higher than CAPS scores at higher levels of PTSD severity while the reverse was true at lower levels of severity. The use of IRT calibration approaches for calibrating self-report and clinical interview measures of PTSD will allow treatment researchers to reflect the treatment effect on PTSD as a construct (regardless of reporter) as opposed to being limited to reporting treatment effects that may be discrepant within patients and specific to the particular assessment measure being employed. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Supplemental Material for Estimation of Equable Scale Scores and Treatment Outcomes From Patient- and Clinician-Reported PTSD Measures Using Item Response Theory Calibration

Supplemental Material for Estimation of Equable Scale Scores and Treatment Outcomes From Patient- and Clinician-Reported PTSD Measures Using Item Response Theory Calibration

The EORTC CAT Core—The computer adaptive version of the EORTC QLQ-C30 questionnaire

BackgroundTo optimise measurement precision, relevance to patients and flexibility, patient-reported outcome measures (PROMs) should ideally be adapted to the individual patient/study while retaining direct comparability of scores across patients/studies. This is achievable using item banks and computerised adaptive tests (CATs). The European Organisation for Research and Treatment of Cancer (EORTC) Quality of Life Questionnaire Core 30 (QLQ-C30) is one of the most widely used PROMs in cancer research and clinical practice. Here we provide an overview of the research program to develop CAT versions of the QLQ-C30's 14 functional and symptom domains. MethodsThe EORTC Quality of Life Group's strategy for developing CAT item banks consists of: literature search to identify potential candidate items; formulation of new items compatible with the QLQ-C30 item style; expert evaluations and patient interviews; field-testing and psychometric analyses, including factor analysis, item response theory calibration and simulation of measurement properties. In addition, software for setting up, running and scoring CAT has been developed. ResultsAcross eight rounds of data collections, 9782 patients were recruited from 12 countries for the field-testing. The four phases of development resulted in a total of 260 unique items across the 14 domains. Each item bank consists of 7–34 items. Psychometric evaluations indicated higher measurement precision and increased statistical power of the CAT measures compared to the QLQ-C30 scales. Using CAT, sample size requirements may be reduced by approximately 20–35% on average without loss of power. ConclusionsThe EORTC CAT Core represents a more precise, powerful and flexible measurement system than the QLQ-C30. It is currently being validated in a large independent, international sample of cancer patients.

Grouping Effects on Jackknifed Variance Estimation for Item Response Theory Scaling and Equating With Cluster‐Based Assessment Data

AbstractEducational assessment data are often collected from a set of test centers across various geographic regions, and therefore the data samples contain clusters. Such cluster‐based data may result in clustering effects in variance estimation. However, in many grouped jackknife variance estimation applications, jackknife groups are often formed by a random grouping method that ignores the cluster structures of the data. In this study, we constructed both random and cluster‐based jackknife groups for data known to have cluster structures and compared the jackknifed standard errors, yielded by two different grouping methods, of item response theory (IRT) scaling coefficient estimates and equated scores. Three independent data samples from an international test of English were used for the study. The cluster‐based jackknife group results showed relatively larger jackknifed standard errors of scaling coefficient estimates and scale scores than the results of the random jackknife groups for all three data samples. For cluster‐based assessment data, the cluster‐based jackknife approach provides a more appropriate way to estimate the standard errors of the parameters of IRT calibration, scaling, and equating analyses.

The emotion dysregulation inventory: Psychometric properties and item response theory calibration in an autism spectrum disorder sample.

This paper describes a new measure of poor emotional control called the Emotion Dysregulation Inventory (EDI). Caregivers of 1,755 youth with ASD completed candidate items, and advanced statistical techniques were applied to identify the best final items. The EDI is unique because it captures common emotional problems in ASD and is appropriate for both nonverbal and verbal youth. It is an efficient and sensitive measure for use in clinical assessments, monitoring, and research with youth with ASD.

Development and psychometric evaluation of the PROMIS Pediatric Life Satisfaction item banks, child-report, and parent-proxy editions.

To describe the psychometric evaluation and item response theory calibration of the PROMIS Pediatric Life Satisfaction item banks, child-report, and parent-proxy editions. A pool of 55 life satisfaction items was administered to 1992 children 8-17years old and 964 parents of children 5-17years old. Analyses included descriptive statistics, reliability, factor analysis, differential item functioning, and assessment of construct validity. Thirteen items were deleted because of poor psychometric performance. An 8-item short form was administered to a national sample of 996 children 8-17years old, and 1294 parents of children 5-17years old. The combined sample (2988 children and 2258 parents) was used in item response theory (IRT) calibration analyses. The final item banks were unidimensional, the items were locally independent, and the items were free from impactful differential item functioning. The 8-item and 4-item short form scales showed excellent reliability, convergent validity, and discriminant validity. Life satisfaction decreased with declining socio-economic status, presence of a special health care need, and increasing age for girls, but not boys. After IRT calibration, we found that 4- and 8-item short forms had a high degree of precision (reliability) across a wide range (>4 SD units) of the latent variable. The PROMIS Pediatric Life Satisfaction item banks and their short forms provide efficient, precise, and valid assessments of life satisfaction in children and youth.

The Delta-Scoring Method of Tests With Binary Items: A Note on True Score Estimation and Equating.

This article presents some new developments in the methodology of an approach to scoring and equating of tests with binary items, referred to as delta scoring (D-scoring), which is under piloting with large-scale assessments at the National Center for Assessment in Saudi Arabia. This presentation builds on a previous work on delta scoring and adds procedures for scaling and equating, item response function, and estimation of true values and standard errors of D scores. Also, unlike the previous work on this topic, where D-scoring involves estimates of item and person parameters in the framework of item response theory, the approach presented here does not require item response theory calibration.

Practical Consequences of Item Response Theory Model Misfit in the Context of Test Equating with Mixed-Format Test Data.

In item response theory (IRT) models, assessing model-data fit is an essential step in IRT calibration. While no general agreement has ever been reached on the best methods or approaches to use for detecting misfit, perhaps the more important comment based upon the research findings is that rarely does the research evaluate IRT misfit by focusing on the practical consequences of misfit. The study investigated the practical consequences of IRT model misfit in examining the equating performance and the classification of examinees into performance categories in a simulation study that mimics a typical large-scale statewide assessment program with mixed-format test data. The simulation study was implemented by varying three factors, including choice of IRT model, amount of growth/change of examinees’ abilities between two adjacent administration years, and choice of IRT scaling methods. Findings indicated that the extent of significant consequences of model misfit varied over the choice of model and IRT scaling methods. In comparison with mean/sigma (MS) and Stocking and Lord characteristic curve (SL) methods, separate calibration with linking and fixed common item parameter (FCIP) procedure was more sensitive to model misfit and more robust against various amounts of ability shifts between two adjacent administrations regardless of model fit. SL was generally the least sensitive to model misfit in recovering equating conversion and MS was the least robust against ability shifts in recovering the equating conversion when a substantial degree of misfit was present. The key messages from the study are that practical ways are available to study model fit, and, model fit or misfit can have consequences that should be considered when choosing an IRT model. Not only does the study address the consequences of IRT model misfit, but also it is our hope to help researchers and practitioners find practical ways to study model fit and to investigate the validity of particular IRT models for achieving a specified purpose, to assure that the successful use of the IRT models are realized, and to improve the applications of IRT models with educational and psychological test data.

Patient-Reported Spectacle Independence Questionnaire (PRSIQ): Development and Validation

To develop a questionnaire quantifying spectacle independence following cataract surgery. Questionnaire design and validation study. A literature review, expert clinical interviews, and patient interviews were undertaken to develop a patient-reported questionnaire assessing spectacle independence. Draft items were assessed and revised in a series of cognitive interviews where the instructions, recall period, and response categories were evaluated for understandability and consistency. The final draft items were evaluated in 2 quantitative studies. Confirmatory factor analyses, including item response theory calibration, were performed; reliability estimates were obtained; and validity analyses of the resulting scores were carried out. Qualitative research demonstrated that patients often considered themselves spectacle independent yet, when probed, it was determined that the respondent actually wore correction for certain activities. Moreover, despite what the respondents claimed about needing to wear correction and their actual use of correction, there were various levels of function reported when carrying out activities at several different distances without the use of correction. Using the qualitative results, the Patient-Reported Spectacle Independence Questionnaire (PRSIQ) was developed to assess spectacle independence via items that assess what patients say and do, and how they function at various distances. All quantitative analyses (eg, discriminant and convergent validity correlations, known-groups analyses) conform to predictions and support the use of the PRSIQ as a measure of spectacle independence. The PRSIQ is a patient-reported measure assessing spectacle independence following cataract surgery. The analyses conducted provide evidence for the use of the PRSIQ total score as a measure of spectacleindependence.

Development and Evaluation of the PROMIS® Pediatric Positive Affect Item Bank, Child-Report and Parent-Proxy Editions.

The purpose of this study is to describe the psychometric evaluation and item response theory calibration of the PROMIS Pediatric Positive Affect item bank, child-report and parent-proxy editions. The initial item pool comprising 53 items, previously developed using qualitative methods, was administered to 1,874 children 8-17 years old and 909 parents of children 5-17 years old. Analyses included descriptive statistics, reliability, factor analysis, differential item functioning, and construct validity. A total of 14 items were deleted, because of poor psychometric performance, and an 8-item short form constructed from the remaining 39 items was administered to a national sample of 1,004 children 8-17 years old, and 1,306 parents of children 5-17 years old. The combined sample was used in item response theory (IRT) calibration analyses. The final item bank appeared unidimensional, the items appeared locally independent, and the items were free from differential item functioning. The scales showed excellent reliability and convergent and discriminant validity. Positive affect decreased with children's age and was lower for those with a special health care need. After IRT calibration, we found that 4 and 8 item short forms had a high degree of precision (reliability) across a wide range of the latent trait (>4 SD units). The PROMIS Pediatric Positive Affect item bank and its short forms provide an efficient, precise, and valid assessment of positive affect in children and youth.

Calibration of the Spanish PROMIS Smoking Item Banks.

The Patient-Reported Outcomes Measurement Information System (PROMIS) Smoking Initiative has developed item banks for assessing six smoking behaviors and biopsychosocial correlates of smoking among adult cigarette smokers. The goal of this study is to evaluate the performance of the Spanish version of the PROMIS smoking item banks as compared to the original banks developed in English. The six PROMIS banks for daily smokers were translated into Spanish and administered to a sample of Spanish-speaking adult daily smokers in the United States (N = 302). We first evaluated the unidimensionality of each bank using confirmatory factor analysis. We then conducted a two-group item response theory calibration, including an item response theory-based Differential Item Functioning (DIF) analysis by language of administration (Spanish vs. English). Finally, we generated full bank and short form scores for the translated banks and evaluated their psychometric performance. Unidimensionality of the Spanish smoking item banks was supported by confirmatory factor analysis results. Out of a total of 109 items that were evaluated for language DIF, seven items in three of the six banks were identified as having levels of DIF that exceeded an established criterion. The psychometric performance of the Spanish daily smoker banks is largely comparable to that of the English versions. The Spanish PROMIS smoking item banks are highly similar, but not entirely equivalent, to the original English versions. The parameters from these two-group calibrations can be used to generate comparable bank scores across the two language versions. In this study, we developed a Spanish version of the PROMIS smoking toolkit, which was originally designed and developed for English speakers. With the growing Spanish-speaking population, it is important to make the toolkit more accessible by translating the items and calibrating the Spanish version to be comparable with English-language scores. This study provided the translated item banks and short forms, comparable unbiased scores for Spanish speakers and evaluations of the psychometric properties of the new Spanish toolkit.

How Does Calibration Timing and Seasonality Affect Item Parameter Estimates?

Continuously administered examination programs, particularly credentialing programs that require graduation from educational programs, often experience seasonality where distributions of examine ability may differ over time. Such seasonality may affect the quality of important statistical processes, such as item response theory (IRT) item calibration and equating. The lead time required for producing pre-equated test forms in the continuous testing framework further complicates issues. This study examines the effect of seasonality in test data on Rasch IRT item parameter estimates. Data came from four credentialing examination programs that represented both programs with and without seasonality, as well as medium and low examinee volume. Results showed that calibrating items during certain times can lead to quite poor item parameter estimates. While certain programs could conduct IRT calibrations without waiting for the full examination cycle to be completed, other types of programs should wait as long as possible before calibrating items.

Linking Fatigue Measures on a Common Reporting Metric

ContextFatigue is one of the most common and debilitating symptoms experienced by patients living with chronic conditions and is also commonly experienced in the general U.S. population. Linking fatigue scores from some of the most widely used measure of fatigue to the same metric will facilitate interpretation of fatigue outcomes. ObjectivesThe goal of this study is to report the methods used to develop linking (crosswalk) tables to enable the direct comparison of Patient-Reported Outcomes Measurement Information System-Fatigue with fatigue scores on the Functional Assessment of Chronic Illness Therapy-Fatigue Scale, the Medical Outcomes Study Short Form-36 four-item Vitality Scale, and the Quality of Life in Neurological Disorders Fatigue Scale. MethodsParticipants were recruited from two data sets (n=1120 and n=803). Two item response theory-based linking methods, the Stocking-Lord calibration and fixed-parameter calibration, were used to establish linking between measures. The item response theory calibrations were derived using the graded response model. ResultsBoth the Stocking-Lord calibration and fixed-parameter calibration linking methods produced comparable results. Final crosswalk tables are reported for the fixed-parameter calibration. ConclusionFindings can facilitate comparison of scores across some of the most widely used fatigue measures and assist in comparing patient-reported fatigue outcomes in clinical trials, comparative effectiveness research, and clinical practice.

EXPLORING ALTERNATIVE TEST FORM LINKING DESIGNS WITH MODIFIED EQUATING SAMPLE SIZE AND ANCHOR TEST LENGTH

ABSTRACTThe purpose of this study was to evaluate the combined effects of reduced equating sample size and shortened anchor test length on item response theory (IRT)‐based linking and equating results. Data from two independent operational forms of a large‐scale testing program were used to establish the baseline results for evaluating the results from two alternative designs. Under the two alternative designs, two simulated conditions were created from the original data. Under one condition, we reduced the equating sample size (from about 2,000 to about 1,000) per anchor item and shortened the anchor test length (by half) per equating sample. Under the other condition, we reduced the sample size (from about 2,000 to about 1,000) per anchor item only. A complete grouped jackknife replication method was used to estimate the standard errors of the linking and equating procedures from 100 jackknife replicate samples; the complete procedures included IRT calibrations, item parameter scaling, and IRT true score equating. The findings from a comparison of the results from the two simulated conditions and the baseline results showed that neither alternative design had any practical impact on the linking and equating results for either test form.

Scale Refinement and Initial Evaluation of a Behavioral Health Function Measurement Tool for Work Disability Evaluation

ObjectiveTo use item response theory (IRT) data simulations to construct and perform initial psychometric testing of a newly developed instrument, the Social Security Administration Behavioral Health Function (SSA-BH) instrument, that aims to assess behavioral health functioning relevant to the context of work. DesignCross-sectional survey followed by IRT calibration data simulations. SettingCommunity. ParticipantsSample of individuals applying for Social Security Administration disability benefits: claimants (n=1015) and a normative comparative sample of U.S. adults (n=1000). InterventionsNone. Main Outcome MeasureSSA-BH measurement instrument. ResultsIRT analyses supported the unidimensionality of 4 SSA-BH scales: mood and emotions (35 items), self-efficacy (23 items), social interactions (6 items), and behavioral control (15 items). All SSA-BH scales demonstrated strong psychometric properties including reliability, accuracy, and breadth of coverage. High correlations of the simulated 5- or 10-item computer adaptive tests with the full item bank indicated robust ability of the computer adaptive testing approach to comprehensively characterize behavioral health function along 4 distinct dimensions. ConclusionsInitial testing and evaluation of the SSA-BH instrument demonstrated good accuracy, reliability, and content coverage along all 4 scales. Behavioral function profiles of Social Security Administration claimants were generated and compared with age- and sex-matched norms along 4 scales: mood and emotions, behavioral control, social interactions, and self-efficacy. Using the computer adaptive test-based approach offers the ability to collect standardized, comprehensive functional information about claimants in an efficient way, which may prove useful in the context of the Social Security Administration's work disability programs.