Item Response Theory Scores Research Articles

Is item response theory superior to classical test theory in scoring A‐IADL?

AbstractBackgroundThe Amsterdam IADL Questionnaire (A‐IADL‐Q) is increasingly being used in Alzheimer disease (AD) trials to assess activities of daily living. The authors of this scale (Sikkes et al., 2012) originally proposed an Item Response Theory (IRT) based scoring. While such an approach has some potential advantages, it requires a complex model, assumptions about population score distributions, and can be difficult to interpret clinically. An alternative to IRT scoring is the Classical Test Theory (CTT) method, which is relatively simple and easier to interpret. The present study compared IRT and CTT for scoring A‐IADL as a function of global cognition in healthy older adults and patients with mild‐to‐moderate AD.MethodAggregated data from three multinational clinical trials including subjects at the preclinical, early symptomatic, and moderate dementia phase of AD were analyzed. A‐IADL and CDR assessments at initial visits were evaluated and participants were classified into three groups based on CDR Global Scores: 0, 0.5, and 1. For IRT scoring, Graded Response Model (GRM) was used to estimate individual’s latent score. For CTT method, scaled average of scored responses were calculated. ROC analysis was conducted to evaluate the utility of each method in distinguishing between CDR groups.ResultThere were a total of 2,694 A‐IADL assessments across the three CDR groups. There was a very high correlation between CTT and IRT estimates of A‐IADL total score (r = 0.996, p < 0.05). As CDR Global Score increased, A‐IADL scores declined for both methods. In IRT scoring, the Item Characteristics Curve showed an overlap for most item responses, and the test information curve was narrow with a peak to the right of theta 0. Importantly, the ROC curve showed slightly better performance for CTT [AUC: 0.829 (0.812‐0.847)] compared to IRT [AUC: 0.809 (0.788‐0.830)].ConclusionThe current study found that CTT scoring of A‐IADL performed slightly better in distinguishing AD populations as a function of global cognition/function. Given the procedural complexities and required assumptions for IRT scoring, the CTT method, which has the advantages of being more straightforward and familiar to clinicians, represents a better choice for most applications of this instrument.

Capturing functional change in early Alzheimer’s disease: Comparing instruments and scoring techniques to detect subtle decline

AbstractBackgroundFunctional decline is a gradual process occurring along the Alzheimer’s disease (AD) continuum, starting years before dementia onset. Different instruments, such as the ADCS‐ADL and the Amsterdam Instrumental Activities of Daily Living Questionnaire (A‐IADL‐Q) aim to capture early functional changes. In the current study, we compared measurement instruments and scoring techniques to find the optimal measurement technique for detecting functional impairment in early AD.MethodStudy partners of participants from the international ‘Capturing Changes in Cognition’ study cohort completed both the A‐IADL‐Q and ADCS‐ADL over the course of one year (mean follow‐up 10.8±2.3 months). For both measures, we computed the following scores: (1) sum score, (2) weighted average, and (3) number of items with maximum score. For the A‐IADL‐Q, we also computed item response theory (IRT) scores, using predetermined parameters that take into account varying difficulty of included items (see Jutten et al., 2017). To compare the ability to detect changes over time for each scoring method, we performed linear mixed modeling, adjusted for sex and baseline age.ResultWe included 159 individuals (72.2±8.4 years old, 45% female) with subjective cognitive decline (SCD, N=12), mild cognitive impairment (MCI, N=66) or mild dementia (N=81). Overall, a significant decline in functioning was detected with all scoring techniques (all p<.01), except the technique of the number of items with most impairment for the ADCS‐ADL (see Table 1). When comparing decline in the diagnostic groups (see Figure 1 and Table 1), linear mixed models showed functional decline in all scoring techniques of both instruments in mild dementia. In MCI, all A‐IADL‐Q scoring methods showed a significant decline over time (p<.05), whereas the ADCS‐ADL scores did not. No decline was detected in SCD.ConclusionThe A‐IADL‐Q showed greater sensitivity to functional decline over one year than the ADCS‐ADL in those with MCI. Our findings suggest that the ability to detect subtle functional changes appears to be mostly dependent on instrument, and to a lesser extent on the scoring technique used. These findings imply that the selection of appropriate instruments is pivotal for optimal longitudinal study design, and clinical trial design in patients with MCI.

Estimating power for clinical trials with Patient Reported Outcomes - using Item Response Theory

ObjectivesPatient reported outcomes (PRO) are widely used in quality of life (QOL) studies, health outcomes research, and clinical trials. The importance of PRO has been advocated by health authorities. Patient Reported Outcomes Measurement Information System (PROMIS) is a collection of standardized measures of PROs using Item Response Theory (IRT). However, in clinical trials with PROs as endpoints, observed scores are routinely used for power estimation rather than IRT scores. This paper aims to fill this gap and estimate power in a two-arm clinical trials with PROMIS measures as endpoints with IRT model. Study Design and SettingWe conducted a series of simulations to study the IRT power with validated PROMIS measures controlling factors including sample size, effect size, number of items, and missing data proportion. ResultsOur results showed that sample size, effect size, and number of items are important indicators of IRT based power estimation for PROMIS measures. When effect size is small and sample size is limited, IRT model provides higher power than the closed form formula. ConclusionIRT based simulation should be used for power estimation in two-armed clinical, especially when there is small effect size or small sample size.

Psychometric and machine learning approaches for diagnostic assessment and tests of individual classification.

Assessments are commonly used to make a decision about an individual, such as grade placement, treatment assignment, job selection, or to inform a diagnosis. A psychometric approach to classify respondents based on the assessment would aggregate items into a score, and then each respondent's score is compared to a cut score. In contrast, a machine learning approach to classify respondents would build a model to predict the probability of belonging to a specific class from assessment items, and then respondents are classified based on their predicted probability of belonging to that class. It remains unclear whether psychometric and machine learning methods have comparable classification accuracy or if 1 method is preferable in all or some situations. In the context of diagnostic assessment, this study used Monte Carlo simulation methods to compare the classification accuracy of psychometric and machine learning methods as a function of the diagnosis-test correlation, prevalence, sample size, and the structure of the diagnostic assessment. Results suggest that machine learning models using logistic regression or random forest could have comparable classification accuracy to the psychometric methods using estimated item response theory scores. Therefore, machine learning models could provide a viable alternative for classification when psychometric methods are not feasible. Methods are illustrated with an empirical example predicting an oppositional defiant disorder diagnosis from a behavior disorders scale in children of age seven. Strengths and limitations for each of the methods are examined, and the overlap between the field of machine learning and psychometrics is discussed. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Allostatic load scoring using item response theory

Allostatic load is commonly operationalized using a sum-score of high-risk biomarkers. However, this method implies that biomarkers contribute equally to allostatic load, as each is given equal weight. Our goal in this methodological paper is to evaluate this, and complementarily, to identify biomarkers that are most informative and least informative for developing an allostatic load index. Item response theory models provide an alternate approach to calculating the allostatic load score, by treating individual biomarkers (e.g. “items”) as indicators of a latent allostatic load construct. Item response theory scores account for the data-driven discriminating power of each biomarker, and an individual’s pattern of biomarker responses. To demonstrate feasibility of this approach, we used data from the 2015–2016 National Health Examination and Nutrition Survey (NHANES; N ​= ​3751), with twelve allostatic load biomarkers representing immune response, metabolic function and cardiovascular health. Item response theory models revealed that body-mass-index and C-reactive protein were the most informative biomarkers for allostatic load. Both higher allostatic load sum-score and allostatic load item response theory score were associated with lower socio-economic status (p ​= ​0.008; p<0.001, respectively). Further, both formulations of allostatic load were positively associated with a nine-item depression screener (p<0.001 for both), but only the item response theory score was also positively associated with the impact of depressive symptoms on daily life (p ​= ​0.045). Item response theory scores may be more finely tuned to tease out effects, compared to sum-scores, and also provide more flexibility when there are missing biomarker measurements. Supplemental R code for our approach are included.

Current status on microsatellite instability, prognosis and adjuvant therapy in colon cancer: A nationwide survey of medical oncologists, colorectal surgeons and gastrointestinal pathologists.

BackgroundThere is significant variation in attitude both towards the role of microsatellite instability (MSI) in predicting prognosis, and towards its role in guiding which Stage II colon cancer patients may benefit from adjuvant chemotherapy.AimTo examine the current status of specialist attitudes towards MSI in guiding prognosis and adjuvant therapy in stage II colon cancer.MethodsThe Pathology in Colon Cancer, Prognosis and Uptake of Adjuvant Therapy (PiCC UP) Australia and New Zealand questionnaire was distributed to colorectal surgeons, medical oncologists and pathologists after institutional board approval. A 5‐scale Likert score was used to assess attitudes towards 23 pathological features for prognosis and 18 features for adjuvant therapy. Data were analysed using a rating scale and graded response model in item response theory (IRT) on STATA (Stata MP, version 15; StataCorp LP).Results164 specialists (45 oncologists, 86 surgeons and 33 pathologists) participated. 80.5% regularly attended colorectal multidisciplinary team (MDT) meetings. 89.63% and 59.26% of specialists reported that MSI status was likely or definitely to influence prognosis in colon cancer and recommendations for adjuvant therapy in Stage II colon cancer respectively.IRT modelling was achieved in 17 pathological features for prognosis. MSI IRT score was 4.47 (95% CI: 4.05‐4.68). IRT modelling was achieved in 10 pathological features for adjuvant therapy. MSI IRT score was 3.62 (2.89‐4.15). MSI ranked 10 (of 17) in order of importance in determining prognosis and ranked three (of 10) in guiding adjuvant therapy.ConclusionMSI status is considered an important biomarker when selecting patients for adjuvant therapy in Stage II colon cancer. MSI is also considered useful in prognostication of colon cancer. MSI status was ranked similar to the tumour grade of differentiation and the presence of perineural invasion.

Cross-cultural adaptation and validation of the Amsterdam Instrumental Activities of Daily Living questionnaire short version German for Switzerland

BackgroundInstrumental Activities of Daily Living (IADL) limitations are associated with reduced health-related quality of life for people with mild cognitive impairment (MCI). For these people, the assessment of IADL is crucial to the diagnostic process, as well as for the evaluation of new interventions addressing MCI. The Amsterdam IADL Questionnaire Short Version (A-IADL-Q-SV) is an established assessment tool with good psychometric properties that has been shown to be robust to cultural differences in Western countries. The aims of this study were to: (1) cross-culturally adapt and validate the A-IADL-Q-SV for the German-speaking population of Switzerland; (2) investigate its cultural comparability; and (3) evaluate further psychometric properties.MethodsThe A-IADL-Q-SV German was pretested on clinicians and participants in a memory clinic setting. The psychometric properties and cultural comparability of the questionnaire were investigated in memory clinic settings including participants with MCI or mild dementia, as well as participants with normal cognition recruited from the community. Item response theory (IRT) was applied to investigate measurement invariance by means of differential item functioning to assess item bias. Additionally, the test–retest reliability on scale level, the construct validity through hypothesis testing and the discriminant validity of the A-IADL-Q-SV German were evaluated.ResultsNinety-six informants of participants with normal cognition, MCI or mild dementia completed the A-IADL-Q-SV German. The basic assumptions for IRT scoring were met. No meaningful differential item functioning for culture was detected between the Swiss and Dutch reference samples. High test–retest reliability on scale level (ICC 0.93; 95% CI 0.9–0.96) was found. More than 75% of the observed correlations between the A-IADL-Q-SV German and clinical measures of cognition and functional status were found to be in the direction and of the magnitude hypothesized. The A-IADL-Q-SV German was shown to be able to discriminate between participants with normal cognition and MCI, as well as MCI and mild dementia.ConclusionsThe A-IADL-Q-SV German is a psychometrically robust measurement tool for a Swiss population with normal cognition, MCI and mild dementia. Thus, it provides a valuable tool to assess IADL functioning in clinical practices and research settings in Switzerland.Trial registration This study was registered retrospectively in July 2019 on ClinicalTrials.gov (NCT04012398).

Testing Latent Variable Distribution Fit in IRT Using Posterior Residuals

This research proposes a new statistic for testing latent variable distribution fit for unidimensional item response theory (IRT) models. If the typical assumption of normality is violated, then item parameter estimates will be biased, and dependent quantities such as IRT score estimates will be adversely affected. The proposed statistic compares the specified latent variable distribution to the sample average of latent variable posterior distributions commonly used in IRT scoring. Formally, the statistic is an instantiation of a generalized residual and is thus asymptotically distributed as standard normal. Also, the statistic naturally complements residual-based item-fit statistics, as both are conditional on the latent trait, and can be presented with graphical plots. In addition, a corresponding unconditional statistic, which controls for multiple comparisons, is proposed. The statistics are evaluated using a simulation study, and empirical analyses are provided.

Two parallel short forms to measure disease- and treatment-associated knowledge in rheumatoid arthritis: application of item response theory.

ObjectiveThe aim was to develop two disease- and treatment-related knowledge about RA (DataK-RA) short forms using item response theory-based linear optimal test design.MethodsWe used the open source Excel add-in solver to program a linear optimization algorithm to develop two short forms from the DataK-RA item bank. The algorithm was instructed to optimize precision (i.e. reliability) of the scores for both short forms, subject to a number of constraints that served to ensure that each short form would include unique items and that the short forms would have similar psychometric properties. Agreement among item response theory scores obtained from the different short forms was assessed using the Bland–Altman method and Student’s paired t-test. Construct validity and relative efficiency of the short forms was evaluated by relating the score to age, sex and educational attainment.ResultsTwo short forms were derived from the DataK-RA item bank that satisfied all content constraints. Both short forms included 15 unique items and yielded reliable scores (r > 0.70), with low ceiling and floor effects. The short forms yielded statistically indistinguishable mean scores according to Student’s paired t-test and Bland–Altman analysis. Scores on short forms 1 and 2 were associated with age, sex and educational attainment to a similar extent.ConclusionIn this study, we developed two DataK-RA short forms with unique items, yet similar psychometric properties, that can be used to assess patients pre‐ and post‐test interventions aimed at improving disease-related knowledge in RA patients.

Fearless Dominance/Boldness Is Not Strongly Related to Externalizing Behaviors: An Item Response-Based Analysis.

There is substantial and ongoing debate regarding the centrality of Fearless Dominance/Boldness (FD/B) to psychopathic personality due, in part, to its generally weak relations with externalizing behaviors. In response to these findings, proponents of FD/B have offered two hypotheses. First, FD/B may have nonlinear associations with externalizing outcomes such that FD/B may lead to resilience at moderate levels, but an overabundance of FD/B will yield maladaptive behavioral outcomes. Second, FD/B may be related to antisocial outcomes when paired with high scores on other psychopathic traits such as self-centered impulsivity, meanness, or disinhibition. The current study tests these two possibilities using two large samples (Study 1: 787 undergraduates; Study 2: 596 Amazon's Mechanical Turk participants). An item response theory scoring approach particularly sensitive to curvilinearity was used to maximize our ability to find a true curvilinear effect, if present. No evidence in favor of the curvilinearity hypothesis was found. Only a single significant interaction predicting substance use was observed between boldness and meanness. These findings contribute to a growing literature raising concerns regarding the relevance of FD/B to psychopathy.

An Application of Item Response Theory to Scoring Patient Safety Culture Survey Data.

Patient safety culture is important in preventing medical errors. Thus, many instruments have been developed to measure it. Yet, few studies focus on the data processing step. This study, by analyzing the Chinese version of the Safety Attitudes Questionnaire dataset that contained 37,163 questionnaires collected in Taiwan, found critical issues related to the currently used mean scoring method: The instrument, like other popular ones, uses a 5-point Likert scale, and because it is an ordinal scale, the mean scores cannot be calculated. Instead, Item Response Theory (IRT) was applied. The construct validity was satisfactory and the item properties of the instrument were estimated from confirmatory factor analysis. The IRT-based domain scores and mean domain scores of each respondent were estimated and compared. As for resolution, the mean approach yielded only around 20 unique values on a 0 to 100 scale for each domain; the IRT method yielded at least 440 unique values. Meanwhile, IRT scores ranged widely at each unique mean score, meaning that the precision of the mean approach was less reliable. The theoretical soundness and empirical strength of IRT suggest that healthcare institutions should adopt IRT as a new scoring method, which is the core step of processing collected data.

Understanding the Gap of Reading Performance between ELL and EOL Children from Low-Income Families in Elementary School Years

This study explored reading development in low income children of English Language learners (ELLs) from kindergarten to the fourth grade. Data used in this study came from the Early Childhood Longitudinal Study, Kindergarten Class of 2010-2011 (ECLS-K: 2011). A sample size was 3,451 students below the poverty threshold. The independent variables were the indicators of home language and gender. The six dependent variables were students’ reading item response theory (IRT) scale scores in the fall and spring semester of the kindergarten year and all the spring semesters from the first to the fourth grade. Six full 2-way analysis of variance (ANOVA) models were used for the statistical analyses. The results found there is a gender difference in children’s reading performance, with female doing slightly better than male students. The low-income children’s performance in reading IRT scores has shown differences among the three groups. The English Only Learners (EOL) had the highest mean scores throughout the five years. The group of Multilingual Learners (ML) and English Language Learner (ELL) group had mixed results of the second or lowest scores among these three groups. Among the six subgroups the EOL female had the highest mean scores throughout the five years.

PROMIS Global Health item nonresponse: is it better to impute missing item responses before computing T-scores?

Item response theory (IRT) scoring provides T-scores for physical and mental health subscales on the Patient-Reported Outcomes Measurement Information System Global Health questionnaire (PROMIS-GH) even when relevant items are skipped. We compared different item- and score-level imputation methods for estimating T-scores to the current scoring method. Missing PROMIS-GH items were simulated using a dataset of complete PROMIS-GH scales collected at a single tertiary care center. Four methods were used to estimate T-scores with missing item scores: (1) IRT-based scoring of available items (IRTavail), (2) item-level imputation using predictive mean matching (PMM), (3) item-level imputation using proportional odds logistic regression (POLR), and (4) T-score-level imputation (IMPdirect). Performance was assessed using root mean squared error (RMSE) and mean absolute error (MAE) of T-scores and comparing estimated regression coefficients from the four methods to the complete data model. Different proportions of missingness and sample sizes were examined. IRTavail had lowest RMSE and MAE for mental health T-scores while PMM had lowest RMSE and MAE for physical health T-scores. For both physical and mental health T-scores, regression coefficients estimated from imputation methods were closer to those of the complete data model. The available item scoring method produced more accurate PROMIS-GH mental but less accurate physical T-scores, compared to imputation methods. Using item-level imputation strategies may result in regression coefficient estimates closer to those of the complete data model when nonresponse rate is high. The choice of method may depend on the application, sample size, and amount of missingness.

Examining personality—Job characteristic interactions in explaining work outcomes

Although research suggests personality traits and job characteristics are each important drivers of work outcomes, there has been little focus on potential interactions between the two. In the current studies, we integrate the theory of purposeful work behavior with the job demands-resources model to examine how five-factor model personality traits interact with job resources and challenging job demands in explaining employee motivation and performance. We tested our hypotheses in two studies with different sample types and utilized item response theory scoring. Results across the studies were mixed, but generally support that there are important interactive effects between certain traits and contextual factors in explaining work behavior. Implications for theory and practice and future directions are discussed.

PCN244 UNBIASED TREATMENT EFFICACY DETECTION METHODS WHEN USING PATIENT REPORTED OUTCOMES IN SURVIVAL-DEPENDENT DROP-OUT

Missing patient reported outcome (PRO) data can arise from several processes. In the context of survival designs, the missingness may very well be outcome-dependent. This type of missing data is known as missing not at random (MNAR). In this circumstance, the PRO scores will produce biased treatment efficacy estimates. This bias can lead to the incorrect inference of no difference in treatment efficacy across arms. Outcome-dependent missingness poses serious challenges and requires the use of statistical approaches that properly adjust for this type of missing data. Currently, the state of the art in modeling PROs in a survival analysis context is to simultaneously model observed scores and the drop-out mechanism. This is referred to as a joint process model. However, interest in item response theory (IRT) approaches has grown in recent years. An IRT-based framework has been developed for evaluating PRO-based scores in survival analysis context. A longitudinal IRT model that incorporates the drop-out mechanism has been created. This allows the IRT scores and the drop-out mechanism to be modeled simultaneously. A simulation study based on empirical PRO data is employed here to illustrate the advantages of such an approach. The results of the simulation study show that the IRT approach to jointly model Quality of Life (QoL) scores and drop-out allows for the accurate estimation of separation of the treatment arms. Other approaches, such as a longitudinal IRT model that ignores drop-out, fail to detect separation. An IRT-based method has been developed that can better estimate the treatment effect in a survival analysis context. Notably, this approach can be implemented in standard commercial IRT software.

Routing Strategies and Optimizing Design for Multistage Testing in International Large‐Scale Assessments

AbstractThis study investigates the effect of several design and administration choices on item exposure and person/item parameter recovery under a multistage test (MST) design. In a simulation study, we examine whether number‐correct (NC) or item response theory (IRT) methods are differentially effective at routing students to the correct next stage(s) and whether routing choices (optimal versus suboptimal routing) have an impact on achievement precision. Additionally, we examine the impact of testlet length on both person and item recovery. Overall, our results suggest that no single approach works best across the studied conditions. With respect to the mean person parameter recovery, IRT scoring (via either Fisher information or preliminary EAP estimates) outperformed classical NC methods, although differences in bias and root mean squared error were generally small. Item exposure rates were found to be more evenly distributed when suboptimal routing methods were used, and item recovery (both difficulty and discrimination) was most precisely observed for items with moderate difficulties. Based on the results of the simulation study, we draw conclusions and discuss implications for practice in the context of international large‐scale assessments that recently introduced adaptive assessment in the form of MST. Future research directions are also discussed.

To bin or not to bin? A comparison of symptom frequency response formats in the assessment of health-related quality of life.

The goal of this study is to compare three different types of retrospective frequency response formats on the Healthy Days Symptoms Module (HDSM). Responses are compared in terms of intra-individual consistency, psychometric value, and participant feedback about each type of response format. Respondents each completed three versions of the HDSM, where items were framed to elicit an open-ended frequency, a fixed choice frequency, or a vague quantifier response. Traditional reliability statistics were used to evaluate intra-individual consistency. Differential item functioning (DIF) was used to test for response format effects, and item response theory (IRT) scale scores and standard errors were computed across the three forms to compare psychometric value. Linear mixed modeling was used to examine the associations of IRT scale scores across response formats with respondent characteristics. People are largely consistent in how they respond to items about their health, regardless of the response format, and no DIF was detected between response formats. The IRT scores computed from the "# of days" frequency response formats tend to have better measurement precision than those from vague quantifiers. Open-ended frequencies capture a greater span of individual differences for people reporting fewer symptoms; however, little measurement precision is lost in collapsing the frequencies into categories. Both the open-ended and fixed choice frequency response formats offer more measurement precision than vague quantifiers. While the open-ended frequency response format may capture more individual differences, respondents tend to report more difficulty with exact frequency recall, and thus, prefer the fixed choice frequency format.

Optimal Scores: An Alternative to Parametric Item Response Theory and Sum Scores.

The aim of this paper is to discuss nonparametric item response theory scores in terms of optimal scores as an alternative to parametric item response theory scores and sum scores. Optimal scores take advantage of the interaction between performance and item impact that is evident in most testing data. The theoretical arguments in favor of optimal scoring are supplemented with the results from simulation experiments, and the analysis of test data suggests that sum-scored tests would need to be longer than an optimally scored test in order to attain the same level of accuracy. Because optimal scoring is built on a nonparametric procedure, it also offers a flexible alternative for estimating item characteristic curves that can fit items that do not show good fit to item response theory models.

Addressing model uncertainty in item response theory person scores through model averaging

Item banks are often created in large-scale research and testing settings in the social sciences to predict individuals’ latent trait scores. A common procedure is to fit multiple candidate item response theory (IRT) models to a calibration sample and select a single best-fitting IRT model. The parameter estimates from this model are then used to obtain trait scores for subsequent respondents. However, this model selection procedure ignores model uncertainty stemming from the fact that the model ranking in the calibration phase is subject to sampling variability. Consequently, the standard errors of trait scores obtained from subsequent respondents do not reflect such uncertainty. Ignoring such sources of uncertainty contributes to the current replication crisis in the social sciences. In this article, we propose and demonstrate an alternative procedure to account for model uncertainty in this context—model averaging of IRT trait scores and their standard errors. We outline the general procedure step-by-step and provide software to aid researchers in implementation, both for large-scale research settings with item banks and for smaller research settings involving IRT scoring. We then demonstrate the procedure with a simulated item-banking illustration, comparing model selection and model averaging within sample in terms of predictive coverage. We conclude by discussing ways that model averaging and IRT scoring can be used and investigated in future research.

PRM92 - Evaluation of Modified Item Characteristics on Item Comprehension and Psychometric Performance

A mixed methods approach was used to determine whether it is possible to shorten and simplify items for electronic clinical outcome assessment (eCOA) administration without compromising patient comprehension and item psychometric performance. From a validated bank of 75 items measuring headache impact, a revised set of items (k=91) was developed by simplifying attributes (e.g., recall period, item stem content and length, response options scales content and length) and restructuring item display. To allow unconfounded assessment of revisions, each revised item only changed a single attribute. Cognitive debriefing interviews evaluated a subset of bank items (k=39 original, k=54 revised) in adult headache sufferers (N=9). In adult headache sufferers (N=2,379), the revised items (k=90) were calibrated with the original items using item response theory (IRT) models and item performance was evaluated. Debriefing results showed that items without a recall period in the item stem (e.g., “In the past 4 weeks…”) presented problems for most respondents, who assumed a much longer recall period than specified in the survey instructions. Specificity within item stems and answer choices made items harder to read for some participants, but were helpful for others. In quantitative analyses, the addition of recall period to an item stem resulted in more information over a wider IRT score range in 77% of tests. Minor changes in response option text (i.e., “Very often” versus “Often”) or even substantial reductions (i.e. from “None of the time” to “None”) had little or no effect on item information value. Rewriting “statement” items into questions improved performance in 81% of tests. In 84% of tests conducted, shortened items performed the same or better than the original version. Items may be shortened and simplified for electronic administration without reducing item performance. However, the recall period should still be included in the item stem.