PRM92 - Evaluation of Modified Item Characteristics on Item Comprehension and Psychometric Performance

Abstract

A mixed methods approach was used to determine whether it is possible to shorten and simplify items for electronic clinical outcome assessment (eCOA) administration without compromising patient comprehension and item psychometric performance. From a validated bank of 75 items measuring headache impact, a revised set of items (k=91) was developed by simplifying attributes (e.g., recall period, item stem content and length, response options scales content and length) and restructuring item display. To allow unconfounded assessment of revisions, each revised item only changed a single attribute. Cognitive debriefing interviews evaluated a subset of bank items (k=39 original, k=54 revised) in adult headache sufferers (N=9). In adult headache sufferers (N=2,379), the revised items (k=90) were calibrated with the original items using item response theory (IRT) models and item performance was evaluated. Debriefing results showed that items without a recall period in the item stem (e.g., “In the past 4 weeks…”) presented problems for most respondents, who assumed a much longer recall period than specified in the survey instructions. Specificity within item stems and answer choices made items harder to read for some participants, but were helpful for others. In quantitative analyses, the addition of recall period to an item stem resulted in more information over a wider IRT score range in 77% of tests. Minor changes in response option text (i.e., “Very often” versus “Often”) or even substantial reductions (i.e. from “None of the time” to “None”) had little or no effect on item information value. Rewriting “statement” items into questions improved performance in 81% of tests. In 84% of tests conducted, shortened items performed the same or better than the original version. Items may be shortened and simplified for electronic administration without reducing item performance. However, the recall period should still be included in the item stem.