A preliminary evaluation of the reliability and validity of a self‐reported communicative functioning item pool

Abstract

Background: Item banking, an approach to test development based in item response theory (IRT), is beginning to be applied to the measurement of communicative functioning in aphasia. This approach involves calibrating a set of test items responding to a particular latent trait to a common measurement scale. One method for validating such scales is to examine the degree to which obtained item calibration estimates agree with a priori item rankings based on theory or expert opinion. Aims: The purpose of this study was to evaluate the utility of magnitude estimation (ME) procedures for validating item banks containing self‐reported functional performance items, and to make a preliminary analysis of the validity of a proposed item pool for measuring self‐reported communicative functioning in aphasia. Methods & Procedures: A total of 14 raters made ME ratings of physical and communicative functioning items. These ratings were evaluated for their intra and inter‐observer reliability and, for subsets of the items, their correspondence with previously published IRT calibration estimates was also evaluated. Outcomes & Results: Intra‐rater reliability was moderate to high, and inter‐rater reliability was high. Correspondence with IRT calibrations was high for physical items, and moderate for communication items. The distribution of ME ratings for the communication items was negatively skewed. Conclusions: ME procedures have utility for investigating the validity of functional performance items. The results suggest that communicative functioning may have a more complex latent structure than physical functioning, and that the proposed item pool might benefit from the inclusion of additional items at the lower end of the scale.