Limitations of the statistics used to measure responsiveness in two longitudinal cohort studies of cognitive function and the utility of approaches derived from item response theory

Abstract

Listen

Translate article

Purpose: We are interested in measuring changes in cognitive function in longitudinal cohort studies. Specifically, we were interested in the responsiveness of the Mini-Mental Status Exam (MMSE), or its ability to detect changes over time. We were unable to find any previous published reports on the responsiveness of the MMSE. Responsiveness has been hailed as the principal criterion for determining the utility of an instrument for assessing change in the context of evaluative studies such as clinical trials or longitudinal cohort studies. Previous statistics that have been advocated to measure responsiveness have implicitly assumed a constant variance. The variance structure can be empirically determined and the assumption of constant variance can be tested. However, such determinations can only be carried out after the fact, after the choice of scale has already been made and the data are available to analyze. Methods: Item response theory (IRT) provided tools that proved useful in analyzing these data with respect to responsiveness. Specifically, IRT enabled the direct visualization of the variance structure across the entire spectrum of cognitive function as measured by the MMSE. Results: In our studies, we found that the assumption of constant variance did not hold for the MMSE, precluding our use of currently available responsiveness statistics. Conclusions: Currently available statistics for assessing responsiveness have significant limitations due to their assumption of constant variance. IRT allows the direct visualization of the expected variance structure across the entire spectrum of the trait measured by the instrument. In our case, this visualization led us to discard existing techniques to measure the responsiveness of the MMSE. IRT should be used as a first step in assessing the responsiveness of instruments in order to determine whether the assumption of constant variance is appropriate.