Scientific and clinical considerations in progress monitoring: When is a brief measure too brief?

Abstract

It is argued that the use of brief measures such as the CORE 10 and the ORS in patient monitoring and feedback systems is problematic because their poor reliability leads to insufficient sensitivity for early decision making. This problem is partially resolved by using the briefer measures as a form of communication between client and therapist rather than a normative index of distress.Keywords: psychotherapy, feedback, measures, reliability, sensitivityThis commentary is in reaction to the special issue of Canadian Psychology on progress tracking (2012, Vol. 53, No. 2). In the interests of clarity and brevity, we have decided to focus our commentary on the article by Duncan (2012).Dr. Duncan presents a useful review of the history and literature of progress monitoring and feedback systems. There is accumulating evidence for the usefulness of these systems, including the particular system Partners for Change Outcome Management System (PCOMS) developed by Duncan and colleagues. While agreeing with the general thrust of his article, we wanted to point out some of the difficulties presented by his choice of measures and offer a possible solution.The world, at large, is full of measures that are bad in at least the sense that they do not discriminate well between members of a population. This is for example of many satisfaction surveys that tend to demonstrate high levels of satisfaction for services of all levels of quality. In the area of Quality of Life measurement, the development of measures that cover a wide range of domains coupled with the need to develop brief measures has resulted in measures that are, in some cases, unresponsive in particular domains. In recognition of this, researchers in the Quality of Life field have developed the concepts of responsiveness, the tendency of a measure to change in response to a relevant intervention, and of sensitivity, the degree to which that change is statistically significant, to describe and evaluate quality of Ufe measures (Norman, Wyrwich, & Patrick, 2007).Clinicians and researchers in mental health and psychotherapy, on the other hand, have produced a number of measures with good psychometric properties that can discriminate quite well between normal and clinical populations and show reliable change in response to relevant interventions. These measures have been developed in a number of ways, driven by various theoretical models both of psychopathology and of measure development. They may have been developed to measure one entity (e.g., depression) and ended up having an identifiable factor structure or have a notional range of domains that are not confirmed empirically. When they are used in psychotherapy research, there is an overwhelming tendency for them to be used as single factor measures of distress.When, however, we move into the area of measuring change between sessions or over a subset of therapy, the sensitivity of a measure or its ability to reliably detect relatively small change becomes more (not less) important We know from the early studies of Lord and Novick (1968) that, other factors being equal, the reliability of a measure increases with its length (and decreases with its brevity). The logical conclusion from these considerations is that if we are concerned with using measures to indicate small changes and thus provide useful feedback to therapists, they should be at least as reliable as those we use to measure the overall impact of therapy and should be of sufficient length to provide that reliability.Establishing the sensitivity of a measure used to detect improvement or deterioration is problematic if only one measure is used because the rate of detection has to be compared with a notional true rate of detection. This problem can be solved by using two measures simultaneously so that relative detection rates (and thus sensitivity) can be calculated. At a recent conference (the 2012 Society for Psychotherapy Research International Meeting held in Virginia Beach) Halstead et al. …