Clinical and scientific considerations in progress monitoring: When is a measure too long?

Abstract

In their reaction to Duncan (2012), Halstead, Youn, and Armijo (2013) pose a useful psychometric question regarding how brief is too brief when considering progress measures. They suggest that measures should be of sufficient length to provide reliability and validity but provide no definition of what constitutes sufficient reliability or validity. Moreover, Halstead et al. overlook the important clinical issue of feasibility, whether the measure will be routinely used by front line clinicians. We assert that there is no doubt that the increased reliability and validity of longer measures likely result in better detection, prediction, and ultimate measurement of outcome, but suggest that empirical investigation is required to determine if these differences are clinically meaningful and offset the low compliance rates. We also assert that while empirical investigation is required to determine how brief is too brief, the answer to the question regarding when a measure is too long is simple: When clinicians won't use it.Keywords: PCOMS, reliability, validity, feasibility, ORSIn their thoughtful commentary in reaction to the special issue of Canadian Psychology on progress tracking (2012, Vol. 53, No. 2), and specifically to the article by Duncan (2012), Halstead, Youn, and Armijo (2013) posed a useful psychometric question regarding how brief is too brief when considering progress measures and raised important issues relevant to the divide between research and practice. They also presented a very interesting distinction concerning existing measures, namely the Normative and the Communicative models.Halstead et al. (2013) made several points: Longer measures of outcome are more reliable and valid than shorter measures; we should use reliable measures that are sensitive to change; two measures are better than one; every session measurement allows more complete information given the unpredictable nature of treatment endings; and perhaps, measures that allow both normative and communicative feedback offer the best choice for clinical practice.Halstead et al. (2013) concluded that measures should be of sufficient length to provide reliability but provide no definition of what constitutes sufficient reliability. They suggested that the Outcome Rating Scale (ORS) has low (averaging .85 across studies, but if the lowest [.79] is removed that used an oral version in a telephonic service, the average is .87), and performs quite poorly on concurrent validity, for example, with the Outcome Questionnaire 45.2 (OQ; Lambert et al., 1996), ranging from .53 to .74, but offer no comparison to a standard of relatively high alpha or performing well on concurrent validity. This lack of comparison is understandable given the absence of agreed on thresholds for reliability and validity. The interpretations offered by Halstead et al., however, are debatable. For example, Cicchetti (1994) proposed that reliability estimates between .80 and .90 are good for clinical significance. Of course, others have suggested more stringent criteria (Nunnally & Bernstein, 1994). When it comes to the reliability and validity of measures, their beauty is in the eye of the beholder. Not only do Halstead et al. not specify what constitutes acceptable psychometrics but they also don't explain how the purported low alpha and poor concurrent validity translate to any meaningful clinical differences.The two unpublished studies cited do little to support their psychometric point given there are so many unknowns regarding how the analyses were conducted, and they are not available for review. For example, although they suggested that the sPaCE detected deterioration at different rates than the CORE, it is unknown which measure achieved the correct rate or how their findings related to final outcomes. Similarly, they reported that the ORS was less stable in predicting change in the first five sessions than the sPaCE, but we are without information regarding what clinical significance that made or how it impacted ultimate outcome. …