Developing and testing a new outcome measure

Abstract

Health outcome measures are used to assess treatment effectiveness. Historically, survival was the most important health outcome. Treatments were assessed based on whether the patient lived or died. Now in the 21st century, treatments for many injuries and illnesses are so effective that many, if not most, patients survive. If patients survive, the quality of their survival becomes an important health outcome. The severity of impairments in body structure (limb loss) and function (weakness) are one way to assess the quality of survival. However, from the perspective of the patient, discomfort and disability are probably more important health outcomes. Limitation in mobility activities is an important component of disability, particularly for individuals with lower-limb loss (LLL). The treatment effectiveness of the rehabilitative care provided to servicemembers with LLL should be assessed by examining its effect on mobility limitations. Unfortunately, it is much more difficult to accurately measure mobility limitations than it is to measure survival. Measurement requires developing a set of rules to assign numbers to represent a concept or health outcome. When outcome measures contain multiple items, rules must also be developed to combine item scores to generate total and subscale scores. Determining the set of rules that will best represent a particular health outcome is affected by both the reason for measuring the outcome and the types of individuals being measured. Outcome measures can be used to examine small changes resulting from a treatment or to place individuals into broad categories. The purpose for using the outcome measure will dictate the types of items selected and the measurement dimension attached to these items. Measures of mobility activity limitations can be either performance-based or self-report. The types of individuals being measured will often determine which approach is best. A performance-based outcome measure requires both a set of rules for performing the test and a set of rules for scoring the test. It is not enough to simply create an outcome measure. It is essential to determine whether the rules used to create the outcome measure work to consistently and accurately represent the concept being measured. RELIABILITY For an outcome measure to be useful, it must be reliable in that it produces consistent findings if no real change has occurred. Performance-based outcome measures use raters. Raters must be trained to follow a standard set of rules to administer and score the measure. If raters do not adhere to rules, measurement errors may occur that adversely affect the reliability of the measure. Two types of rater reliability can be examined. Intrarater reliability indicates how consistently a rater administers and scores an outcome measure. Interrater reliability indicates how well two raters agree in the way they administer and score an outcome measure. To evaluate a measure's ability to detect real change, we must also examine score consistency over short periods in which no real change should occur. This is called test-retest reliability. An outcome measure used to evaluate progress over time must also be responsive in its ability to detect real change. Test-retest reliability is a critical factor in determining how well an outcome measure will detect real change. Reliability can be examined experimentally by testing how well scores agree between raters and time periods. Agreement is expressed mathematically by calculating a reliability coefficient representing the ratio of true score variance divided by true score variance plus error variance. A reliability coefficient of 1.0 represents perfect reliability, indicating that all of the differences between scores represent real differences between individuals. A reliability coefficient of 0.43 indicates that 43 percent of the variance is due to true score and 57 percent of the variance is due to measurement error. …