Rater Discrimination Using the Visual Analog Scale of the Physical Therapist Clinical Performance Instrument

Abstract

Background and Purpose. A central issue surrounding the use of a visual analog scale (VAS) of the Physical Therapist Clinical Performance Instrument (CPI) is the level of measurement represented by the method. An often-made assumption is that the VAS is an interval-level measure of the construct of interest. In order for the scale to he valid and for this assumption to be supported, raters must demonstrate the ability to discriminate among 100 levels of student performance. The purpose of this study was to use a measurement model to investigate the validity of the VAS of the CPI when used by clinical instructors to evaluate student performance. Subjects. Final CPI raw scores were collected from 182 subjects from 3 different physical therapist education programs across 4 full-time clinical experiences. Methods. Visual analog scale raw scores for each item were measured (in millimeters) along with rater use of the With Distinction Box. An analysis using the Rasch Measurement Model was performed on the collected data. Step calibration values and average measure values provided by the model were used to analyze rater use of the VAS. Results. A valid rating scale was achieved when the VAS was considered to be a 6-level ordinal scale. Following the collapse of response categories, the 6 groupings of VAS raiu scores were as follows: 0-30 mm=l, 31-50 mm=2, 51-70 mm=3, 71-90 mm=4, 91-100 mm=5, and 100 mm + With Distinction Box=6. Discussion and Conclusion. The inability of clinical instructors to discriminate among 100 levels of student is most likely contributing to the invalid scale. Key Words: Clinical Performance Instrument, Rasch Measurement Model, Rater discrimination, Visual analog scale. INTRODUCTION The Physical Therapist Clinical Performance Instrument (CPI) was recently introduced for use in the assessment of physical therapist students during clinical fieldwork assignments.1 The CPI is a 24-item assessment tool that is believed to describe all essential aspects of professional practice of a physical therapist clinician performing at entry level. Individual competency items are rated by placing a mark on the given 100-mm visual analog scale (VAS), with novice clinical performance to entry-level performance anchoring the ends of the scale. A central issue surrounding the use of a VAS is the level of measurement represented by the method.2'6 An often-made assumption is that the VAS represents an interval-level measure of the construct of interest. Because an investigator applies a ruler to the 100-mm scale to extract a score, a conceptual leap to assuming interval-level measurement is often made/' If the construct of interest is length, the VAS represents a linear scale. However, when the VAS is utilized to assess another construct, such as clinical performance, the VAS is not necessarily assessing the construct at an interval level. Subsequently, research using the VAS of the CPI as an interval-level measure may lead to invalid inferences from the data. Psychometric investigation is required to support the use of the VAS as an interval-level measurement. The CPI has not been subjected to such a study. Related to the interval-level assumption of the VAS is the capacity for discrimination of the rater. If a pencil or pen mark is 1 mm wide, there are approximately 100 different levels of clinical the rater must distinguish among when using the VAS. The validity of the WS as an interval measure is based on the premise that a higher raw score represents a numerically defined increase in the construct of interest. Thus, an 80-mm mark on the WS for item 1 (safety) is assumed to represent twice as much ability related to safety as a 40-mm mark. The assumption is that the rater is able to distinguish among up to 100 distinct levels of response categories, or stimuli. The ability of individuals to distinguish among multiple levels of response categories, or stimuli, has been investigated in the psychological literature in the area of absolute judgment. …