Adapting Smart Methodology for use with Subjective Rating Data

Abstract

Decision making based on multiattribute utility theory involves analysis of complex decisions. Each possible course of action or choice is evaluated on a number of important characteristics, and the individual merit scores are then recombined into a single, global utility value for that option. This analytic approach reduces the demands placed on decision makers, and permits review and reconsideration of each step in the decision process. The Simple Multiattribute Rating Technique (SMART) is a method which employs uncomplicated data collection methods, yet produces results similar to more methodologically demanding techniques. Unfortunately, descriptions of SMART procedures do not describe a number of special tests and procedures that can be used with subjective rating data to improve their reliability and validity. A number of biases can occur in rating data, such as: lieniency and severity effects, the tendency for raters to make heavy use of one or the other extreme of rating scale values; sequential effects, biases produced by the type of ratings preceding a given rating item; and halo effects, the tendency for ratings of one characteristic to influence ratings of another. This paper describes several procedures that were employed to minimize these sources of error in a real-world application of SMART-based decision making. It is argued that the procedures employed enhance the credibility of the SMART results, and should therefore be used whenever the decision situation permits collection of appropriate data.