The evaluation of decision-aiding systems: Guidelines and methods

Abstract

Few decision-aiding systems, such as Expert or Decision Support Systems, are formally evaluated. Yet when used to aid a decision making process, there is an assumption that an organization receives a return on the investment inherent in developing (or purchasing) and implementing the system. This return is generated by more effective, more efficient, or delegated (and hence cheaper) decision making. This paper considers the conceptual problems in evaluating decision-aiding systems. It proposes nine guidelines to which an evaluation method should adhere. Three general methods for evaluation are then discussed: control group experiments, attitude scales, and multiple-criteria models. Each is reviewed in the light of the conceptual requirements for evaluation, as represented by the nine guidelines. A general multiple-criteria method that fits the nine guidelines is then presented.