Evaluation designs for technology assessments and forecasts

Abstract

The worth of technology assessments and technological forecasts need not remain a matter of faith; indeed, it cannot. Evaluation of technology assessments and forecasts is indicated both to assess their effectiveness and to provide feedback useful for improving the state of the art. Drawing upon evaluation research, we present a range of alternative evaluation designs that fit into four general categories; pre-planned or post-hoc evaluations of single studies; pre-planned or post-hoc comparisons among multiple studies. We distinguish among three classes of performance objectives meriting evaluative attention: validity, utility, and advancement of the methodological art. For each objective we in turn consider pros and cons of several specific evaluation designs. We conclude with recommendations for two coordinated, but selective, evaluation programs, one focused on comparison among extrapolative forecasting techniques and one devoted to certain aspects of technology assessment methods and utilization.