Validity in Evaluation Research: a critical assessment of current issues

Abstract

This article reviews current major methods of ensuring validity in evaluation research. In reviewing rationales advanced for the primacy of the experimental paradigm, I argue that the use of randomized experiments in program evaluation may not be as powerful as its advocates claim. Furthermore, the rigidity of the randomized experimental design may emphasize internal validity at the expense of other types of validity. Especially important, I argue, is that evaluations based on experimental paradigms tend not to yield information useful to policy makers. Various quasi-experiments have been considered inferior to randomized experiments in internal validity by Campbell and Stanley. However, due to later developments of sophisticated methods as ARIMA models and modelling selection bias, the gap between randomized experiments and quasi-experiments in dealing with internal validity has narrowed. Furthermore, quasi-experiments also have other advantages such as unobtrusiveness and greater applicability to the evaluation of human services programs. Current approaches to validity tend to be a single-validity oriented approach; that is, they tend to achieve one type of validity at the expense of others. I propose a theory-driven approach that attempts to provide a more balanced view. The emphasis of this approach is to stress the incorporation of formal models or theory into program evaluation, thereby articulating clearly the identified potential threats to validity in a research. The theory-driven approach can and should deal with internal, external, construct, and statistical conclusion validity simultaneously.