Understanding the use and usability of research evaluation studies1,2

Abstract

Many research funding and research performing organizations evaluate their operations or are subject to evaluations. A Working Group within Science Europe undertook to illustrate how ex post evaluation of research is actually used. Based on 11 case studies from seven organizations and countries, we explore the use of evaluation studies commissioned or conducted by member organizations. We draw on a theoretical framework that categorizes types of evaluation use, including legitimation, information, learning, steering, mediating, tactical, and ritual use. To explore the factors influencing the use of evaluations, we draw on a ‘logic model of evaluation use’ developed by Mark and Henry (2004). The study shows that many uses of evaluation studies can (co-)exist. In particular, direct influence on decision-making (steering use) is not the only type of use that practitioners see in their daily work. Equally important types of use include contribution to the knowledge base and cognitive resources within organizations and beyond. Our study highlights that a methodologically rigorous and high-quality evaluation study is a necessary, but not sufficient, condition to use evaluation results. It draws attention to the fact that the context and other factors external to the evaluation itself—but often within the realm of the commissioning organization—determine the actual use of the evaluation. The findings suggest applying logic models not only to evaluation objects but also to the evaluation exercise itself, so as to consider the underlying mechanisms and chains of influence through which evaluation may have its effects.