Methodology for evaluating energy R&D. Final report

Abstract

Recent budgetary shortfalls and hightened concern over balancing the federal budget have placed increasing demand on federal agencies to document the cost effectiveness of the programs they manage. In fact, the 1993 Government Performance and Results Act (GPRA) requires that by 1997 each executive agency prepare a Strategic Plan that includes measurable performance goals. By the year 2000, the first round of Annual Reports will become due which describes actual program performance. Despite the growing emphasis on measuring performance of government programs, the technology policy literature offers little in terms of models that program managers can implement in order to assess the cost effectiveness of the programs they manage. While GPRA will pose a major challenge to all federal government agencies, that challenge is particularly difficult for research-oriented agencies such as the Department of Energy. Its basic research programs provide benefits that are difficult to quantify since their values are uncertain with respect to timing, but are usually reflected in the value assigned to applied programs. The difficulty with quantifying benefits of applied programs relates to the difficulties of obtaining complete information on industries that have used DOE`s supported technologies in their production processes and data on cost-savings relative to conventional technologies. Therefore, DOE is one of several research-oriented agencies that has a special need for methods by which program offices can evaluate the broad array of applied and basic energy research programs they administer. The general findings of this report are that few new methods are applicable for evaluation of R&D programs. It seems that peer review and bibliometrics are methods of choice for evaluating basic research programs while more quantitative approaches such as ROI, cost-benefits, etc. might be followed in evaluating applied programs.