Research quality and efficiency: An analysis of assessments and management issues in Dutch economics and business research programs

Abstract

Assessments of quality and productivity of academic research programs become more and more important in gaining financial support, in hiring and promoting research staff, and in building academic reputation. Most assessments are based on peer review or on bibliometric information. In this paper we analyze both bibliometric data and peer review assessments of 169 research groups in economics, econometrics and business administration. The evaluations are achieved in two independent rounds in 1995 and in 2001, permitting replication of our study. The purpose of this study is twofold. In the first part we want to see to what degree bibliometric information relates to peer review judgments. The results convey how evaluators weight different output categories in their final overall judgment of academic quality. The results also have practical meaning, since they indicate what the predictive ability of bibliometric data is for future peer review outcomes. In the second part of this study we aim at explaining differences in research output quality and productivity by organizational factors, like size of the research group, composition of staff, sources of research funding and academic discipline. In this part, a composite indicator is used to represent the review committees’ overall assessment. The bibliometric data most strongly related to the peer reviews’ overall assessment are used to construct data envelopment analyses’ efficiency scores as measure of research productivity. The main conclusions from our study are that the number of publications in international top journals is the best predictor of peer review assessment results. Changes in the classification of bibliometric information, as introduced in the second evaluation round, do not alter this conclusion. Size of the research group appears to be the only permanent characteristic associated with research quality and productivity. Size is positively related to research quality but negatively related to research productivity. Larger groups appear to have the potential to improve quality, but as groups become larger, they also experience problems in maintaining the research productivity of the research team's members. The remaining organizational characteristics appear to be temporarily related to research quality and productivity. In the first evaluation round, research productivity and quality are associated with the discipline variable: research programs in more quantitative areas and characterized by a higher level of paradigm development like econometrics and operations research achieved higher levels of research quality and productivity than programs in more diverse and less quantitative areas like business administration. This relation however is not permanent, since it becomes insignificant in the second evaluation round. Instead, funding relations become more apparent in the second review round. The relative amount of national funding in the research group's funding becomes positively related to academic quality, whereas the portion of income from committed research is negatively related to academic quality of the programs’ research output. This may have been caused by the increased importance of alternative sources of research funding in the period of the second review.