Expert panels evaluating research: decision-making and sources of bias

Abstract

This study consists of in-depth analysis of six international evaluations of Norwegian research. There was little overlapping competence on the panels, a high degree of task division and the composition of an expert panel, the organisation of its work and lack of group interaction, may have been decisive for the conclusions of the evaluations. Moreover, there seems to have been a serious disparity between the processes and resources of the studied evaluations and the demands that ideally should be met when judging scholarly quality. The revealed weaknesses are believed to be inherent to the concept of expert panel evaluation of research as an instrument for national research policy, and not specific for the studied evaluations.