Responsible Research Assessment requires structural more than procedural reforms

Abstract

In their target articles, Schönbrodt et al. (2022) and Gärtner et al. (2022) propose new metrics and their practical implementation to improve responsible research assessment. Generally, I welcome the inclusion of open science and scientific rigor into evaluating job candidates. However, the proposed reform mainly focuses on the first stage of selecting candidates who then continue towards a second stage of in-depth evaluation of research quality. Yet, this second selection stage is underdeveloped but likely more critical concerning responsible research assessment and hiring decisions. I argue that an adequate assessment of research quality at this second stage requires the representation of specific knowledge in the subfield of a discipline that the candidate should be hired for by the hiring committee. This is rarely achieved given the current structural organization of departments, especially in German-speaking countries, and potentially explains the reliance on suboptimal indicators such as h-index and Journal Impact factor. Therefore, I argue that responsible research assessment requires structural reform to ensure that institutions have several researchers in permanent positions with specific knowledge in different subfields to provide an adequate and responsible assessment of research quality by hiring committees at all evaluation stages.