Abstract
Research evaluation is often understood as something similar to a competition, where an evaluation panel’s task is to award the most excellent researchers. This interpretation is challenging, in as far as excellence it is at best a multi-dimensional concept and at worst an ill-defined term because it assumes that there exists some ground truth as to who the very best researchers are and all that an evaluation panel needs to do is uncover this ground truth. Therefore, instead of focusing on competition, the Swiss National Science Foundation focused on active decision-making and sought inspiration in the deliberation proceedings of a jury trial for the design of a new evaluation procedure of an academic award. The new evaluation procedure is based upon fully anonymised documents consisting of three independent parts (achievements, impact and prominence). Before the actual evaluation meeting, the panel, which includes non-academic experts, pre-evaluates all nominations through a pseudo-randomly structured network, such that every nomination is reviewed by six members of the panel only. Evaluation decisions are based upon anonymous votes, structured discussions in the panel, ranking as opposed to rating of nominees and data-rich figures providing an overview of the positioning of the nominee along various dimensions and the ranking provided by the individual panel members. The proceedings are overseen by an academic chair, focusing on content, and a procedural chair, focusing on the process and compliance. Combined, these elements form a highly-structure deliberation procedure, consisting of individual steps, through which nominations proceed and which each either feed into the next step or into the final verdict. The proposed evaluation process has been successfully applied in the real world in the evaluation of the Swiss Science Prize Marcel Benoist, Switzerland’s most prestigious academic award.
Highlights
Prestigious academic prizes are usually awarded based on evaluation procedures where a group of experts decides who to award
Whereas in other competitions one can call upon millimetres and milliseconds as measures of merit, in research evaluation it is the formulation of a convincing argument within the evaluation panel, which calls the winner
A proposal Inspired by the analogy of a jury trial and the lack of such clear and transparent processes among academic prize committees, we propose a new award evaluation procedure
Summary
Prestigious academic prizes are usually awarded based on evaluation procedures where a group of experts decides who to award. Separating the nomination text into three sections this way allowing the members of the evaluation panel to assess the actual achievements of the individual researcher independently of their prestigious associations, and the impact of their work without it being conflated with the work’s quality. For example, a panel member has to pre-evaluate 15 nominations in their sub-set, they would have to separately assign five bronze, five silver and five gold medals for achievements and for impact, respectively. Compliance of individual panel members (e.g. no googling of nominations) and overall adherence to DORA guidelines (e.g. weighting the actual achievements of a nomination higher than the journal in which it was published) were some of the main challenges, which we encountered during the implementation of this process; they are, not unique or particular to this evaluation procedure. The experience with this evaluation procedure has been very positive and the SNSF will continue using it for the evaluation of the MBP and possibly other prizes in the future and some of its aspects are being adapted to other funding procedures
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
