A numerical algorithm with preference statements to evaluate the performance of scientists.

Martin Ricker

doi:10.1007/s11192-014-1521-2

Abstract

Academic evaluation committees have been increasingly receptive for using the number of published indexed articles, as well as citations, to evaluate the performance of scientists. It is, however, impossible to develop a stand-alone, objective numerical algorithm for the evaluation of academic activities, because any evaluation necessarily includes subjective preference statements. In a market, the market prices represent preference statements, but scientists work largely in a non-market context. I propose a numerical algorithm that serves to determine the distribution of reward money in Mexico’s evaluation system, which uses relative prices of scientific goods and services as input. The relative prices would be determined by an evaluation committee. In this way, large evaluation systems (like Mexico’s Sistema Nacional de Investigadores) could work semi-automatically, but not arbitrarily or superficially, to determine quantitatively the academic performance of scientists every few years. Data of 73 scientists from the Biology Institute of Mexico’s National University are analyzed, and it is shown that the reward assignation and academic priorities depend heavily on those preferences. A maximum number of products or activities to be evaluated is recommended, to encourage quality over quantity.

Highlights

In Mexico, a traditional stepwise promotion system for scientists is complemented with a periodic performance evaluation system that contributes additional income to scientists’ salaries
In addition to the basic salary according to those levels, there are two reward payments based upon performance
Scientists are evaluated for promotion to a higher category in the traditional sense, and every few years according to their productivity during the last period

Summary

Introduction

In Mexico, a traditional stepwise promotion system for scientists is complemented with a periodic performance evaluation system that contributes additional income to scientists’ salaries. The experience is that both reward systems, the PRIDE and the SNI, struggle to evaluate many scientists in academic evaluation committees, without a framework how to weigh different product categories relative to each other, e.g., publishing a book against publishing less articles. The lowest reward with US$1,827 is for finishing as coauthor a non-indexed article, a book chapter, or a technical report This first scenario is called ‘‘balanced values for a product mix,’’ because the median relative value (w) is 0.5 (the mean 0.68), and the interquartile range from 0.45 to 0.9. The second scenario is called ‘‘priority on indexed articles’’, with a median relative value for all product categories of 0.1 (mean of 0.22), the interquartile range from 0.1 to 0.15, and technical reports receiving a relative value of 0. We could think of applied science, where the values of actual or potential users and the educated laymen in society should be reflected

Make a list of all acceptable products to consider

Findings

Discussion