What is meaningful research and how should we measure it?

Sven Helmer,Kathrin Paschen,David B. Blumenthal

doi:10.1007/s11192-020-03649-5

Sven Helmer, Kathrin Paschen + Show 1 more

Open Access

https://doi.org/10.1007/s11192-020-03649-5

Copy DOI

Abstract

We discuss the trend towards using quantitative metrics for evaluating research. We claim that, rather than promoting meaningful research, purely metric-based research evaluation schemes potentially lead to a dystopian academic reality, leaving no space for creativity and intellectual initiative. After sketching what the future could look like if quantitative metrics are allowed to proliferate, we provide a more detailed discussion on why research is so difficult to evaluate and outline approaches for avoiding such a situation. In particular, we characterize meaningful research as an essentially contested concept and argue that quantitative metrics should always be accompanied by operationalized instructions for their proper use and continuously evaluated via feedback loops. Additionally, we analyze a dataset containing information about computer science publications and their citation history and indicate how quantitative metrics could potentially be calibrated via alternative evaluation methods such as test of time awards. Finally, we argue that, instead of over-relying on indicators, research environments should primarily be based on trust and personal responsibility.

Highlights

Quantitative metrics are used for managing education, evaluating health services, and measuring employee performance in corporations, e.g. see Austin (1996)
Scientometrics (2020) 125:153–169 in the academic environment were disproportionately based on quantitative metrics” such as publication and citation count, h-index, i10-index, number of Ph.D. students graduated, and grant income (Royal Society 2017)
In the “Current trends” section, we argue that using metrics in performance management in an area as risky and as unclearly defined as research has an impact: researchers adapt to the metrics used to manage them and this adaptation changes the research practice

Summary

Introduction

Quantitative metrics are used for managing education, evaluating health services, and measuring employee performance in corporations, e.g. see Austin (1996). Keywords Research evaluation · Quantitative metrics · Essentially contested concepts Meaningful research as an essentially contested concept It can be argued (Ferretti et al 2018) that the concept of meaningful research is essentially contested in the sense introduced by Gallie (1955).

Results

Conclusion