Heuristics as conceptual lens for understanding and studying the usage of bibliometrics in research evaluation

Lutz Bornmann,Julian N Marewski

doi:10.1007/s11192-019-03018-x

Lutz Bornmann, Julian N Marewski

Open Access

https://doi.org/10.1007/s11192-019-03018-x

Copy DOI

Abstract

While bibliometrics are widely used for research evaluation purposes, a common theoretical framework for conceptually understanding, empirically studying, and effectively teaching its usage is lacking. In this paper, we outline such a framework: the fast-and-frugal heuristics research program, proposed originally in the context of the cognitive and decision sciences, lends itself particularly well for understanding and investigating the usage of bibliometrics in research evaluations. Such evaluations represent judgments under uncertainty in which typically not all possible options, their consequences, and those consequences’ probabilities of occurring may be known. In these situations of incomplete information, candidate descriptive and prescriptive models of human behavior are heuristics. Heuristics are simple strategies that, by exploiting the structure of environments, can aid people to make smart decisions. Relying on heuristics does not mean trading off accuracy against effort: while reducing complexity, heuristics can yield better decisions than more information-greedy procedures in many decision environments. The prescriptive power of heuristics is documented in a cross-disciplinary literature, cutting across medicine, crime, business, sports, and other domains. We outline the fast-and-frugal heuristics research program, provide examples of past empirical work on heuristics outside the field of bibliometrics, explain why heuristics may be especially suitable for studying the usage of bibliometrics, and propose a corresponding conceptual framework.

Highlights

Imagine the following scene: Agonizing with burning pain in his chest, a man is brought into a hospital
In conceiving of the fast-and-frugal heuristics program as a conceptual lens for studying bibliometrics in research evaluation, we suggest that those heuristics might ease the complex process of assessing scientific quality and deciding on units
We propose to reflect about bibliometrics in terms of the fast-and-frugal heuristics research program in order to study it through the lens of a theoretical framework

Summary

Introduction

Imagine the following scene: Agonizing with burning pain in his chest, a man is brought into a hospital. This tree is built around three questions and can be summarized in terms of three simple if- rules for decision making: (1) If the electrocardiogram reveals a change in the so-called ST-segment, “the patient is immediately sent to the coronary care unit... (3) If there are no ST-segment changes and if chest pain is the chief complaint and any one of five other signs is there, the patient is directed to the coronary care unit; else the patient ends up in a regular nursing bed As it turns out, basing an action on the answers to the tree’s three questions can lead to better outcomes (e.g., adequate patient assignments) than adopting the more complex regression-based Heart Disease Predictive Instrument (e.g., Gigerenzer 2007).

Objectives

Conclusion