Abstract
Abstract During the Italian research assessment exercise (2004–2010), the governmental agency (ANVUR) in charge of its realization performed an experiment on the concordance between peer review and bibliometrics at an individual article level. The computed concordances were at most weak for science, technology, engineering, and mathematics. The only exception was the moderate concordance found for the area of economics and statistics. In this paper, the disclosed raw data of the experiment are used to shed light on the anomalous results obtained for economics and statistics. In particular, the data permit us to document that the protocol of the experiment adopted for economics and statistics was different from the one used in the other areas. Indeed, in economics and statistics the same group of scholars developed the bibliometric ranking of journals for evaluating articles, managing peer reviews and forming the consensus groups for deciding the final scores of articles after having received the referee’s reports. This paper shows that the highest level of concordance in economics and statistics was an artifact mainly due to the role played by consensus groups in boosting the agreement between bibliometrics and peer review.
Highlights
During the research assessment exercise for the years 2004-2010, the Italian governmental Agency for Evaluation of Universities and Research (ANVUR) performed an experiment on the agreement between peer review and bibliometrics at an individual article level (for a recent review of literature see (Baccini, Barabesi, & De Nicolao, 2020))
For the field of economics and statistics, results of the experiment and a big part of the official report were published by Research Policy as a research paper authored by some of the members of the panel appointed by ANVUR to carry out the research assessment in the field (Bertocchi, Gambardella, Jappelli, Nappi, & Peracchi, 2015)
On the basis of the raw data available, this paper aims to establish: (i) if the protocol of the experiment adopted for economics and statistics was different from the one adopted in the other areas; (ii) if this difference was responsible of the anomalous agreement in economics and statistics; (iii) if the description of the experiment published in Bertocchi et al (2015) is correct; (iv) if the claims about the experiment contained in Bertocchi et al (2016) are true of false
Summary
During the research assessment exercise for the years 2004-2010, the Italian governmental Agency for Evaluation of Universities and Research (ANVUR) performed an experiment on the agreement between peer review and bibliometrics at an individual article level (for a recent review of literature see (Baccini, Barabesi, & De Nicolao, 2020)). In a first paper (Baccini & De Nicolao, 2016a), they highlighted an anomalous high level of agreement reached for economics and statistics with respect to all the other research areas They argued that it was due to substantial modifications of the protocol of the experiment in this field with respect to the other areas. 3 This disclosure has permitted Baccini et al (2020) to reconsider in full the experiment by providing the correct design-based setting for it They showed that “for each research areas of science, technology, engineering and mathematics the degree of agreement between bibliometrics and peer review is – at most – weak at an individual article level”.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
