Do altmetrics correlate with the quality of papers? A large-scale empirical study based on F1000Prime data.

Lutz Bornmann,Robin Haunschild,Pablo Dorta-González

doi:10.1371/journal.pone.0197133

Lutz Bornmann, Robin Haunschild + Show 1 more

Open Access

https://doi.org/10.1371/journal.pone.0197133

Copy DOI

Abstract

In this study, we address the question whether (and to what extent, respectively) altmetrics are related to the scientific quality of papers (as measured by peer assessments). Only a few studies have previously investigated the relationship between altmetrics and assessments by peers. In the first step, we analyse the underlying dimensions of measurement for traditional metrics (citation counts) and altmetrics–by using principal component analysis (PCA) and factor analysis (FA). In the second step, we test the relationship between the dimensions and quality of papers (as measured by the post-publication peer-review system of F1000Prime assessments)–using regression analysis. The results of the PCA and FA show that altmetrics operate along different dimensions, whereas Mendeley counts are related to citation counts, and tweets form a separate dimension. The results of the regression analysis indicate that citation-based metrics and readership counts are significantly more related to quality, than tweets. This result on the one hand questions the use of Twitter counts for research evaluation purposes and on the other hand indicates potential use of Mendeley reader counts.

Highlights

Altmetrics denote non-traditional metrics which represent an alternative form of impact measurement instead of using citations [1]
One important reason for the use of altmetrics in research evaluation is the measurement of wider research impact: research with important societal or cultural impacts may be undervalued if assessed with citation-based indicators
Following earlier studies by Bornmann [10] and Bornmann [11], we address in this study the question whether altmetrics are related to the scientific quality of papers

Summary

Introduction

Altmetrics denote non-traditional metrics which represent an alternative form of impact measurement instead of using citations [1]. Altmetrics usually cover activities on social media platforms including, for example, mentions in blog posts, readership counts on Mendeley, shares on Facebook, and tweets, and mentions in mainstream media and policy documents [2,3]. Altmetrics refer to a heterogeneous set of metrics which is gaining increasing popularity amongst researchers, research communicators, publishers, and research funders [4]. Wiley, Springer, BioMed Central and the Nature Publishing Group are adding altmetrics to papers in their collections. One important reason for the use of altmetrics in research evaluation is the measurement of wider research impact: research with important societal or cultural impacts may be undervalued if assessed with citation-based indicators.

Methods

Results

Conclusion