The Association between Four Citation Metrics and Peer Rankings of Research Influence of Australian Researchers in Six Fields of Public Health

Gemma Elizabeth Derrick,Wayne D Hall,Abby Haynes,Simon Chapman,Howard Browman

doi:10.1371/journal.pone.0018521

Gemma Elizabeth Derrick, Wayne D Hall + Show 3 more

Open Access

https://doi.org/10.1371/journal.pone.0018521

Copy DOI

Abstract

Doubt about the relevance, appropriateness and transparency of peer review has promoted the use of citation metrics as a viable adjunct or alternative in the assessment of research impact. It is also commonly acknowledged that research metrics will not replace peer review unless they are shown to correspond with the assessment of peers. This paper evaluates the relationship between researchers' influence as evaluated by their peers and various citation metrics representing different aspects of research output in 6 fields of public health in Australia. For four fields, the results showed a modest positive correlation between different research metrics and peer assessments of research influence. However, for two fields, tobacco and injury, negative or no correlations were found. This suggests a peer understanding of research influence within these fields differed from visibility in the mainstream, peer-reviewed scientific literature. This research therefore recommends the use of both peer review and metrics in a combined approach in assessing research influence. Future research evaluation frameworks intent on incorporating metrics should first analyse each field closely to determine what measures of research influence are valued highly by members of that research community. This will aid the development of comprehensive and relevant frameworks with which to fairly and transparently distribute research funds or approve promotion applications.

Highlights

There are two broad approaches in evaluating research and researchers: traditional methods of peer assessment used for publishing, grant proposals and promotion purposes; and the newer use of citation metrics for comparative evaluation
The aims of this study were: (1) to identify relevant metrics that could be used in conjunction with peer review to assess researchers; and (2) to evaluate the relationship between peer assessment and selected citation metrics among Australian-based public health researchers in six fields
We found a modest positive correlation between peer rankings and research citation-based metrics in all fields except for tobacco and injury

Summary

Introduction

There are two broad approaches in evaluating research and researchers: traditional methods of peer assessment used for publishing, grant proposals and promotion purposes; and the newer use of citation metrics for comparative evaluation. If metrics are to be a useful evaluation method they must be seen to deliver similar results as the considered judgement of peers [2]. This research, sought to investigate how peer assessment corresponds with four different citation metrics. Peer reviewers tend to evaluate work in terms of their own research interests and may not possess the granular knowledge required for expert analysis outside their immediate field nor the broader knowledge needed for making ‘big picture’ judgements about the importance of research. As Langfeldt (2001) argues, ‘while there is a certain set of criteria that reviewers pay attention to – more or less explicitly – these criteria are interpreted or operationalized differently by various reviewers’ [5]

Objectives

Methods

Results

Conclusion