A 20-year evaluation of PLAAS research outputs: Impact on the scholarly domain and in social media

Gillian Kerchhoff,Michelle Kahn,Mary Nassimbeni

doi:10.17159/sajs.2019/5655

Abstract

Patterns and methods of scholarly communication have changed with the growth in information technology, particularly the Internet and the social web. The changes have necessitated a broader definition of scholarly communication and the role of social media in the research process. We sought to record the body of work that the Institute for Poverty, Land and Agrarian Studies (PLAAS), a research institute at the University of the Western Cape, produced over a 20-year period (1995–2015) – the first two decades of its existence – and to measure its visibility and impact using bibliometrics and altmetrics. A survey was also carried out to investigate to what extent PLAAS researchers knew and used social media in their research practice. Scopus and Google Scholar were used as citation indices and Altmetric.com provided Altmetric scores – a measure of impact through social and mainstream media. The full list of PLAAS outputs showed a composition of 54% grey literature and 46% journal articles and monographs. Given that over half of PLAAS research outputs were in the form of grey literature, and therefore not indexed in traditional bibliometric databases, we suggest that alternative metrics be used in conjunction with bibliometrics, to measure the impact of a body of work on the scholarly domain. Although the bibliometrics in this study were a useful quantitative indicator of the impact of PLAAS research, this study was inconclusive with regard to determining the impact of the research output via altmetrics, partly because not any of the grey literature, nor any author from PLAAS, had a unique identifier, thus making it difficult to track and find quantitative indicators. Nonetheless, the potential benefit for PLAAS of using altmetrics was demonstrated in selected case studies of the output of three PLAAS researchers active on social media platforms. Significance:  This study demonstrates that the use of bibliometric and altmetric analyses can yield a rich picture of research output and significance, providing insight into the patterns of scholarly communication of research and policy institutions. The application of the research design in other research units and departments could generate results that are useful to research management within those institutions.

Highlights

Research evaluation is an established practice in scholarly communication, and is important in the allocation of scarce funding to priority areas, as well as in decision-making around tenure and promotions.[1,2] There is growing interest in the metrics used to evaluate research and the people who produce it, as individual peer review becomes more difficult with the growing volume of research produced.[2,3] Peer review is expensive, subjective and suited to small groups or individuals
We evaluated the impact of the scholarly outputs of the Institute for Poverty, Land and Agrarian Studies (PLAAS) at the University of the Western Cape (UWC) from 1995 to 2015 using bibliometric and altmetric analyses
We have shown that in the period under review, PLAAS produced a high number of outputs of many different types, scholarly journal articles and books and much grey literature

Summary

Introduction

Research evaluation is an established practice in scholarly communication, and is important in the allocation of scarce funding to priority areas, as well as in decision-making around tenure and promotions.[1,2] There is growing interest in the metrics used to evaluate research and the people who produce it, as individual peer review becomes more difficult with the growing volume of research produced.[2,3] Peer review is expensive, subjective and suited to small groups or individuals. Bibliometrics is a quantitative method of evaluation and it is emphasised throughout the literature that qualitative peer review should be part of an overall evaluation, and that citation analysis, even using a number of different metrics, should not be used as the only basis on which to base decisions regarding promotion and tenure. Gorraiz et al.[7] state that ‘it cannot be stressed often enough that citations are only used as a proxy for the impact (and not for the quality)’ of the publications in scholarly communication

Objectives

Methods

Findings

Conclusion