Measuring Research Impact Research Articles

Encouraging translation and assessing impact of the Centre for Research Excellence in Integrated Quality Improvement: rationale and protocol for a research impact assessment

IntroductionThere is growing recognition among health researchers and funders that the wider benefits of research such as economic, social and health impacts ought to be assessed and valued alongside academic...

The new alchemy: Online networking, data sharing and research activity distribution tools for scientists

There is an abundance of free online tools accessible to scientists and others that can be used for online networking, data sharing and measuring research impact. Despite this, few scientists know how these tools can be used or fail to take advantage of using them as an integrated pipeline to raise awareness of their research outputs. In this article, the authors describe their experiences with these tools and how they can make best use of them to make their scientific research generally more accessible, extending its reach beyond their own direct networks, and communicating their ideas to new audiences. These efforts have the potential to drive science by sparking new collaborations and interdisciplinary research projects that may lead to future publications, funding and commercial opportunities. The intent of this article is to: describe some of these freely accessible networking tools and affiliated products; demonstrate from our own experiences how they can be utilized effectively; and, inspire their adoption by new users for the benefit of science.

The #AustralianBeachspace Project: Examining Opportunities for Research Dissemination Using Instagram

The #AustralianBeachspace Project: Examining Opportunities for Research Dissemination Using Instagram

Assessing the impact of healthcare research: A systematic review of methodological frameworks.

BackgroundIncreasingly, researchers need to demonstrate the impact of their research to their sponsors, funders, and fellow academics. However, the most appropriate way of measuring the impact of healthcare research is subject to debate. We aimed to identify the existing methodological frameworks used to measure healthcare research impact and to summarise the common themes and metrics in an impact matrix.Methods and findingsTwo independent investigators systematically searched the Medical Literature Analysis and Retrieval System Online (MEDLINE), the Excerpta Medica Database (EMBASE), the Cumulative Index to Nursing and Allied Health Literature (CINAHL+), the Health Management Information Consortium, and the Journal of Research Evaluation from inception until May 2017 for publications that presented a methodological framework for research impact. We then summarised the common concepts and themes across methodological frameworks and identified the metrics used to evaluate differing forms of impact. Twenty-four unique methodological frameworks were identified, addressing 5 broad categories of impact: (1) ‘primary research-related impact’, (2) ‘influence on policy making’, (3) ‘health and health systems impact’, (4) ‘health-related and societal impact’, and (5) ‘broader economic impact’. These categories were subdivided into 16 common impact subgroups. Authors of the included publications proposed 80 different metrics aimed at measuring impact in these areas. The main limitation of the study was the potential exclusion of relevant articles, as a consequence of the poor indexing of the databases searched.ConclusionsThe measurement of research impact is an essential exercise to help direct the allocation of limited research resources, to maximise research benefit, and to help minimise research waste. This review provides a collective summary of existing methodological frameworks for research impact, which funders may use to inform the measurement of research impact and researchers may use to inform study design decisions aimed at maximising the short-, medium-, and long-term impact of their research.

The new alchemy: Online networking, data sharing and research activity distribution tools for scientists

There is an abundance of free online tools accessible to scientists and others that can be used for online networking, data sharing and measuring research impact. Despite this, few scientists know how these tools can be used or fail to take advantage of using them as an integrated pipeline to raise awareness of their research outputs. In this article, the authors describe their experiences with these tools and how they can make best use of them to make their scientific research generally more accessible, extending its reach beyond their own direct networks, and communicating their ideas to new audiences. These efforts have the potential to drive science by sparking new collaborations and interdisciplinary research projects that may lead to future publications, funding and commercial opportunities. The intent of this article is to: describe some of these freely accessible networking tools and affiliated products; demonstrate from our own experiences how they can be utilized effectively; and, inspire their adoption by new users for the benefit of science.

Altmetrics: A Measure of Social Attention toward Scientific Research

The quality and impact of scientific research is traditionally measured by citation-based metrics. However, the internet and social media revolution has led to radical changes in the way scientific information is shared and how it impacts the world. Alternative metrics (altmetrics) is a rapidly evolving measure to quantify social media attention received by a scientific work. It is a complement to the conventional metrics, and has tremendous potential in enabling faster alternative ways of evaluating research impact. We discuss the Altmetric Score and the growing role of altmetrics for measuring research impact.

Impact in participatory health research: what can we learn from research on participatory evaluation?

Participatory Health Research is a collective term adopted globally for participatory action research in a health context. As an approach to research, it challenges current ways used within the health sciences to measure research impact as research, learning and action are integrated throughout the research process and dependent on context and participation. The literature on participatory evaluation is explored to see if it can offer some insights into how best to articulate impact. Similar debates are taking place particularly concerning the relationship between degree of participation and impact, how impact should be defined and the role of wider social forces on the evaluation process. A focus on core values, such as social justice, differentiates transformative evaluation from more pragmatic participatory evaluation, but this is poorly conceptualised in evaluation models.

Methodological Rigor in Preclinical Cardiovascular Studies: Targets to Enhance Reproducibility and Promote Research Translation.

Methodological sources of bias and suboptimal reporting contribute to irreproducibility in preclinical science and may negatively affect research translation. Randomization, blinding, sample size estimation, and considering sex as a biological variable are deemed crucial study design elements to maximize the quality and predictive value of preclinical experiments. To examine the prevalence and temporal patterns of recommended study design element implementation in preclinical cardiovascular research. All articles published over a 10-year period in 5 leading cardiovascular journals were reviewed. Reports of in vivo experiments in nonhuman mammals describing pathophysiology, genetics, or therapeutic interventions relevant to specific cardiovascular disorders were identified. Data on study design and animal model use were collected. Citations at 60 months were additionally examined as a surrogate measure of research impact in a prespecified subset of studies, stratified by individual and cumulative study design elements. Of 28 636 articles screened, 3396 met inclusion criteria. Randomization was reported in 21.8%, blinding in 32.7%, and sample size estimation in 2.3%. Temporal and disease-specific analyses show that the implementation of these study design elements has overall not appreciably increased over the past decade, except in preclinical stroke research, which has uniquely demonstrated significant improvements in methodological rigor. In a subset of 1681 preclinical studies, randomization, blinding, sample size estimation, and inclusion of both sexes were not associated with increased citations at 60 months. Methodological shortcomings are prevalent in preclinical cardiovascular research, have not substantially improved over the past 10 years, and may be overlooked when basing subsequent studies. Resultant risks of bias and threats to study validity have the potential to hinder progress in cardiovascular medicine as preclinical research often precedes and informs clinical trials. Stroke research quality has uniquely improved in recent years, warranting a closer examination for interventions to model in other cardiovascular fields.

Understanding Traditional Research Impact Metrics.

Traditionally, the success of a researcher has been judged by the number of publications he or she has published in peer-review, indexed, high impact journals. However, to quantify the impact of research in the wider scientific community, a number of traditional metrics have been used, including Impact Factor, SCImago Journal Rank, Eigenfactor Score, and Article Influence Score. This article attempts to provide a broad overview of the main traditional impact metrics that have been used to assess scholarly output and research impact. We determine that there is no perfect all-encompassing metric to measure research impact, and, in the modern era, no single traditional metric is capable of accommodating all facets of research impact. Academics and researchers should be aware of the advantages and limitations of traditional metrics and should be judicious when selecting any metrics for an objective assessment of scholarly output and research impact.

Editorial Volume 8, Issue 1 2017

Welcome to the first issue of Student Success for 2017. Following the tradition initiated in Volume 7, this editorial has two parts: The first part continues our objective of creating awareness of good practice in scholarly publishing in an Open Access (OA) forum. Future editorials will briefly discuss other aspects and issues pertaining to the evolving scholarly publishing landscape that this journal adheres to, such as: Creative Commons Licencing; Considerations of new peer review models and importantly; Measuring research impact in OA publishing. The second part presents the usual editorial summary of the content of this issue.

A Bibliometric Analysis of the Top 100 Most Cited Chronotype Research Papers.

Bibliometric indices are a widely used measure of research impact. The aim of the current study was to identify and characterise the top one hundred most-cited research articles in the topic of chronotype research. A search of the Thomson Reuters Web of Science database returned 974 eligible articles (published between 1990 and 2016). Citations for the 100 most-cited articles ranged between 438 and 29. The most represented journal was Chronobiology International (n = 30). Nearly 50% of articles originated in Germany and the U.S. The bibliometrics reported identify key publications and provide insight into trends within the topic of chronotype research.

Supporting Research Impact Metrics in Academic Libraries: A Case Study

Measuring research impact has become a nearly ubiquitous facet of scholarly communication. At the University of Minnesota Medical School, new administrative directives have directly tied impact metrics to faculty assessment, promotion, and tenure. In this paper, I describe a platform for the analysis and visualization of research impact that was developed in the University of Minnesota Libraries specifically to support these directives. The implementation of that platform, named Manifold, has required not only major technological resources but also new investments in user education. The libraries’ experience with Manifold offers a case study of the changing roles academic librarians increasingly face in managing expectations around research impact metrics.

A Bayesian h‐index: How to measure research impact

The quality of academic research is difficult to measure and rather controversial. Hirsch has proposed the h index [1], a measure that has the advantage of summarizing in a single summary statistic the information that is contained in the citation counts of each scientist. Although the h index has received a great deal of interest, only a few papers have analyzed its statistical properties and implications. We claim that statistical modeling can give a lot of added value over a simple summary like the h index. To show this, in this paper we propose a negative binomial distribution to jointly model the two main components of the h index: the number of papers and their citations. We then propose a Bayesian model that allows to obtain posterior inferences on the parameters of the distribution and, in addition, a predictive distribution for the h index itself. Such a predictive distribution can be used to compare scientists on a fairer ground, and in terms of their future contribution, rather than on their past performance.

Rims: The Research Impact Measurement Service At The University Of New South Wales

In 2005, the Library at UNSW began a comprehensive restructuring process that fundamentally changed the provision of services to its academic community. A primary aim of this process was to increase flexibility of service development and delivery and so to improve research support. The motivation for reformed services arose from considerations including the University Library need to realign its services to support the university’s strategic goals, the increasingly competitive nature of the research environment, the introduction of the RQF/ERA, and a renewed emphasis on research outcomes by UNSW. The measurement of research impact using bibliometrics was seen as a strategy for supporting UNSW researchers. University Library staff consulted the bibliometrics literature and apropriate methodologies were devised to measure the impact of publications, authors and departments. The result was the creation of a Research Impact Measurement Service (RIMS) that now produces over 30 reports every month and employs 6...

Reflection on: “RIMS: The Research Impact Measurement Service at the University of New South Wales”

Seven years ago, I was approached by my then colleague, Richard Wartho, to co-write an article about a service we had been delivering for a couple years at UNSW Library called the Research Impact M...

Academic Librarians’ Knowledge of Bibliometrics and Altmetrics

Objective – To measure the knowledge and opinions that academic librarians have of established and emerging research metrics.
 
 Methods – An online survey was distributed to all academic librarians in Oklahoma during Summer 2015.
 
 Results – Librarians were less familiar with altmetrics than with bibliometrics, but they viewed altmetrics as effective and were interested in receiving training to learn more about them. Librarians who had been in the profession for over five years knew more about both bibliometrics and altmetrics than newer librarians.
 
 Conclusions – Technological advances and changes in the ways that research products are shared have led to the possibility of and need for new ways of measuring research impact. Altmetrics have emerged to fill this need, but academic librarians need more familiarity and training to be able to fulfill a role as providers of these metrics.

Altmetrics: an analysis of the state-of-the-art in measuring research impact on social media

Altmetrics is an emergent research area whereby social media is applied as a source of metrics to assess scholarly impact. In the last few years, the interest in altmetrics has grown, giving rise to many questions regarding their potential benefits and challenges. This paper aims to address some of these questions. First, we provide an overview of the altmetrics landscape, comparing tool features, social media data sources, and social media events provided by altmetric aggregators. Second, we conduct a systematic review of the altmetrics literature. A total of 172 articles were analysed, revealing a steady rise in altmetrics research since 2011. Third, we analyse the results of over 80 studies from the altmetrics literature on two major research topics: cross-metric validation and coverage of altmetrics. An aggregated percentage coverage across studies on 11 data sources shows that Mendeley has the highest coverage of about 59 % across 15 studies. A meta-analysis across more than 40 cross-metric validation studies shows overall a weak correlation (ranging from 0.08 to 0.5) between altmetrics and citation counts, confirming that altmetrics do indeed measure a different kind of research impact, thus acting as a complement rather than a substitute to traditional metrics. Finally, we highlight open challenges and issues facing altmetrics and discuss future research areas.

An approach to measuring and encouraging research translation and research impact.

BackgroundResearch translation, particularly in the biomedical area, is often discussed but there are few methods that are routinely used to measure it or its impact. Of the impact measurement methods that are used, most aim to provide accountability – to measure and explain what was generated as a consequence of funding research. This case study reports on the development of a novel, conceptual framework that goes beyond measurement. The Framework To Assess the Impact from Translational health research, or FAIT, is a platform designed to prospectively measure and encourage research translation and research impact. A key assumption underpinning FAIT is that research translation is a prerequisite for research impact.MethodsThe research impact literature was mined to understand the range of existing frameworks and techniques employed to measure and encourage research translation and research impact. This review provided insights for the development of a FAIT prototype. A Steering Committee oversaw the project and provided the feedback that was used to refine FAIT.ResultsThe outcome of the case study was the conceptual framework, FAIT, which is based on a modified program logic model and a hybrid of three proven methodologies for measuring research impact, namely a modified Payback method, social return on investment, and case studies or narratives of the process by which research translates and generates impact.ConclusionAs funders increasingly seek to understand the return on their research investments, the routine measurement of research translation and research impact is likely to become mandatory rather than optional. Measurement of research impact on its own is insufficient. There should also be a mechanism attached to measurement that encourages research translation and impact – FAIT was designed for this task.

Volume 7, Issue 1 2016

Welcome to Volume 7 of Student Success. This editorial has two parts: The first part maintains the “doing things differently” tradition, making readers aware by chronicling the publishing of the journal in an open access (OA) forum. Future editorials will briefly discuss other aspects and issues pertaining to the new scholarly publishing landscape that this journal adheres to, such as: Creative Commons Licencing; ORCID IDs; considerations of new peer review models and importantly; measuring research impact in OA publishing. The second part presents the usual editorial summary of the content of this issue.

Investigating singapore's altmetric landscape

ABSTRACTAltmetrics is an emerging measure for academic impact and it is gaining in global importance. In this paper, we analyse the altmetric landscape of Singapore, a young nation with a fast growing international research sector. We aim to find out if the coverage of altmetrics across the different disciplines is increasing along with the fast increase in the amount of research publications in recent years. We also look into how altmetrics relate to traditional citation counts as a measure of research impact. From our results, we see that there is overall an 18% coverage of altmetrics of Singapore publications from 2009 to 2013. The number of publications with available altmetric data has also been increasing over the years for most disciplines. Correlation results between citation counts and altmetrics show medium to low correlations with distinct differences amongst the various disciplines. A high coverage of altmetrics however does not seem to lead to larger correlations with citation counts. Singapore thus remains an intriguing case study to watch in the coming years.