Measuring Research Impact Research Articles

Towards new ways of assessing the impact of local medical journals: A proposal and call for change

The relevance of research output to the local community is critical to changing practice. Research relevance has to be determined using measurements that show that the knowledge that arose from that research has made an impact on society. This paper, based on a literature review and preliminary research results, advocates for research impact measurements which take account of local contexts when evaluating the relevance of a journal article or indeed any research output. It concludes that a journal should go beyond traditional measurement metrics of citation analysis and bibliometrics alone as a measure of research impact. Although it is important to standardise measurements, it is also important that local communities should be encouraged to choose measurements of research output that matter to them. The proposed ways of assessing research impact are: (a) change in policies in the health sector, (b) effect on local medical treatment guidelines, (c) effect on case management, (d) use in continuous professional development, and (e) impact on local knowledge production.

Approaching Impact Meaningfully in Medical Education Research.

Medical education research faces increasing pressure to demonstrate impact and utility. These pressures arise amidst a climate of accountability and within a culture of outcome measurement. Conventional metrics for assessing research impact such as citation analysis have been adopted in medical education, despite researchers' assertion that these quantitative measures insufficiently reflect the value of their work. Every knowledge community has its own definitions of what counts as knowledge, how that knowledge should be produced, and how the quality of that knowledge production should be evaluated. Definitions of impact and knowledge shape and constrain researchers' foci and endeavors. Therefore, metrics that meaningfully evaluate the knowledge outputs of researchers need to be defined within each field. It is time for medical education research, as a field, to examine how to measure research impact and carefully consider the broader implications these measures may have. The authors discuss developments in research metrics more broadly, then critically examine impact metrics currently used in the medical education field and propose alternatives to more meaningfully track and represent impact in medical education research. Grey metrics and narrative impact stories to more fully capture the richness and nuanced nature of impact in medical education research are introduced. The authors advocate for a continual examination of how impact is defined, eschewing unquestioned use of conventional metrics. A new conversation is needed, as well as a research agenda to help medical education conceptualize and study metrics more appropriate for the field.

Investigation and analysis of research support services in academic libraries

PurposeThis paper aims at understanding the current situation of research support services offered by academic libraries in world-leading universities and providing useful implications and insights for other academic libraries.Design/methodology/approachOf the top 100 universities listed in the QS World University Rankings in 2017, 76 libraries were selected as samples and a website investigation was conducted to explore the research support services. The statistical method and visualization software was used to generalize the key services, and the text analysis and case analysis were applied to reveal the corresponding implementation.FindingsResearch support service has become one of the significant services of academic libraries in the context of e-research and data-intensive research. The research support services can be generally divided into seven aspects, as follows: research data management (62, 81.58 per cent), open access (64, 84.21 per cent), scholarly publishing (59, 77.63 per cent), research impact measurement (32, 42.11 per cent), research guides (47, 61.84 per cent), research consultation (59, 77.63 per cent) and research tools recommendation (38, 50.00 per cent).Originality/valueThis paper makes a comprehensive investigation of research support services in academic libraries of top-ranking universities worldwide. The findings will help academic libraries improve research support services; thus, advancing the work of researchers and promoting scientific discovery.

В поисках сбалансированных интернет-метрик научной активности: пример Rg Score

Основной целью реформирования государственной системы науки в России является повышение эффективности, что требует пересмотра калькулируемых индикаторов научной деятельности. Процесс осложняется трансформацией научной коммуникации, преимущественно детерминированной развитием коммуникационных технологий, благодаря которым видоизменяются не только средства, но и формы научного взаимодействия. В подобных условиях классические библиометрические показатели не могут выступать в качестве исчерпывающих метрик, так как возникает необходимость учета научной деятельности во всей полноте, а не только на интертекстуальном уровне. Необходимые метрики могут быть разработаны с учетом процесса цифровизации науки. На данный момент «невидимый колледж» становится осязаемым благодаря развитию ряда научных интернет-площадок, предоставляющих возможность работать не только с огромными массивами данных, но и с собственными индексами. В данной статье рассматривается один из таких показателей — Rg Score, призванный измерить активность пользователей самой крупной на данной момент академической социальной сети — ResearchGate. Архитектура этого интернет-сервиса предоставляет возможность изучения и, отчасти, моделирования показателей нового типа, учитывающих не только публикационную, но и коммуникативную активность исследователей. Конечно, данный показатель пока не может быть использован в рамках управленческой деятельности, однако подобный тип альтметрик видится нам вполне перспективным.

Strengthening and measuring research impact in global health: lessons from applying the FAIT framework

BackgroundTo date, efforts to measure impact have largely focused on health research in high-income countries, reflecting where the majority of health research funding is spent. Nevertheless, there is a growing body of health and medical research being undertaken in low- and middle-income countries (LMICs), supported by both development aid and established research funders. The Framework to Assess the Impact of Translational health research (FAIT) combines three approaches to measuring research impact (Payback, economic assessment and case study narrative). Its aim is to strengthen the focus on translation and impact measurement in health research. FAIT has been used by several Australian research initiatives; however, it has not been used in LMICs. Our aim was to apply FAIT in an LMIC context and evaluate its utility.MethodsWe retrospectively applied all three FAIT methods to two LMIC studies using available data, supplemented with group discussion and further economic analyses. Results were presented in a scorecard format.ResultsFAIT helped clarify pathways of impact for the projects and provided new knowledge on areas of impact in several domains, including capacity-building for research, policy development and economic impact. However, there were constraints, particularly associated with calculating the return on investment in the LMIC context. The case study narrative provided a layperson’s summary of the research that helped to explain outcomes and succinctly communicate lessons learnt.ConclusionUse of FAIT to assess the impact of LMIC research was both feasible and useful. We make recommendations related to prospective use, identification of metrics to support use of the Payback framework, and simplification of the economic assessment, which may facilitate further application in LMIC environments.

Development and validation of a questionnaire to measure research impact

Abstract Although questionnaires are widely used in research impact assessment, their metric properties are not well known. Our aim is to test the internal consistency and content validity of an instrument designed to measure the perceived impacts of a wide range of research projects. To do so, we designed a questionnaire to be completed by principal investigators in a variety of disciplines (arts and humanities, social sciences, health sciences, and information and communication technologies). The impacts perceived and their associated characteristics were also assessed. This easy-to-use questionnaire demonstrated good internal consistency and acceptable content validity. However, its metric properties were more powerful in areas such as knowledge production, capacity building and informing policy and practice, in which the researchers had a degree of control and influence. In general, the research projects represented an stimulus for the production of knowledge and the development of research skills. Behavioural aspects such as engagement with potential users or mission-oriented projects (targeted to practical applications) were associated with higher social benefits. Considering the difficulties in assessing a wide array of research topics, and potential differences in the understanding of the concept of ‘research impact’, an analysis of the context can help to focus on research needs. Analyzing the metric properties of questionnaires can open up new possibilities for validating instruments used to measure research impact. Further to the methodological utility of the current exercise, we see a practical applicability to specific contexts where multiple discipline research impact is requires.

A proposed framework and tool for non-economic research impact measurement

ABSTRACTResearch impact features heavily in debates about ‘the measured university’ and is now formally assessed by governments in the UK and Australia. Yet clear guidance on how impact can be measured in non-monetary ways is often lacking because of confused thinking and the context-specific nature of outcomes. To help resolve this, we first propose a general impact model of inputs, processes, outputs and outcomes framework. Then, using a survey approach of research users, we measure outcomes at different levels of abstraction using usefulness as the central construct and impact categories from the European (EU) and Association to Advance Collegiate Schools of Business (AACSB). The survey measures are simple, comparable between different impact cases, cost-effective, externally verifiable, and easily administered by those for whom impact measurement is new and puzzling. They can also be combined to form an impact index to address criticism that there is little standardisation in impact measurement. To improve the standardisation of context-specific measures, we suggest a common methodology for deriving these (SROI). The article discusses limitations of using surveys including administration, self-report data, and impact timescale problems, and suggests ways to reduce these. Implications for researchers, research managers, and assessors interested in measuring impact, such as who should do it and pay for it, are discussed.

Preprints in Scholarly Communication: Re-Imagining Metrics and Infrastructures

Digital scholarship and electronic publishing within scholarly communities change when metrics and open infrastructures take center stage for measuring research impact. In scholarly communication, the growth of preprint repositories as a new model of scholarly publishing over the last three decades has been one of the major developments. As it unfolds, the landscape of scholarly communication is transitioning—with much being privatized as it is made open—and turning towards alternative metrics, such as social media attention, author-level, and article-level metrics. Moreover, the granularity of evaluating research impact through new metrics and social media changes the objective standards of evaluating research performance. Using preprint repositories as a case study, this article situates them in a scholarly web, examining their salient features, benefits, and futures. Moves towards scholarly web development and publishing on the semantic and social web with open infrastructures, citations, and alternative metrics—how preprints advance building the web as data—is discussed. We determine that this will viably demonstrate new metrics and, by enhancing research publishing tools in the scholarly commons, facilitate various communities of practice. However, for preprint repositories to be sustainable, scholarly communities and funding agencies should support continued investment in open knowledge, alternative metrics development, and open infrastructures in scholarly publishing.

Does the normalized citation impact of universities profit from certain properties of their published documents – such as the number of authors and the impact factor of the publishing journals? A multilevel modeling approach

Abstract Research evaluation based on bibliometrics is prevalent in modern science. However, the usefulness of citation counts for measuring research impact has been questioned for many years. Empirical studies have demonstrated that the probability of being cited might depend on many factors that are not related to the accepted conventions of scholarly publishing. The current study investigates the relationship between the performance of universities in terms of field-normalized citation impact (NCS) and four factors (FICs) with possible influences on the citation impact of single papers: journal impact factor (JIF), number of pages, number of authors, and number of cited references. The study is based on articles and reviews published by 49 German universities in 2000, 2005 and 2010. Multilevel regression models have been estimated, since multiple levels of data have been analyzed which are on the single paper and university level. The results point to weak relationships between NCSs and number of authors, number of cited references, number of pages, and JIF. Thus, the results demonstrate that there are similar effects of all FICs on NCSs in universities with high or low NCSs. Although other studies revealed that FICs might be effective on the single paper level, the results of this study demonstrate that they are not effective on the aggregated level (i.e., on the institutional NCSs level).

Why We Need Better Measures Of Research Impact in Advertising

When it comes to measuring and assessing the impact and effectiveness of advertising research, we often see “who's who” reports of top-performing authors published in premier advertising journals as evidence of research impact. These data cannot tell us what influences advertising research has

OpenUP Measuring Research Impact: Concepts, Methods, Limitations and Solutions

Watch the VIDEO.The immediate Return on Investment (ROI) for basic scientific research is scientific impact – improvements in knowledge of our physical, biological and social world. The Return on Investment (ROI) for applied research and the long-term Return on Investment for basic research is societal impact (i.e. impact on health, environment, the economy, etc.). However, many of these impacts are hard to measure and may only be apparent decades after the original investment. This creates demand from funders and policy makers for metrics that predict impacts before they can be measured. Here we define a preliminary conceptual framework describing the chain of events leading from the outputs of basic research (publications, data, software, cell lines, equipment, methodologies, theories etc.) to the outputs of applied research (products, treatments, technology components etc.) to societal, financial, health and environmental impact. We go on to discuss how these impacts are currently measured in the short term (days and weeks), the medium term (years) and the long term (decades), and to identify the main providers of impact metrics. We highlight the different ways in which research metrics are used by different categories of user (researchers, institutions, national and European policymakers). Finally, we discuss the limitations of current metrics and possible solutions.

Use and Application of Federal Advisory Committee Act (FACA) Database

University experts can offer uniquely valuable insights for informing policy based on expertise they develop through research. The application of knowledge through public service is an important and understudied mechanism for translating academic expertise to government and other communities. Today universities encourage researchers to engage in public service, and often they actively provide institutional support to create a culture and environment where such pro bono work is regarded as an important activity by the research community. Yet the question remains as to whether or not a systematic mechanism exists to track, record, and measure the value of university expertise influencing policy within the context of research. We explore a useful but underutilized administrative data source, the Federal Advisory Committee Act (FACA) database, with an eye towards linking the federal service data to other sources in order to measure research impact in a sociopolitical setting. This publicly available dataset contains rich information on federal advisory committees that play an important role in shaping national programs and policies. Each year an average of 900 advisory committees with more than 60,000 members have provided either policy or grant review advice in 40 different issue areas. Our exploratory findings suggest a steady increase of academics in federal service, the different level of federal service contribution by universities, and the association between federal service and university R&D spending. We also discuss the importance of data cleaning when using administrative data for research and data linkage methods when linking federal service data to university research spending records.

Implementing a protocol for a research impact assessment of the Centre for Research Excellence in Stroke Rehabilitation and Brain Recovery

BackgroundThere is growing recognition that the wider benefits of research (economic, social and health impacts) should be assessed and valued alongside traditional research performance metrics such as peer-reviewed papers. Translation of findings into policy and practice needs to accelerate and pathways to impact need to be better understood. This research protocol outlines a mixed methods study to apply the Framework to Assess the Impact from Translational health research (FAIT) to the Centre for Research Excellence in Stroke Rehabilitation and Brain Recovery (CRE-Stroke). FAIT is purpose-designed to encourage research translation and assess research impact but lacks validation.Methods/DesignPhase 1 involves application of the FAIT-modified programme logic model to each CRE-Stroke research stream including identifying process, output and impact metrics, as well as end users of the research. A scoping review will inform potential impacts anticipated from CRE-Stroke. In Phase 2, audit and feedback on achievements against plans will track and encourage research translation. Logic models will be updated to account for changes in the research pathways over time. In Phase 3, three proven methods for measuring research impact – Payback, economic assessment and narratives – will be applied to each research stream and the data triangulated and reported in Phase 4. The feasibility of applying FAIT will also be assessed as part of Phase 3.DiscussionUse of prospective, comprehensive research impact frameworks for large interdisciplinary programmes of research is rare. FAIT’s application to CRE-Stroke will provide opportunity for the impact of CRE-Stroke to be assessed and a range of impacts beyond standard academic achievements to be reliably reported. The feasibility of FAIT’s application will also be assessed and, if necessary, refined. The usefulness of FAIT for encouraging research translation will also be described and may prove useful for other programmes looking to implement a research impact framework.

Technology Enhanced Learning, Research Impact and Open Scholarship

This guest editorial explores the potential impact of alternative metrics and social research networks to enhance Technology Enhanced Learning (TEL) research and practice, particularly in the realm of open scholarship. Conventional measures of research impact are based upon a publications’ impact factor. Article or author-level metrics can provide a better view of the contribution to the field of research of a specific research project. However, metrics are based upon citation counts that can take years to accumulate. This can result in a significant lag between the original research, communicating research outcomes to the wider research community, and research impact. TEL researchers are in a unique position to spearhead the growing movement towards open social scholarship and alternative metrics. Alternative metrics provide a powerful addition to conventional measures of research impact, facilitating a dynamic conversation around TEL research. Social media conversations provide the foundation for a collaborative open Scholarship of Technology Enhanced Learning as both an enhancement and alternative to conventional modes of research impact measurement. This guest editorial outlines key tools that are currently used to measure impact. We argue that a culture of open scholarship enhanced via social media and measured via alternative metrics creates an effective feedback loop between research and practice.

Measuring research impact: a large cancer research funding programme in Australia

BackgroundMeasuring research impact is of critical interest to philanthropic and government funding agencies interested in ensuring that the research they fund is both scientifically excellent and has meaningful impact into health and other outcomes. The Beat Cancer Project (BCP) is a AUD $34 m cancer research funding scheme that commenced in 2011. It was initiated by an Australian charity (Cancer Council SA), and supported by the South Australian Government and the state’s major universities.MethodsThis study applied Buxton and Hanney’s Payback Framework to assess research impact generated from the BCP after 3 years of funding. Data sources were an audit of peer-reviewed publications from January 2011 to September 2014 from Web of Knowledge and a self-report survey of investigators awarded BCP research funding during its first 3 years of implementation (2011–2013). Of the 104 surveys, 92 (88%) were completed.ResultsThe BCP performed well across all five categories of the Payback Framework. In terms of knowledge production, 1257 peer-reviewed publications were generated and the mean impact factor of publishing journals increased annually. There were many benefits to future research with 21 respondents (23%) reporting career advancement, and 110 higher degrees obtained or expected (including 84 PhDs). Overall, 52% of funded projects generated tools for future research. The funded research attracted substantial further income yielding a very high rate of leverage. For every AUD $1 that the cancer charity invested, the BCP gained an additional AUD $6.06. Five projects (5%) had informed policy and 5 (5%) informed product development, with an additional 31 (34%) and 35 (38%) projects, respectively, anticipating doing so. In terms of health and sector and broader economic benefits, 8 (9%) projects had influenced practice or behaviour of health staff and 32 (34%) would reportedly to do so in the future.ConclusionsResearch impact was a priority of charity and government funders and led to a deliberate funding strategy. Emphasising research impact while maintaining rigorous, competitive processes can achieve the joint objectives of excellence in research, yielding good research impact and a high rate of leverage for philanthropic and public investment, as indicated by these early results.

Measuring the impact of pharmacoepidemiologic research using altmetrics: A case study of a CNODES drug-safety article.

PurposeTo provide an overview of altmetrics, including their potential benefits and limitations, how they may be obtained, and their role in assessing pharmacoepidemiologic research impact.MethodsOur review was informed by compiling relevant literature identified through searching multiple health research databases (PubMed, Embase, and CIHNAHL) and grey literature sources (websites, blogs, and reports). We demonstrate how pharmacoepidemiologists, in particular, may use altmetrics to understand scholarly impact and knowledge translation by providing a case study of a drug‐safety study conducted by the Canadian Network of Observational Drug Effect Studies.ResultsA common approach to measuring research impact is the use of citation‐based metrics, such as an article's citation count or a journal's impact factor. “Alternative” metrics, or altmetrics, are increasingly supported as a complementary measure of research uptake in the age of social media. Altmetrics are nontraditional indicators that capture a diverse set of traceable, online research‐related artifacts including peer‐reviewed publications and other research outputs (software, datasets, blogs, videos, posters, policy documents, presentations, social media posts, wiki entries, etc).ConclusionCompared with traditional citation‐based metrics, altmetrics take a more holistic view of research impact, attempting to capture the activity and engagement of both scholarly and nonscholarly communities. Despite the limited theoretical underpinnings, possible commercial influence, potential for gaming and manipulation, and numerous data quality‐related issues, altmetrics are promising as a supplement to more traditional citation‐based metrics because they can ingest and process a larger set of data points related to the flow and reach of scholarly communication from an expanded pool of stakeholders. Unlike citation‐based metrics, altmetrics are not inherently rooted in the research publication process, which includes peer review; it is unclear to what extent they should be used for research evaluation.

Measuring research impact in medical research institutes: a qualitative study of the attitudes and opinions of Australian medical research institutes towards research impact assessment frameworks

BackgroundThe question of how to measure, assess and optimise the returns from investment in health and medical research (HMR) is a highly policy-relevant issue. Research Impact Assessment Frameworks (RIAFs) provide a conceptual measurement framework to assess the impact from HMR. The aims of this study were (1) to elicit the views of Medical Research Institutes (MRIs) regarding objectives, definitions, methods, barriers, potential scope and attitudes towards RIAFs, and (2) to investigate whether an assessment framework should represent a retrospective reflection of research impact or a prospective approach integrated into the research process. The wider objective was to inform the development of a draft RIAF for Australia’s MRIs.MethodsPurposive sampling to derive a heterogeneous sample of Australian MRIs was used alongside semi-structured interviews with senior executives responsible for research translation or senior researchers affected by research impact initiatives. Thematic analysis of the interview transcriptions using the framework approach was then performed.ResultsInterviews were conducted with senior representatives from 15 MRIs. Participants understood the need for greater research translation/impact, but varied in their comprehension and implementation of RIAFs. Common concerns included the time lag to the generation of societal impacts from basic or discovery science, and whether impact reflected a narrow commercialisation agenda. Broad support emerged for the use of metrics, case study and economic methods. Support was also provided for the rationale of both standardised and customised metrics. Engendering cultural change in the approach to research translation was acknowledged as both a barrier to greater impact and a critical objective for the assessment process. Participants perceived that the existing research environment incentivised the generation of academic publications and track records, and often conflicted with the generation of wider impacts. The potential to improve the speed of translation through prospective implementation of impact assessment was supported, albeit that the mechanism required development.ConclusionThe study found that the issues raised regarding research impact assessment are less about methods and metrics, and more about the research activities that the measurement of research translation and impact may or may not incentivise. Consequently, if impact assessment is to contribute to optimisation of the health gains from the public, corporate and philanthropic investment entrusted to the institutes, then further inquiry into how the assessment process may re-align research behaviour must be prioritised.

Awareness of Altmetrics among LIS Scholars and Faculty

Altmetrics track the attention paid to scholarship via mentions in social media, the press, and other non-traditional venues. For library and information science (LIS) faculty, altmetrics are also a new and important area for research and teaching. We conducted a survey of LIS faculty teaching in US and Canadian graduate LIS programs accredited by the American Library Association in which we asked about their familiarity with and awareness of measures of research impact, including altmetrics. Our results indicate that while most LIS faculty in our sample had some awareness of altmetrics, they reported greater familiarity with traditional measures of research impact such as citation counts and usage statistics. We also confirmed that, among our sample, there was a relationship between years of teaching experience and awareness of altmetrics, as well as among familiarity with altmetrics, familiarity with citation counts, and familiarity with usage statistics. Among the robust, global body of research related to the use of new measures of research impact among scientists and scholars, there are few studies that use survey methods and focus on faculty scholars within a specific discipline. The results of this study contribute new knowledge to the existing body of research on altmetrics and may contribute to the development of LIS graduate curricula devoted to measures of research impact and their application in practice.

Measuring Research Impact of Astronomers/Astrophysicists by using Astrophysics Data System Beta: A Powerful New Interface: A case study with Special Reference to Prof. Jayant V. Narlikar

This paper highlights qualitatively and quantitatively research and presents a valuable overview of new citation enhanced databases in the context of research evaluation for the productivity of Prof. Jayant V. Narlikar. He has total 472 research publications in the fields of Astronomy and Astrophysics (AA) published from 1961-2015. In addition, this paper provides an overview of the citation-enhanced databases viz Astrophysics Data System Beta: A powerful new interface for performing citation analysis. Previously, scientometrics had been used to measure the publication productivity of Prof. Jayant V. Narlikar using the Astrophysics Data System (ADS). The scope of this paper is limited to Astrophysics Data System Beta. The result indicates that most of his papers are published in peer reviewed journals having the highest Impact Factor. The average number of publications per year is 8.74 and with the maximum papers published during 1981-1990 and 1996-2000. The total number of citations for his publications is 3516 covering 665 of his papers published during 1961-1970. He had many research collaborations, specifically with Prof. F. Hoyle (87 papers), Prof. G. Burbidge (38 papers), Prof. N. Wickramasinghe (22 papers) and Prof. T. Padmanabhan (21 papers).

Rethinking policy \u2018impact\u2019: four models of research-policy relations

Political scientists are increasingly exhorted to ensure their research has policy ‘impact’, most notably via Research Excellence Framework (REF) impact case studies, and ‘pathways to impact’ statements in UK Research Council funding applications. Yet the assumptions underpinning these frameworks often fail to reflect available evidence and theories. Notions of ‘impact’, ‘engagement’ and ‘knowledge exchange’ are typically premised on simplistic, linear models of the policy process, according to which policy-makers are keen to ‘utilise’ expertise to produce more ‘effective’ policies. Such accounts overlook the rich body of literature in political science, policy studies, and sociology of knowledge, which offer more complex and nuanced accounts. Drawing on this wider literature, this paper sets out four different approaches to theorising the relationship: (1) knowledge shapes policy; (2) politics shapes knowledge; (3) co-production; and (4) autonomous spheres. We consider what each of these four approaches suggests about approaches to incentivising and measuring research impact.