Research Impact Assessment Research Articles

Challenges with measures used for assessing research impact in higher education institutions

Internationally, there has been a push for the prioritisation of research impact beyond its scholarly contribution. Traditionally, research impact assessments have focused on academic impact and quantitative measures, at the expense of researchers for whom research impact cannot be quantified. Bibliometric indicators and other quantitative measures are still the most widely used method for evaluating research impact because these measures are easy to use and provide a quick solution for evaluators. Conversely, metric indicators fail to capture important dimensions of high-quality research. Hence, in this study, we explored challenges with metric indicators. We adopted a case study of the University of Cape Town and used document analysis, a questionnaire survey to collect data from academics and researchers, as well as semi-structured interviews with a sample of academic and research staff. The findings highlight common challenges with quantitative measures, such as bias and discipline coverage, and the ability of measures to drive researchers’ behaviour in another direction. We propose the adoption of responsible research metrics and assessment in South African higher education institutions for more inclusive and equitable research impact assessments.

Research Impact Assessment on Sustainable Development Goals in Higher Education Institutions: A Qualitative Study

Research Impact Assessment on Sustainable Development Goals in Higher Education Institutions: A Qualitative Study

Research impact assessment: Developing and applying a viable model for the social sciences

Abstract Prompted by research funding agencies and regulatory bodies, universities need to demonstrate and manage the impact of their research. Therefore, models for evaluating research impacts that can be applied quickly and at a reasonable cost are needed. However, the development of these models presents considerable challenges: research impact is multidimensional and can occur over time because of multiple interactions between different agents. Furthermore, impact assessment can demand substantial time and resources. Considering this challenging context, this study proposes a research impact assessment model capable of optimizing benefits and costs. The model was based on a multidimensional impact perspective and an application-oriented way of generating and transmitting knowledge. It was applied to a large teaching and research institution with a predominant role in the social sciences. The assessment involved a survey of the leaders of 133 research projects. The main findings of the application of the assessment model were as follows: first, knowledge production is transitioning from a traditional mode (aimed to impact science) to a new mode (aimed to impact multiple stakeholders); second, project leaders still perceive the prevalence of scientific impact over other types of impact; and third, the survey revealed how certain characteristics of the knowledge production mode relate to (perceived) impact. The viable research impact assessment model was demonstrated to be useful and cost-effective; thus, with adaptations and improvements, it may be used by other institutions, particularly those with internal research funding programs. This article also presents the model’s limitations, suggesting directions for future research.

Abstract PO5-25-07: A research impact assessment method to evaluate Susan G. Komen’s research funding impact on drugs approved for breast cancer treatment

Abstract Susan G. Komen (Komen) is the largest private funder of breast cancer research, having invested nearly $1.1 billion in cutting edge breast cancer research with a focus on conquering deadly metastatic and aggressive breast cancers, advancing precision medicine, and addressing disparities in breast cancer care and outcomes. There is a growing demand on Komen and other research funders from patients, donors, and other constituents to understand how investments in biomedical research translate into clinical impact that is meaningful to patients, the public, and others beyond academic publications and other bibliometrics. Komen uses a variety of approaches to evaluate the impact of our research investments, including a recently developed method to determine how Komen’s funding has impacted discovery and development of breast cancer drugs approved by the FDA from 2012-2022. Utilizing bibliometric analysis derived from the Dimensions for Funders platform, we systematically identified links between Komen-funded research, Komen-funded researchers, and FDA approved drugs. First, we identified fundamental clinical trials that led to initial and expanded FDA approval of each drug from prescribing information labels and press releases. Using API technology, we then performed a bibliometric linkage analysis and analyzed citation results from fundamental clinical trial studies, working backwards to capture fundamental pre-clinical and basic research studies that identified each drug’s target and its functions in the context of breast cancer biology. After collecting and curating publications in an unbiased manner, we evaluated each published study and contributing authors for Komen support. We identified specific “touchpoints” for fundamental studies including direct Komen funding for the published study and/or Komen support of key personnel. Early results revealed direct and indirect Komen research touchpoints across the research pipeline for several drugs, including direct touchpoints for basic, pre-clinical, and clinical trial studies that led to FDA approval of the PI3K inhibitor alpelisib, basic research and pre-clinical studies that led to FDA approval of CDK4/6 inhibitors, and basic research studies that led to FDA approval of PARP inhibitors olaparib and talazoparib. By adopting this approach, we can demonstrate the impact of Komen’s research investment across the full development timeline for each drug. This will allow us to measure and communicate how Komen’s investment in innovative research from discovery to clinical development is driving advances in breast cancer clinical care that improves outcomes. Citation Format: Dana Brantley-Sieders, Lauren Leslie, Amy Dworkin, Kimberly Sabelko, Kari Wojtanik. A research impact assessment method to evaluate Susan G. Komen’s research funding impact on drugs approved for breast cancer treatment [abstract]. In: Proceedings of the 2023 San Antonio Breast Cancer Symposium; 2023 Dec 5-9; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2024;84(9 Suppl):Abstract nr PO5-25-07.

The Conceptualization and Measurement of Research Impact in Primary Health Care: Protocol for a Rapid Scoping Review.

The generation of research evidence and knowledge in primary health care (PHC) is crucial for informing the development and implementation of interventions and innovations and driving health policy, health service improvements, and potential societal changes. PHC research has broad effects on patients, practices, services, population health, community, and policy formulation. The in-depth exploration of the definition and measures of research impact within PHC is essential for broadening our understanding of research impact in the discipline and how it compares to other health services research. The objectives of the study are (1) to understand the conceptualizations and measures of research impact within the realm of PHC and (2) to identify methodological frameworks for evaluation and research impact and the benefits and challenges of using these approaches. The forthcoming review seeks to guide future research endeavors and enhance methodologies used in assessing research impact within PHC. The protocol outlines the rapid review and environmental scan approach that will be used to explore research impact in PHC and will be guided by established frameworks such as the Canadian Academy of Health Sciences Impact Framework and the Canadian Health Services and Policy Research Alliance. The rapid review follows scoping review guidelines (PRISMA-ScR; Preferred Reporting Items for Systematic Review and Meta-Analysis Extension for Scoping Reviews). The environmental scan will be done by consulting with professional organizations, academic institutions, information science, and PHC experts. The search strategy will involve multiple databases, citation and forward citation searching, and manual searches of gray literature databases, think tank websites, and relevant catalogs. We will include gray and scientific literature focusing explicitly on research impact in PHC from high-income countries using the World Bank classification. Publications published in English from 1978 will be considered. The collected papers will undergo a 2-stage independent review process based on predetermined inclusion criteria. The research team will extract data from selected studies based on the research questions and the CRISP (Consensus Reporting Items for Studies in Primary Care) protocol statement. The team will discuss the extracted data, enabling the identification and categorization of key themes regarding research impact conceptualization and measurement in PHC. The narrative synthesis will evolve iteratively based on the identified literature. The results of this study are expected at the end of 2024. The forthcoming review will explore the conceptualization and measurement of research impact in PHC. The synthesis will offer crucial insights that will guide subsequent research, emphasizing the need for a standardized approach that incorporates diverse perspectives to comprehensively gauge the true impact of PHC research. Furthermore, trends and gaps in current methodologies will set the stage for future studies aimed at enhancing our understanding and measurement of research impact in PHC. PRR1-10.2196/55860.

Establishing research impact assessment in Iran: The first report from a non-high-income country.

This study presents the first report on research impact assessment (RIA) in non-high-income countries, undertaken as a pilot initiative in 2021. Within it, we aimed to explore the feasibility of employing the 'payback' model for evaluating the impact of health research and enhancing the accountability of universities. We focussed on three key impact domains: 'production of decision support documents and knowledge-based products,' 'implementation of research results,' and 'health and economic impact.' We adopted a case study approach to assess the impact of 5334 health research projects conducted by researchers from 18 universities from 2018 to 2020. Researchers were required to submit evidence related to at least one of the specified impact domains; six scientific committees verified and scored claimed impacts at the national level. Only 25% of the assessed projects achieved impact in at least one domain, with the production of decision support documents and knowledge products being the most reported impact. Notably, economic impact was verified in only three projects, indicating room for improvement in this area. Technology research exhibited the highest acceptance rate of claimed impact, suggesting a positive correlation between technology-focused projects and impactful outcomes. This study demonstrates the feasibility of employing a case study approach and the 'payback' model to evaluate the impact of health research, even within the constraints of a moderately equipped research infrastructure. These findings underscore the potential of integrating RIA into the governance of health research in Iran and other non-high-income countries, as well as the importance of using RIA to assess the accountability of health research systems, guide the allocation of research funding, and advocate for the advancement of health research. The study sets a precedent for future assessments in similar contexts and contributes to the ongoing global dialogue on the societal impact of health research.

Metrics in research impact assessment and grant funding: Insights from researchers in the “Reviewer 2 Must Be Stopped!” Facebook group

Metrics in research impact assessment and grant funding: Insights from researchers in the “Reviewer 2 Must Be Stopped!” Facebook group

Participatory ex-ante impact assessment for interactive research and development in agriculture and food systems

ABSTRACT Drawing upon literature from both impact assessment of development projects and agricultural research, the aim of this article is to analyse the pilot testing of a new multi-dimensional assessment framework for defining and evaluating the societal impact of agricultural research and corresponding education. The research approach involves an action research effort of pilot testing in three case studies from three different countries. The framework assumes a systems view, understands impact as socially embedded, and adopts the concepts of contribution and productive interactions rather than cause-and-effect attribution. The emerging experiences include developing both project-specific and universal indicators; gauging impacts at different levels and sustainability dimensions; handling the issues of attribution and time frame; and the role of stakeholder involvement. The results have the potential to support the development of a new role for impact assessment, by enabling principal actors in research and higher education institutions to take responsibility for contributing to concrete and demonstrable sustainability changes in society.

Atmospheric impacts and regulation framework of shipping emissions: achievements, challenges and frontiers

Currently, over 80% of the international trade volume is carried by sea. Marked by persistent growth, evident atmospheric impacts, intricate mitigation challenges, international shipping has been recognized as one of the most “hard-to-abate” sectors gathering increasing attention from both academic community and governmental sectors in recent years. Against the backdrop of the ambitious climate and clean air objectives, the quantitative shipping emission characterization, impact assessment and policy effectiveness research are not only fundamental to understand the status quo and ramifications of shipping emissions but also beneficial for future emission regulations. Here, we summarized the achievements in shipping emission modelling and impact research in the past two decades, and identified the challenges lying in the transition pathway towards a clean and carbon-neutral shipping. To address the pressing demand for this, we proposed an innovative framework which aims to facilitate emission abatement. Finally, promising directions for future work were delineated, including the indirect effects of shipping emitted aerosols on the climate, the emissions and impacts of novel contaminants, synergies and conflicts among different emission reduction measures, projections on future shipping emission inventories, Arctic shipping emissions, etc.

Unleashing the power of AI: a systematic review of cutting-edge techniques in AI-enhanced scientometrics, webometrics and bibliometrics

PurposeThe study aims to analyze the synergy of artificial intelligence (AI), with scientometrics, webometrics and bibliometrics to unlock and to emphasize the potential of the applications and benefits of AI algorithms in these fields.Design/methodology/approachBy conducting a systematic literature review, our aim is to explore the potential of AI in revolutionizing the methods used to measure and analyze scholarly communication, identify emerging research trends and evaluate the impact of scientific publications. To achieve this, we implemented a comprehensive search strategy across reputable databases such as ProQuest, IEEE Explore, EBSCO, Web of Science and Scopus. Our search encompassed articles published from January 1, 2000, to September 2022, resulting in a thorough review of 61 relevant articles.Findings(1) Regarding scientometrics, the application of AI yields various distinct advantages, such as conducting analyses of publications, citations, research impact prediction, collaboration, research trend analysis and knowledge mapping, in a more objective and reliable framework. (2) In terms of webometrics, AI algorithms are able to enhance web crawling and data collection, web link analysis, web content analysis, social media analysis, web impact analysis and recommender systems. (3) Moreover, automation of data collection, analysis of citations, disambiguation of authors, analysis of co-authorship networks, assessment of research impact, text mining and recommender systems are considered as the potential of AI integration in the field of bibliometrics.Originality/valueThis study covers the particularly new benefits and potential of AI-enhanced scientometrics, webometrics and bibliometrics to highlight the significant prospects of the synergy of this integration through AI.

Effective mission-oriented research: A new framework for systemic research impact assessment

Abstract Mission-oriented research combines a wide array of natural and social science disciplines to offer solutions for complex and multi-dimensional challenges such as climate change, loss of biodiversity, and scarcity of natural resources. The utilization of the outputs of mission-oriented research aims for changes in behavior, policy and practice resulting in real world impacts. Systematically assessing such research impacts and impact-generating processes is novel and offers great potential to plan for impactful research. This article develops a framework for systemic research impact assessment (RIA) on the basis of a literature review taking natural resource management (NRM) research as an example. The review compiles and analyzes 70 relevant RIA approaches. The resulting framework combines four components for improving societal impacts (1) an integrated component enabling reflection of impacts on all sustainability dimensions, (2) a missions component orienting toward societal goals to ensure societal relevance, (3) an inclusive component enabling wide participation to ensure legitimacy of research and its impact, and (4) a strategic component to choose appropriate assessment scales and time dimensions to ensure effectiveness. We provide suitable examples for the framework and we conclude with a call for an increased use of systemic and formative RIA that incorporate participatory strategies for research priority setting as well as socially deliberated target systems (e.g. SDGs), to plan for impactful mission-oriented research.

Utilizing a composite citation index for evaluating clinical ophthalmology research: insights into gender, nationality, and self-citation among top ophthalmology researchers.

To compare the performance of a composite citation score (c-score) and its six constituent citation indices, including H-index, in predicting winners of the Weisenfeld Award in ophthalmologic research. Secondary objectives were to explore career and demographic characteristics of the most highly cited researchers in ophthalmology. A publicly available database was accessed to compile a set of top researchers in the field of clinical ophthalmology and optometry based on Scopus data from 1996 to 2021. Each citation index was used to construct a multivariable model adjusted for author demographic characteristics. Using area under the receiver operating curve (AUC) analysis, each index's model was evaluated for its ability to predict winners of the Weisenfeld Award in Ophthalmology, a research distinction presented by the Association for Research in Vision and Ophthalmology (ARVO). Secondary analyses investigated authors' self-citation rates, career length, gender, and country affiliation over time. Approximately one thousand unique authors publishing primarily in clinical ophthalmology/optometry were analyzed. The c-score outperformed all other citation indices at predicting Weisenfeld Awardees, with an AUC of 0.99 (95% CI: 0.97-1.0). The H-index had an AUC of 0.89 (95% CI: 0.83-0.96). Authors with higher c-scores tended to have longer career lengths and similar self-citation rates compared to other authors. Sixteen percent of authors in the database were identified as female, and 64% were affiliated with the United States of America. The c-score is an effective metric for assessing research impact in ophthalmology, as seen through its ability to predict Weisenfeld Awardees.

Impact through research in education and studies in human society: A review of Australian Research Council 'high-for-impact' case studies.

Research impact is an important measure of the effective transmission and ongoing contribution of research beyond the scope of initial research publication outputs; however, determining what constitutes 'high-for-impact' research can be difficult for specific fields of study. This review of the Australian Research Council's Engagement and Impact Assessment 2018 analyses high-for-impact case studies submitted in the fields of Education (n = 17) and Studies in Human Society (n = 11) with the aim of understanding and explicating how high impact research has been evidenced in these fields. The review was guided by three research questions that concern the identification of the key characteristics of high-for-impact case studies, their reported impacts, and the evidence researchers cite to support claims of impact. The review highlights an important limitation in how impact is defined and understood by researchers, particularly cultural and social impact. Half of the analysed case studies involved international engagement, with minimal partner collaboration in the global south and countries in the Indo-Pacific, despite the region's strategic geo-political importance for Australia. Our findings draw into question the distribution of funding to universities and where investment might best be made for the highest potential return on research impact. Another key finding is that reported impacts across the domains of economy, society, culture, national security, public service, health, environment and quality of life offer little satisfactory evidence of impact, despite affording valuable insights into the nature of impact claimed. Accordingly, we conclude that to enhance the value of research and demonstrate impact in Education and Social Sciences, improved impact literacy is required among researchers. We assert that a better understanding of what constitutes impact and how it can be evidenced will support more impactful research designs. Wider adoption of the holistic anthropological definition of culture, which integrates values, practices and products, would enhance impact case studies by expanding their focus to include the broader cultural changes that underpin sustained social change. While the ARC engagement and impact agenda is a step in the right direction, improving the value of research for society will require a radical reconceptualisation of research and its funding, well beyond the current assessment framework. The Lowitja Institute's research-for-impact framework [1] is proposed as an alternative approach to research priority-setting based on explicit evidence gap analysis.

Prioritising and incentivising productivity within indicator-based approaches to Research Impact Assessment: a commentary

Research Impact Assessment (RIA) represents one of a suite of policies intended to improve the impact generated from investment in health and medical research (HMR). Positivist indicator-based approaches to RIA are widely implemented but increasingly criticised as theoretically problematic, unfair, and burdensome. This commentary proposes there are useful outcomes that emerge from the process of applying an indicator-based RIA framework, separate from those encapsulated in the metrics themselves. The aim for this commentary is to demonstrate how the act of conducting an indicator-based approach to RIA can serve to optimise the productive gains from the investment in HMR. Prior research found that the issues regarding RIA are less about the choice of indicators/metrics, and more about the discussions prompted and activities incentivised by the process. This insight provides an opportunity to utilise indicator-based methods to purposely optimise the research impact. An indicator-based RIA framework specifically designed to optimise research impacts should: focus on researchers and the research process, rather than institution-level measures; utilise a project level unit of analysis that provides control to researchers and supports collaboration and accountability; provide for prospective implementation of RIA and the prospective orientation of research; establish a line of sight to the ultimate anticipated beneficiaries and impacts; Include process metrics/indicators to acknowledge interim steps on the pathway to final impacts; integrate ‘next’ users and prioritise the utilisation of research outputs as a critical measure; Integrate and align the incentives for researchers/research projects arising from RIA, with those existing within the prevailing research system; integrate with existing peer-review processes; and, adopt a system-wide approach where incremental improvements in the probability of translation from individual research projects, yields higher impact across the whole funding portfolio.Optimisation of the impacts from HMR investment represents the primary purpose of Research Impact policy. The process of conducting an indicator-based approach to RIA, which engages the researcher during the inception and planning phase, can directly contribute to this goal through improvements in the probability that an individual project will generate interim impacts. The research project funding process represents a promising forum to integrate this approach within the existing research system.

Does Altmetric Attention Score Correlate with Citations of Articles Published in High CiteScore Journals

In the ever-evolving landscape of scholarly communication, research impact assessment has expanded beyond traditional citation metrics. Altmetrics, which encompass a wide range of online indicators like social media mentions, downloads, and blog posts, provide insights into the broader societal engagement with scholarly works. This study explores the relationship between Altmetric Attention Scores (AAS) and traditional citation metrics within the Library and Information Sciences (LIS) domain, focusing on high CiteScore open-access journals. The study collected bibliographic data of top 100 articles from Dimensions.ai database that were published between 2013 and 2022 and assesses the alignment between scholarly recognition and online engagement. The top 100 articles were selected based on highest AAS to focus on the most influential and widely-discussed research in order to provide a comprehensive overview of current trends and impactful findings in the field. The dissemination pattern of research highlights across various social web platforms is examined, shedding light on the diverse channels through which research reaches audiences. Correlation analysis reveals a weak and statistically insignificant connection between AAS and citations for the studied journals, emphasising the nuanced nature of scholarly impact in today’s digital landscape. The findings underscore the need to consider both traditional and alternative metrics when evaluating research influence and engagement in the evolving scholarly communication paradigm of the digital age. While this study is confined to specific journals within the LIS domain, its insights contribute to a deeper understanding of the multifaceted nature of research impact assessment.

Article Citation Tracking Exercise: An Activity in Assessing Research Impact and the Interdisciplinary Nature of Chemistry

Article Citation Tracking Exercise: An Activity in Assessing Research Impact and the Interdisciplinary Nature of Chemistry

Understanding the impact of environmental impact assessment research on policy and practice

There is an enormous and ever-growing body of environmental impact assessment (EIA) research, much of which is grounded in practice or seeks to advance it. In this paper we show how the impact of EIA research on policy and practice might be conceptualised and how to set about evidencing it. A framework is developed through literature review to account for impact in four areas pertaining to instrumental impact, conceptual impact, capacity building and knowledge brokerage and co-production. Methods for implementing the framework include citations within policy documents along with content analysis to determine influence and interviews or surveys with policy makers and practitioners; all subsequently presented as narratives. We provide examples and further discussion of each, drawn from recent analysis of the EIA research of one of the authors. Whilst proving impact of EIA research on policy and practice is challenging, we found the framework to be a useful way for structuring and guiding such an investigation. This approach to understanding the uptake and influence of EIA research on impact assessment practitioners and other stakeholders (i.e., government regulators/policy-makers, consultants, proponents or NGOs) could be applied by many professionals in the field to showcase positive impact on policy and practice locally, nationally and internationally. It may also usefully serve academic EIA researchers applying for new positions or for promotion within universities.

Co-creation and societal impact: Toward a generic framework for research impact assessment

Academia focuses on the interplay of Higher Education Institutions and external stakeholders. In this context, academia is concerned with the societal impact and impact created in interactions with external stakeholders; the latter is often referred to as impact co-creation. There is agreement that the related processes leading to an impact are complex and multi-dimensional. However, academics disagree on how the ultimate, wider impact of research should be measured. This study seeks to conceptualize societal impact through the lens of value co-creation, arguing that societal impact is best conceptualized as the uptake of research. Based on this, we developed a generic research impact assessment framework to facilitate evaluations and enable cross-sector learning. This study contributes to academia by providing an overarching understanding of impact creation, including wider research impact, and offers the perspective that any research project involving stakeholders to a certain extent, also entails co-production.

The post-award effort of managing and reporting on funded research: a scoping review

Introduction: Reporting is a mechanism for funding organisations to monitor and manage the progress, outputs, outcomes, and impacts of the research they fund. Inconsistent approaches to reporting and post-award management, and a growing demand for research information, can lead to perception of unnecessary administrative effort that impacts on decision-making and research activity. Identifying this effort, and what stakeholders see as unmet need for improvement, is crucial if funders and Higher Education Institutions (HEIs) are to streamline their practices and provide better support with reporting activities. In this review, we summarise the processes in post-award management, compare current practices, and explore the purpose of collecting information on funded research. We also identify areas where unnecessary effort is perceived and improvement is needed, using previously reported solutions to inform recommendations for funders and HEIs. Methods: We conducted a scoping review of the relevant research and grey literature. Electronic searches of databases, and manual searches of journals and funder websites, resulted in inclusion of 52 records and 11 websites. Information on HEI and funder post-award management processes was extracted, catalogued, and summarised to inform discussion. Results: Post-award management is a complex process that serves many purposes but requires considerable effort, particularly in the set up and reporting of research. Perceptions of unnecessary effort stem from inefficiencies in compliance, data management and reporting approaches, and there is evidence of needed improvement in mechanisms of administrative support, research impact assessment, monitoring, and evaluation. Solutions should focus on integrating digital systems to reduce duplication, streamlining reporting methods, and improving administrative resources in HEIs. Conclusions: Funders and HEIs should work together to support a more efficient post-award management process. The value of research information, and how it is collected and used, can be improved by aligning practices and addressing the specific issues highlighted in this review.

Literature review and analysis of the social impact of a just energy transition

IntroductionThe energy transition is now the focus of global attention. This transition will have a significant impact on the global energy system as well as the political and economic landscape. Research on the social impact of the energy transition can help us get a deeper understanding of the energy transition and propose suggestions for future development.MethodsThis paper focuses on the history and characteristics of the social impact evaluation of the energy transition, using bibliometric methods and the Web of Science Core Collection database with the help of HistCite and VOSviewer analysis tools. This paper discusses what is energy transition, just energy transition and its social impact and how to assess social impacts of just energy transition. In order to reduce the negative impacts of energy transition, interdisciplinary research, social impact research and social impact assessment of construction projects are three directions to be discussed.ResultsThe study find that existing research mainly focuses on the connotation, pathways and different types of a just energy transition. There are three hot areas of research on the impact of a just energy transition: the sustainable development of energy, political economy and society. There is a general lack of research on the social impact of a just energy transition and an even greater lack of relevant research on social impact evaluation.DiscussionBased on the global goal of just energy transition, this paper makes specific recommendations on what developed and developing countries, as well as sectors and enterprises within countries, should do.