Evaluation Of Research Impact Research Articles

Challenges with measures used for assessing research impact in higher education institutions

Internationally, there has been a push for the prioritisation of research impact beyond its scholarly contribution. Traditionally, research impact assessments have focused on academic impact and quantitative measures, at the expense of researchers for whom research impact cannot be quantified. Bibliometric indicators and other quantitative measures are still the most widely used method for evaluating research impact because these measures are easy to use and provide a quick solution for evaluators. Conversely, metric indicators fail to capture important dimensions of high-quality research. Hence, in this study, we explored challenges with metric indicators. We adopted a case study of the University of Cape Town and used document analysis, a questionnaire survey to collect data from academics and researchers, as well as semi-structured interviews with a sample of academic and research staff. The findings highlight common challenges with quantitative measures, such as bias and discipline coverage, and the ability of measures to drive researchers’ behaviour in another direction. We propose the adoption of responsible research metrics and assessment in South African higher education institutions for more inclusive and equitable research impact assessments.

277 Translating for equity impact: A community-engaged approach to integrate health equity into the Translational Science Benefits Model

OBJECTIVES/GOALS: The Translational Science Benefits Model (TSBM) offers an approach for evaluating research impact on public health and society. Since its development in 2017, there has been an increased focus on science’s impacts on equity. This poster describes efforts to integrate equity into the TSBM. METHODS/STUDY POPULATION: Adaptation of the TSBM includes 3 phases: 1) literature scan of equity impacts, 2) community engagement listening sessions, and 3) model refinement. First, we conducted a scan of the clinical and translational sciences literature for articles that measured equity impacts associated with the TSBM. From the articles, we extracted both equity considerations related to existing TSBM benefits and potential equity-focused benefits. Next, we will present the dimensions of new and existing benefits to several community member panels engaged in research and evaluation. We will use their input to prioritize and refine adaptations to the model (phase 3). RESULTS/ANTICIPATED RESULTS: Our literature scan identified equity dimensions of the original 30 TSBM benefits and 15 potential benefits for inclusion in the model, including community partnerships, community capacity building, workforce development, and social determinants of health, among others. To ensure that community priorities also inform identification of equity impacts of clinical and translational research, we will review and refine the model through a series of community engagement sessions planned for Fall 2023. The sessions will inform final adaptations, which will result in an expanded approach evaluating the impact of scientific activities to include impacts on health equity. DISCUSSION/SIGNIFICANCE: The TSBM provides a framework for clinical and translational scientists to plan for, track, and demonstrate the broader impacts of their work. Including equity impacts can help increase consideration of equity in science and help to more clearly make the link between research and equity impact.

APPLICATION OF BIBLIOMETRIC ANALYSIS ON ROOT GROWTH ALGORITHM

The root growth algorithm has often been used to solve challenging optimization issues. It is one of the metaheuristic algorithms inspired by root growth in plant behaviors. An article reviewed and analyzed bibliographic data on metaheuristics but is not specific on the topic of root growth algorithm. Therefore, this article presents a bibliometric analysis based on the topic of the root growth algorithm. It reviews the publication from the Scopus database. Based on the search process done on 14 February 2023 using the keywords of root growth algorithm, this article managed to gather 1836 articles from 1976-2023. However, this article only focuses on Engineering, Computer Science, and Mathematics to ensure it relates to the area the researcher wants to explore. Moreover, articles in the year 2023 will also be excluded due to incomplete data. After the researcher limited the data to those areas, it was reduced to 837 articles. This article uses three types of software: Microsoft Excel, VOSviewer software, and Harzing’s Publish or Perish software to analyze the frequency, visualization mapping, and citation metrics analysis, respectively. The finding from the bibliometric analysis shows the researcher has noticed that the total publication keeps increasing rapidly from 2014 until 2022. The country that contributes the most to the root growth algorithm topic is China. Furthermore, the most productive author is Chen, H. and all the top 10 productive authors are from China except for Christofides, P. D., who comes from the United States. The top 10 sources of root growth algorithm research contributed over a quarter of the total articles (231 or 27.60%). The results of this study have significant implications for increasing the number of practices using the root growth algorithm in future research. Last but not least, bibliometric analysis on the topic of root growth algorithm is needed to identify influential publications, understand research trends, evaluate research impact, and identify potential research gaps in a concise manner, which helps researchers and readers stay informed, make informed decisions, and contribute to the advancement of knowledge in the field.

Designing an Impact-Oriented Model of Research and Technology Evaluation: An Experience of I.R.Iran.

Research impact assessment is already being institutionalized in health research and innovation systems. In developing countries, there are many different research assessment models which have focused more on research output in academic levels and less on impact. The aim of this study is designing an Iranian impact-oriented model of research and technology evaluation. This is a mixed study. In the quantitative part, by reviewing the literature, a list of research impact indicators that existed were gathered, reviewed, and scored by participants on importance, relevance, and measurability via a 5-point Likert scale. All indicators with a mean score equal to or greater than 3.5 entered the qualitative part, which were discussed in depth by engaging key stakeholders regarding their validity and feasibility through focus groups, interviews, and expert panels. The Iranian research impact evaluation model was developed with four main pillars (including input and process, output, outcome, and impact), four areas (stewardship, advancing knowledge and translation, technology, and impact), and 30 indicators through key stakeholders participation in the Iranian health research system. This model has been introduced as the first model designed to evaluate the impact of health research and can be one of the most important tools for allocating limited funding resources while maximizing the desired impact of research in the community.

From ‘publish or perish’ to ‘publish for purpose’

AbstractPressures to publish prolifically have led to an abundance of research on trivial matters rather than on issues of significance. This trend is particularly pronounced in developing countries, where limited access to paid content has fueled the growth of open‐access publications. Additionally, the prevailing “publish or perish” culture has encouraged the pursuit of inconsequential and conformist research. The scholarly integrity of academic institutions is eroded when they struggle with evaluating research impact through metrics, potentially overlooking substantive contribution. The emergence of AI technology adds a fresh dimension to the issue, creating new possibilities for mass output rather than work that is innovative or informed by social values and priorities. This commentary serves as a call to action for scholars, institutions, and policymakers collectively to reshape the trajectory of academic publishing, restoring its sense of purpose through making lasting contributions to the betterment of society.

The correlation between altmetric score and traditional measures of article impact for studies published within the Surgeon Journal

Alternative metrics, or altmetrics, have emerged as a promising tool for measuring the social impact of research, which is increasingly important in today's digital and social media-driven world. Altmetric Attention Score (AAS) is a weighted count of all the online attention garnered by a study, and it is currently unclear whether a relationship with traditional bibliometrics exists. The purpose of this article was to retrospectively review articles published in the Surgeon Journal from 2003 to 2020 to compare AAS with bibliometric parameters using an Independent t-test and Pearson's correlation analysis. There were statistically significant weakly positive relationships between AAS and sample size, number of reads, and number of citations. There was no statistically significant relationship between AAS and number of authors, H-index, or level of evidence. This study highlights the potential value of altmetrics by measuring the social impact of research as altmetrics can provide valuable information not captured by traditional metrics. It is currently unclear what the optimal balance of social and academic impact is in evaluating research impact and how altmetrics can be integrated into existing research frameworks.

THE ROAD LESS TRAVELED: NURTURING A MEDICAL JOURNAL'S QUEST FOR INTERNATIONAL INDEXATION AND IMPACT FACTOR

Journal Impact Factor (JIF), proposed by Eugene Garfield in early 1960s, is considered as the one of the most widely recognized and influential metrics for scholarly journal ranking in the world. It provided a standardized measure based on the citation data for evaluation of journals. In 1975, the Institute for Scientific Information introduced the initial version of the Journal Citation Reports (JCR) as an essential part of the Science Citation Index, spearheaded by Eugene Garfield. First JCR was based on 4,248,065 citations of around 400,OOO source items published during 1974 in 2400 journals.1 Since then, JCR & JIF have undergone significant developments and transformations. They became widely adopted tools for evaluating research impact and journal prestige. Over the years, the IF and JCR expanded their coverage, incorporating more journals and disciplines. Recently released JCR 2023 comprised of 21,522 journals (including 13,668 science journals) from 112 countries across 254 research disciplines.2

Embedded Research: Possibilities for Learning Health Systems Comment on "'We're Not Providing the Best Care if We are Not on the Cutting Edge of Research': A Research Impact Evaluation at a Regional Australian Hospital and Health Service".

Brown et al show that research investments in an organization with a research and translation mandate can make important gains for research impact across domains, including quality of care and patient outcomes. Their multi-stage mixed methods evaluation provides insight into research capacity development in rural health systems in Australia and draws attention towards persistent geographic inequities. In extension of this important contribution, here, a focus on the "what and the why" of embedded research is offered. Specific attention is paid to the sustainability potentials of systematized data capture systems, funding-operational mandate alignments, researcher-scientist career pathways, and networked approaches to mentorship.

Exploring the application of machine learning to expert evaluation of research impact.

The objective of this study is to investigate the application of machine learning techniques to the large-scale human expert evaluation of the impact of academic research. Using publicly available impact case study data from the UK's Research Excellence Framework (2014), we trained five machine learning models on a range of qualitative and quantitative features, including institution, discipline, narrative style (explicit and implicit), and bibliometric and policy indicators. Our work makes two key contributions. Based on the accuracy metric in predicting high- and low-scoring impact case studies, it shows that machine learning models are able to process information to make decisions that resemble those of expert evaluators. It also provides insights into the characteristics of impact case studies that would be favoured if a machine learning approach was applied for their automated assessment. The results of the experiments showed strong influence of institutional context, selected metrics of narrative style, as well as the uptake of research by policy and academic audiences. Overall, the study demonstrates promise for a shift from descriptive to predictive analysis, but suggests caution around the use of machine learning for the assessment of impact case studies.

Bibliometric Analysis of Allelopathy Journal

Allelopathy Journal is the official publication of ‘Allelopathy Books’ that promotes allelopathy research and publish researches related to plant sciences and health sciences. This study aimed to present an insight about the researches published in this journal, to help the Authors, Readers and Editorial team to understand the journal in depth. We analyzed the scholarly publications published in the Allelopathy Journal, their Citations, focussing on other aspects such as Publication output, Citation analysis, Co-authorship patterns, Collaboration networks, Research productivity, Research topics and International collaboration. This study is useful to identify emerging trends and evaluate research impact. The data for the study was extracted from the Scopus Database for the period from 1996 to 2022. The results showed that the journal has published 1209 papers during this period, which received 9826 citations with an average of 8.13 citations per paper. F.Z. Wu was the most prolific author with 22 papers receiving 127 citations and Prof. S.S. Narwal, with second highest papers in the journal, received the highest number of citations (242). China was the most productive country with 425 papers in the journal, followed by India (243 papers). The Chinese Academy of Sciences was the most productive institute in the journal and published 67 papers with 678 citations. A total of 4870 authors have appeared in 1209 papers with 0.91 degree of collaboration and a collaboration index of 4.03. Thematic maps indicated that the topics represented by keywords such as "Distribution", "Habitat", "Essential Oil", “Lettuce", "Morphology," have potential importance for future research.

The post-award effort of managing and reporting on funded research: a scoping review

Introduction: Reporting is a mechanism for funding organisations to monitor and manage the progress, outputs, outcomes, and impacts of the research they fund. Inconsistent approaches to reporting and post-award management, and a growing demand for research information, can lead to perception of unnecessary administrative effort that impacts on decision-making and research activity. Identifying this effort, and what stakeholders see as unmet need for improvement, is crucial if funders and Higher Education Institutions (HEIs) are to streamline their practices and provide better support with reporting activities. In this review, we summarise the processes in post-award management, compare current practices, and explore the purpose of collecting information on funded research. We also identify areas where unnecessary effort is perceived and improvement is needed, using previously reported solutions to inform recommendations for funders and HEIs. Methods: We conducted a scoping review of the relevant research and grey literature. Electronic searches of databases, and manual searches of journals and funder websites, resulted in inclusion of 52 records and 11 websites. Information on HEI and funder post-award management processes was extracted, catalogued, and summarised to inform discussion. Results: Post-award management is a complex process that serves many purposes but requires considerable effort, particularly in the set up and reporting of research. Perceptions of unnecessary effort stem from inefficiencies in compliance, data management and reporting approaches, and there is evidence of needed improvement in mechanisms of administrative support, research impact assessment, monitoring, and evaluation. Solutions should focus on integrating digital systems to reduce duplication, streamlining reporting methods, and improving administrative resources in HEIs. Conclusions: Funders and HEIs should work together to support a more efficient post-award management process. The value of research information, and how it is collected and used, can be improved by aligning practices and addressing the specific issues highlighted in this review.

If Providing Best Care Means Being at the Cutting Edge of Research, Should It Be Implemented System-wide? Comment on "'We're Not Providing the Best Care If We Are Not on the Cutting Edge of Research': A Research Impact Evaluation at a Regional Australian Hospital and Health Service".

The COVID-19 pandemic intensified debates about the desirability of integrating health research systems into healthcare systems. An excellent evaluation undertaken prior to the pandemic examined a purposeful strategy to improve healthcare through an expansion in research capacity in the Townsville Hospital and Health Service (THHS), a regional service in northern Queensland. This comment puts that evaluation into a rapidly developing wider context, drawing on other work showing an association between research engagement in healthcare organisations and their improved healthcare. In most previous studies this impact arose as a by-product of the research activity. The Townsville scheme went further. But while the evaluation identified some progress and impacts, they were patchy, not system-wide. Recent pre-pandemic studies showed that going even further and integrating a health research system across a national healthcare system markedly improved healthcare, despite continuing challenges. The UK's research experiences during COVID-19 are giving additional momentum to this approach globally.

Cutting Edge Research? Realistic Expectations of Priorities, Scope and Engagement Comment on "'We're Not Providing the Best Care If We Are Not on the Cutting Edge of Research': A Research Impact Evaluation at a Regional Australian Hospital and Health Service".

While research is linked with informed decision-making and improved healthcare delivery and patient outcomes, the process of generating and translating research evidence in practice and capturing its impact can often be challenging. Based on document and database reviews and interviews in a regional Australian health system, Brown et al discuss the challenges of assessing the impact of research investments over a ten-year period. This commentary explores three inter-related lessons from this article for developing and sustaining a research culture and supporting translation in a health system: (i) achieving a shared definition and expectation of research; (ii) the importance of stakeholder engagement particularly for research prioritisation; and (iii) enabling research across a system. In doing so, it highlights the role and value of engaging knowledge generators and end-users from clinical, management and community domains not only in research development but most importantly in research prioritisation.

Bibliometrics, the "Clinical Diagnostics" of Research Impact Comment on "'We're Not Providing the Best Care If We Are Not on the Cutting Edge of Research': A Research Impact Evaluation at a Regional Australian Hospital and Health Service".

Following the Townsville Hospital and Health Service (THHS) strategic revision as a "research-based" institution, Brown et al have investigated the impact of THHS research, and its key drivers, based on 15 stakeholder interviews and two quantitative indicators. This commentary argues that the quantitative analyses and findings would have benefitted from applying evaluative bibliometrics, hopefully, conducted by experienced bibliometricians. We present the potential of bibliometrics for assessing the scholarly impact of research, as well as a few examples of its application to the case of THHS, for informing research policies and strategies.

Evaluating Research Impact Based on Semantic Scholar Highly Influential Citations, Total Citations, and Altmetric Attention Scores: The Quest for Refined Measures Remains Illusive

Background: The evaluation of scholarly articles’ impact has been heavily based on the citation metrics despite the limitations of this approach. Therefore, the quest for meticulous and refined measures to evaluate publications’ impact is warranted. Semantic Scholar (SS) is an artificial intelligence-based database that allegedly identifies influential citations defined as “Highly Influential Citations” (HICs). Citations are considered highly influential according to SS when the cited publication has a significant impact on the citing publication (i.e., the citer uses or extends the cited work). Altmetrics are measures of online attention to research mined from activity in online tools and environments. Aims: The current study aimed to explore whether SS HICs provide an added value when it comes to measuring research impact compared to total citation counts and Altmetric Attention Score (AAS). Methods: Dimensions was used to generate the dataset for this study, which included COVID-19-related scholarly articles published by researchers affiliated to Jordanian institutions. Altmetric Explorer was selected as an altmetrics harvesting tool, while Semantic Scholar was used to extract details related to HICs. A total of 618 publications comprised the final dataset. Results: Only 4.57% (413/9029) of the total SS citations compiled in this study were classified as SS HICs. Based on SS categories of citations intent, 2626 were background citations (29.08%, providing historical context, justification of importance, and/or additional information related to the cited paper), 358 were result citations (3.97%, that extend on findings from research that was previously conducted), and 263 were method citations (2.91%, that use the previously established procedures or experiments to determine whether the results are consistent with findings in related studies). No correlation was found between HICs and AAS (r = 0.094). Manual inspection of the results revealed substantial contradictions, flaws, and inconsistencies in the SS HICs tool. Conclusions: The use of SS HICs in gauging research impact is significantly limited due to the enigmatic method of its calculation and total dependence on artificial intelligence. Along with the already documented drawbacks of total citation counts and AASs, continuous evaluation of the existing tools and the conception of novel approaches are highly recommended to improve the reliability of publication impact assessment.

Retrospective Impact Evaluation Continuing to Prove Challenging Irrespective of Setting: A Study of Research Impact Enablers and Challenges Cloaked as an Impact Evaluation? Comment on "'We're Not Providing the Best Care If We Are Not on the Cutting Edge of Research': A Research Impact Evaluation at a Regional Australian Hospital and Health Service".

The original article provides a detailed and insightful presentation of enablers and detractors for research participation, translation, and impact, at a regional Australian hospital and health service. This information builds on existing knowledge, from the perspective of a non-metropolitan healthcare organisation. It stands to inform all healthcare organisations keen to embed research into their institutions. However, what the article fails to do is present the results of the research impact evaluation in a systematic and useful way for the reader to assess the benefits of research investment by a healthcare organisation including delivery of better quality care and improved patient outcomes. This commentary suggests why such information is critical to justify continued research investment by healthcare organisations and to showcase the potential benefits of the embedded research model. It also discusses the limitations of undertaking impact evaluation retrospectively and suggests that a prospective approach coupled with proper data collection systems and processes upfront could help future reporting of organisational research impact.

The impact of participation in research for speech and language therapy departments and their patients: A case example of the Big CACTUS multicentre trial of self-managed computerized aphasia therapy.

In order to conduct research that is meaningful to speech and language therapy services and their patients, it is often desirable to conduct the research within routine clinical services. This can require considerable time and commitment from speech and language therapists (SLTs). It is therefore important to understand the impact that such participation in research can have. To explore the impact of research participation in the Big CACTUS study of self-managed computerized aphasia therapy conducted in 21 UK NHS speech and language therapy departments. An online survey was sent to SLTs who took the lead role for the study at their NHS Trust to evaluate the impact of study participation in three domains: capacity-building, research development and health services. The questionnaire, based on the VICTOR framework for evaluating research impact, included Likert scale statements and closed and open-ended questions. The results from open-ended questions were coded and analysed using framework analysis in NVivo 12 and the data from closed questions were analysed descriptively. A total of 12 SLTs returned the survey. Nine codes were identified from open-ended questions and 20 predefined from the literature. Analysis of the responses demonstrated the perceived impact including improvements in practices and access to therapy, investments in infrastructure, increased SLT profile, and impact on research culture among SLTs. The usefulness of the intervention during the COVID-19 pandemic was also highlighted. The results suggest participation in Big CACTUS has resulted in improvements in patient care and SLT research capacity and culture in speech and language therapy departments. What is already known on the subject Practice-based research is encouraged to assist with the clinical relevance of the research findings. Participation in research can be seen as an activity that is additional to the core business of patient care and it can be difficult to secure time to participate or conduct research in clinical settings. Impact evaluation initiatives of individual trials facilitate early identification of benefits beyond the trial. What this paper adds to existing knowledge This study describes specific examples of the impact on services, staff and patients from SLT participation and leadership in the Big CACTUS speech and language therapy trial in clinical settings. What are the potential or actual clinical implications of this study? Clinical services participating in research may benefit from improved clinical care for patients both during and after the study, an improved professional reputation, and increased research capacity and culture within the clinical settings.

Impact of COVID-19 Dentistry-Related Literature: An Altmetric Study

Scientific literature on COVID-19 has grown rapidly during the pandemic. The aim of this study was to provide a comprehensive overview of the popularity on the web of the available dental publications on COVID-19 and to examine associations amongst article characteristics, online mentions, and citations. An Altmetric Explorer search was conducted for COVID-19 articles published in dental journals using 3 keywords: COVID-19, SARS-CoV-2, and pandemic. The following Altmetric data were collected: Altmetric attention score (AAS), mentions by news outlets, tweets, Mendeley readers, and Web of Science citations. Additionally, article title, type, topic, origin and open access status, journal title, quartile of impact factor (IF) distribution, and time lapse between COVID-19 pandemic onset and publication date were analysed. In all, 253 articles published in 48 dental journals were eligible for the study. AAS was significantly influenced by article topic, type, origin, and journal IF quartile. There was a negligible correlation between AAS and Web of Science citations. Mendeley was the only Altmetric source highly correlated with citations. There was substantial online interest in COVID-19 dentistry-related literature, as depicted by the AAS of the reviewed articles and social media metrics. Mendeley reader counts were highly correlated with citations, and they may therefore be valuable in research impact evaluation.

Open Access Monographs: Making Mandates Reality

Watch the VIDEO. In early 2022 the major research funding body in the UK (UKRI), released a policy statement mandating Open Access (OA) monographs, with an implementation date of January 1st, 2024. This date will see a shift from a hypothetical future for OA books to a concrete policy with a hard deadline. Add to this the forthcoming mandates on OA books from cOAlition S/Plan S in Europe, and potential implications of the next research impact evaluation (REF) in the UK, and it is clear that there is a pressing need for libraries and academic book publishers to understand how the sector will meet the challenges of implementing these policies. In this session we will present the following questions: what is the transition to open access for scholarly books going to look like? What pitfalls and opportunities will it bring? Who is at risk of being excluded? How do we get from where publishing and libraries are today, to making these policies a workable reality in 2024? We will also pose the crucial question: how will this transition be paid for? We’ll demonstrate how Book Processing Charges (BPCs) worsen inequality by favouring the most wealthy institutions and authors. And we’ll highlight some collective library funding models that seek to spread the funding of OA books so that no single institution bears a disproportionate cost. These models range from the large and well-funded like MIT’s Direct 2 Open and Michigan’s Fund to Mission, to the COPIM project’s own Opening the Future model which is aimed at smaller mission-driven publishers like CEU Press and Liverpool University press. With the clock ticking on imminent policies, conversation is urgently needed on the practicalities of making mandates reality. We’ll discuss what libraries can do to meet the challenge, and how supporting programmes like Opening the Future will ensure that the transition to OA is a sustainable and bibliodiverse one. The transition to OA should not leave smaller and medium-sized presses behind; nor should it rely on paying BPCs, which risks excluding any author without funding; and the transition must be sustainable for libraries. We will demonstrate that the time to support collective funding programme pilots is now, in order that they can transition from pilot to implementable policy, to the benefit of all: opening up research for the public good and building an open, global ‘collection’ that is shared by libraries in common, around the world.

Altmetric Analysis of the Most Mentioned Articles Online in the Contemporary Craniofacial Surgery Literature.

As online media acquires increased utilization as a means of disseminating scientific research, Altmetric analyses are useful to identify socially impactful publications and their characteristics. Accordingly, the objective of this investigation was to determine the most mentioned articles online regarding craniofacial surgery and compare these articles with their most cited counterparts. Using the Web of Science database, 7097 craniofacial surgery publications were identified. Altmetric Attention Score (AAS) and average citations per year (ACpY) were extracted, and articles were ranked by their social and scientific impact. Descriptive, comparative, and correlational analyses were performed to examine characteristics including AAS, ACpY, country of origin, journal, open access (OA) status, publication year, study design, and topic. The average AAS of the 50 most mentioned articles was 83.68 (SD±107.40), and the average citation count of the 50 most cited articles was 76.38 (SD±35.15). OA status was significantly more prevalent among the most mentioned articles (24.0%) relative to all articles (10.72%) and the most cited articles (4.0%). The most mentioned and the most cited articles primarily originated from the United States (64.0% and 38.0%, respectively), discussed topics related to the medical and surgical management of patients (48.0% and 58.0%, respectively), and were systematic reviews (16.0% and 28.0%, respectively). However, there was significantly more geographic diversity in the latter cohort. In summary, this comparative examination reveals important differences between the most mentioned and most cited articles, highlighting the distinct utility of AAS and citation count when evaluating research impact.