Evaluation Of Research Impact Research Articles

Four Methodological Guidelines to Evaluate the Research Impact of Co-produced Climate Services

As climate change impacts unfold across the globe, growing attention is paid toward producing climate services that support adaptation decision-making. Academia, funding agencies, and decision-makers generally agree that stakeholder engagement in co-producing knowledge is key to ensure effective decision support. However, co-production processes remain challenging to evaluate, given their many intangible effects, long time horizons, and inherent complexity. Moreover, how such evaluation should look like is understudied. In this paper, we therefore propose four methodological guidelines designed to evaluate co-produced climate services: (i) engaging in adaptive learning by applying developmental evaluation practices, (ii) building and refining a theory of change, (iii) involving stakeholders using participatory evaluation methods, and (iv) combining different data collection methods that incorporate visual products. These methodological guidelines offset previously identified evaluation challenges and shortcomings, and can be used to help stakeholders rethink research impact evaluation through their complementary properties to identify complex change pathways, external factors, intangible effects, and unexpected outcomes.

"We're Not Providing the Best Care If We Are Not on the Cutting Edge of Research": A Research Impact Evaluation at a Regional Australian Hospital and Health Service.

Research is central to high functioning health services alongside clinical care and health professional training. The impact of embedded research includes delivery of high-quality care and improved patient outcomes. Evaluations of research impact help health service leadership ensure investments lead to the greatest healthcare benefits for patients. This study aimed to retrospectively evaluate the impact of research investment from 2008 to 2018 at Townsville Hospital and Health Service (THHS), a regional Hospital and Health Service (HHS) in Queensland, Australia. The evaluation also sought to identify contextual conditions that enable or hinder intended impacts. A mixed-methods realist-informed evaluation was conducted using documentation, interviews with 15 staff and available databases to identify and measure research investments, impacts and contextual conditions influencing impact outcomes. Between 2008 and 2018, THHS increased resources for research by funding research projects, employing research personnel, building research-enabling facilities, hosting research events, and providing research education and training. Clinical practice, policy and workforce impacts were successful in isolated pockets, championed by individual researchers and facilitated by their policy and community-of-practice networks. However, there was little organisational-level support for continuity of research and implementation into practice and policy. Availability of research supports varied geographically across THHS, and across disciplines. Definitive steps in the development of THHS as a credible and productive research centre and leading hospital research centre in Northern Australia are evident. Continuing investments should address support for the research continuum through to translation and establish ongoing, systematic processes for evaluating research investment and impact.

A Novel Web Scraping Approach to Identify Stroke Outcome Measures: A Feasibility Study

<h3>Research Objectives</h3> Web scraping is an innovative, efficient and automatic software application to extract large amounts of website data, commonly used for price tracking, product comparison, weather monitoring, and tracking online presence. While this approach provides a promising method to identify and summarize text-based online data, it has yet been utilized in rehabilitation research. This study applies a web-scraping method to rehabilitation research by investigating the scope of existing outcome instruments for stroke patients from one website. <h3>Design</h3> This is a feasibility study using Python programming language and Scrapy framework for identifying web-scraping measurement domains. <h3>Setting</h3> We used the Rehabilitation Measures Database website (RMD; www.sralab.org/rehabilitation-measures) to extract information on stroke outcome instruments. The RMD provides instrument information, such as the International Classification of Functioning, Disability and Health (ICF) and measurement domains, cost, and administration time. <h3>Participants</h3> Not applicable. <h3>Interventions</h3> Not applicable. <h3>Main Outcome Measures</h3> Measurement domains were extracted and summarized using counts and percentages. <h3>Results</h3> Less than fifteen minutes were taken for accessing each RMD page to store measurement domains of instruments for stroke patients in a csv formatted file by running python programming queries. Among 124 instruments identified for stroke patients, motor (38.5%, n=65) was the most frequently measured domain, followed by activities of daily living (23.1%, n=39), general health (12.4%, n=21), cognition (10.7% n=18), emotion (7.7%, n=13), sensory (5.9%, n=10), and participation (1.8%, n=3). <h3>Conclusions</h3> This study demonstrated that a web-scraping method could be a convenient tool to retrieve publicly available online information for clinical or research purposes. A web-scraping method allows users to obtain target information in analytically friendly formats without requiring laborious manual efforts. Future rehabilitation research studies could leverage web scraping to support making efficient clinical decisions, classifying rehabilitation data, evaluating research impact, and exploring online attitudes, sentiment and behaviors. <h3>Author(s) Disclosures</h3> All authors listed in the abstract do not have any conflicts or lack thereof.

Building AIPedia Ontology to Evaluate Research Impact in Artificial Intelligence Area

Building AIPedia Ontology to Evaluate Research Impact in Artificial Intelligence Area

Understanding, measuring, and encouraging public policy research impact

AbstractAcademics undertaking public policy research are committed to tackling interesting questions driven by curiosity, but they generally also want their research to have an impact on government, service delivery, or public debate. Yet our ability to capture the impact of this research is limited because impact is under‐theorised, and current systems of research impact evaluation do not allow for multiple or changing research goals. This article develops a conceptual framework for understanding, measuring, and encouraging research impact for those who seek to produce research that speaks to multiple audiences. The framework brings together message, medium, audience, engagement, impact, evaluation, and affordance within the logics of different fields. It sets out a new way of considering research goals, measurements, and incentives in an integrated way. By accounting for the logics of different fields, which encompass disciplinary, institutional, and intrinsic factors, the framework provides a new way of harnessing measurements and incentives towards fruitful learning about the contribution diverse types of public policy research can make to wider impact.

Embedding biodiversity research into climate adaptation policy and practice.

Addressing climate change risks requires collaboration and engagement across all sectors of society. In particular, effective partnerships are needed between research scientists producing new knowledge, policy-makers and practitioners who apply conservation actions on the ground. We describe the implementation of a model for increasing the application and useability of biodiversity research in climate adaptation policy and practice. The focus of the program was to increase the ability of a state government agency and natural resource practitioners in Australia to manage and protect biodiversity in a changing climate. The model comprised a five-stage process for enhancing impact (i) initiation of research projects that addressed priority conservation policy and management issues; (ii) co-design of the research using a collaborative approach involving multiple stakeholders; (iii) implementation of the research and design of decision tools and web-based resources; (iv) collaborative dissemination of the tools and resources via government and community working groups; and (v) evaluation of research impact. We report on the model development and implementation, and critically reflect on the model's impact. We share the lessons learnt from the challenges of operating within a stakeholder group with diverse objectives and criteria for success, and provide a template for creating an environmental research program with real world impact.

A Pilot Study for Evaluating Economic and Societal Impact of Engineering Research

Quantitative estimation of research societal and economic impact remains an ongoing issue. This is due to the difficulty to attribute specific economic or societal impact to particular research activity. Nevertheless, evaluating research impact is necessary to justify research spending especially for engineering research studies, as they are generally more costly than humanities or social studies for example. This work considered research impact evaluation in NSF (The National Science Foundation in the USA), EPSRC (Engineering and Physical Sciences Research Council in the UK), and TRC (The Research Council in Oman) to propose a low-cost survey-based approach to evaluate the engineering research impact. A questionnaire was developed and distributed to all faculty staff at the Department of Mechanical and Industrial Engineering at Sultan Qaboos University (SQU) to assess their recent research impact in the years 2016 and 2017. The evaluation was carried at three levels; knowledge, societal, and economic impact. The results of the study were compared with an independent study produced by an international consultancy group. This study offers an important contribution to ‎the subject of evaluating impact factors of engineering research in Oman.&#x0D; Keywords: SQU; societal impact; socioeconomic impact; NSF; EPSRC; RAND; TRC; knowledge impact; Oman

Policy options to increase motivation for improving evidence-informed health policy-making in Iran

BackgroundCurrent incentive programmes are not sufficient to motivate researchers and policy-makers to use research evidence in policy-making. We conducted a mixed-methods design to identify context-based policy options for strengthening motivations among health researchers and policy-makers to support evidence-informed health policy-making (EIHP) in Iran.MethodsThis study was conducted in 2019 in two phases. In the first phase, we conducted a scoping review to extract interventions implemented or proposed to strengthen motivations to support EIHP. Additionally, we employed a comparative case study design for reviewing the performance evaluation (PE) processes in Iran and other selected countries to determine the current individual and organizational incentives to encourage EIHP. In the second phase, we developed two policy briefs and then convened two policy dialogues, with 12 and 8 key informants, respectively, where the briefs were discussed. Data were analysed using manifest content analysis in order to propose contextualized policy options.ResultsThe policy options identified to motivate health researchers and policy-makers to support EIHP in Iran were: revising the criteria of academic PE; designing appropriate incentive programmes for nonacademic researchers; developing an indicator for the evaluation of research impact on policy-making or health outcomes; revising the current policies of scientific journals; revising existing funding mechanisms; presenting the knowledge translation plan when submitting a research proposal, as a mandatory condition; encouraging and supporting mechanisms for increasing interactions between policy-makers and researchers; and revising some administrative processes (e.g. managers and staff PEs; selection, appointment, and changing managers and reward mechanisms).ConclusionsThe current individual or organizational incentives are mainly focused on publications, rather than encouraging researchers and policy-makers to support EIHP. Relying more on incentives that consider the other impacts of research (e.g. impacts on health system and policy, or health outcomes) is recommended. These incentives may encourage individuals and organizations to be more involved in conducting research evidence, resulting in promoting EIHP.Trial registrationNA.

Standards for Evaluating Impact in Entrepreneurship Education Research: Using a Descriptive Validity Framework to Enhance Methodological Rigor and Transparency

Given calls for more rigorous research able to measure the impact of entrepreneurship education, this study proposes guidelines for enhancing methodological and reporting practices. First, drawing on prior research syntheses, we developed a descriptive validity framework that outlines key elements for rigorous evaluation research. Second, we use this framework to examine 61 quantitative, university-based entrepreneurship education impact studies to identify and describe methodological and reporting practices that are most prevalent. The result is a set of Impact Evaluation Research Standards for entrepreneurship educators and scholars wishing to improve education evaluation research.

Charting the policy development process of social housing bonds in Australia through an impact narrative approach

AbstractThe Australian Housing and Urban Research Institute (AHURI) has, since its inception, prioritised policy relevance in its research. Drawing on an impact narrative approach, this paper presents a case study of the contribution of AHURI research to the development of a specific policy initiative: the Affordable Housing Bond Aggregator function of the National Housing and Finance Investment Corporation (NHFIC). The paper will explore the development of this policy initiative through a series of research projects and publications, and highlight processes of dissemination and engagement with the policy and practice community. This will reveal the value of a strong evidence‐base, and effective knowledge transfer mechanisms in the development and implementation of policy. It will also demonstrate the utility of a policy impact narrative approach in the evaluation of research impact, particularly in relation to policy development processes subject to the vagaries of politics.

Evaluating Social Marketing Messages in New Zealand’s Like Minds Campaign and Its Effect on Stigma

Background: A key objective of government and social marketers is to remove the institutionalized stigma of mental illness, increasing mental health service uptake. While research has evaluated past campaigns based on changes in attitudes and beliefs, very little research has examined the communication messages used in social marketing campaigns. Focus of the Article: This impact evaluation research identifies the institutionalized cultural-moral norms incorporated into New Zealand’s Like Minds mental health advertisements and examines how attitudes and beliefs changed over time in response to these norms. Importance to the Social Marketing Field: This research offers a new approach to social marketing evaluation and demonstrates the importance of consistent incorporation of cultural-moral institutional norms in social marketing campaigns. Method: Using macro-social marketing theory, thematic analysis is used to identify the cultural-moral institutional norms in the Like Minds campaign advertisements over a 10-year period (2002–2012). Results: The Like Minds campaign was found to have multiple cultural-moral institutional norms, such as Mental illness as a villain, Personal responsibility, and Inherent human dignity, as well as utilizing two different institutionalization processes of Socialization and Identity Formation. However, these norms were inconsistently and sometimes contradictorily presented and as a result, not all changes in mental health stigma beliefs and attitudes show long term change. Rates for service uptake also had mixed results during the campaign duration, though overall an increase in uptake was found. Recommendations for Research and Practice: The research highlights the importance of understanding the underlying institutionalized cultural-moral norms presented in communications and aligning those with the overall objectives of a social marketing campaign. Limitations: Like Minds campaign phases 2 to 5 are analyzed, phase 1 was inaccessible for analysis and advertisements after 2012 are not analyzed.

Research impact evaluation and academic discourse

The introduction of ‘impact’ as an element of assessment constitutes a major change in the construction of research evaluation systems. While various protocols of impact evaluation exist, the most articulated one was implemented as part of the British Research Excellence Framework (REF). This paper investigates the nature and consequences of the rise of ‘research impact’ as an element of academic evaluation from the perspective of discourse. Drawing from linguistic pragmatics and Foucauldian discourse analysis, the study discusses shifts related to the so-called Impact Agenda on four stages, in chronological order: (1) the ‘problematization’ of the notion of ‘impact’, (2) the establishment of an ‘impact infrastructure’, (3) the consolidation of a new genre of writing–impact case study, and (4) academics’ positioning practices towards the notion of ‘impact’, theorized here as the triggering of new practices of ‘subjectivation’ of the academic self. The description of the basic functioning of the ‘discourse of impact’ is based on the analysis of two corpora: case studies submitted by a selected group of academics (linguists) to REF2014 (no = 78) and interviews (n = 25) with their authors. Linguistic pragmatics is particularly useful in analyzing linguistic aspects of the data, while Foucault’s theory helps draw together findings from two datasets in a broader analysis based on a governmentality framework. This approach allows for more general conclusions on the practices of governing (academic) subjects within evaluation contexts.

Towards Evaluating the Research Impact made by Universities of Applied Sciences

Abstract Given the mandate of Universities of Applied Sciences (UASs) to create an impact on society, the evaluation of their research impact is of great importance. And yet, the methodology for evaluating this impact appear less explicitly in research literature then other forms of research. The purpose of this article is to present a literature-based analysis to discover from the complex world of existing theories and frameworks what criteria, assumptions and requirements are relevant for evaluating the impact of applied research. This article will also discuss the relevancy of frameworks currently used for research impact evaluation and the potential they have for operationalising, enriching and supporting the current national evaluation framework used by Dutch UASs. Finally, this article will conclude that the recommendations necessitate the creation of a new framework where the context and process of practice-based research and their stakeholders are included.

Impact Indicators for Biodiversity Conservation Research: Measuring Influence within and beyond Academia

ABSTRACT Measuring, reporting, and forecasting research impact beyond academia has become increasingly important to demonstrate and understand real-world benefits. This is arguably most important in crisis disciplines such as medicine, environmental sustainability and biodiversity conservation, where application of new knowledge is urgently needed to improve health and environmental outcomes. Increasing focus on impact has prompted the development of theoretical guidance and practical tools tailored to a range of disciplines, but commensurate development of tools for conservation is still needed. In the present article, we review available tools for evaluating research impact applicable to conservation research. From these, and via a survey of conservation professionals, we compiled and ranked a list of 96 impact indicators useful for conservation science. Our indicators apply to a logic chain of inputs, processes, outputs, outcomes, and impacts. We suggest the list can act as a clear guide to realize and measure potential impacts from conservation research within and beyond academia.

Evaluating impact from research: A methodological framework

BackgroundInterest in impact evaluation has grown rapidly as research funders increasingly demand evidence that their investments lead to public benefits. AimsThis paper analyses literature to provide a new definition of research impact and impact evaluation, develops a typology of research impact evaluation designs, and proposes a methodological framework to guide evaluations of the significance and reach of impact that can be attributed to research. MethodAn adapted Grounded Theory Analysis of research impact evaluation frameworks drawn from cross-disciplinary peer-reviewed and grey literature. ResultsRecognizing the subjective nature of impacts as they are perceived by different groups in different times, places and cultures, we define research impact evaluation as the process of assessing the significance and reach of both positive and negative effects of research.Five types of impact evaluation design are identified encompassing a range of evaluation methods and approaches: i) experimental and statistical methods; ii) textual, oral and arts-based methods; iii) systems analysis methods; iv) indicator-based approaches; and v) evidence synthesis approaches.Our guidance enables impact evaluation design to be tailored to the aims and context of the evaluation, for example choosing a design to establish a body of research as a necessary (e.g. a significant contributing factor amongst many) or sufficient (e.g. sole, direct) cause of impact, and choosing the most appropriate evaluation design for the type of impact being evaluated. ConclusionUsing the proposed definitions, typology and methodological framework, researchers, funders and other stakeholders working across multiple disciplines can select a suitable evaluation design and methods to evidence the impact of research from any discipline.

Are Altmetric.com scores effective for research impact evaluation in the social sciences and humanities?

Altmetric.com is one of the leading and widely used commercial providers of altmetrics. Altmetrics is a broad term that encapsulates the collection of multiple digital indicators related to scholarly work (NISO, 2016). In this paper, the group of altmetrics provided by Altmetric.com is called Altmetric.com scores. This study assesses whether Altmetric.com scores are currently effective indicators to evaluate the research impact of social sciences and humanities (SSH). This research covers 1,253,028 records from the SSCI and A&HCI for five years (2013–2017), three publication types (articles, reviews, and proceedings papers), and 10 altmetrics provided by Altmetric.com to test the effectiveness of Altmetric.com scores through four proposed dimensions, including transparency, fairness, availability, and validity. Results show that the transparency of Altmetric.com scores is underexposed. And Altmetric.com scores favored English publications authored by North American researchers. Except for Altmetric attention score, Posts, Readers, and Tweeters, the data presence of most Altmetric.com scores was insufficient. As to research impact, Altmetric.com scores can help to reflect different impacts of SSH research. Overall, the results suggest that Altmetric.com scores are not yet sufficiently effective for evaluating the impact of SSH research though they can provide new insights into impact evaluation.

Classifying the ideational impact of Information Systems review articles: A content-enriched deep learning approach

Ideational impact refers to the uptake of a paper's ideas and concepts by subsequent research. It is defined in stark contrast to total citation impact, a measure predominantly used in research evaluation that assumes that all citations are equal. Understanding ideational impact is critical for evaluating research impact and understanding how scientific disciplines build a cumulative tradition. Research has only recently developed automated citation classification techniques to distinguish between different types of citations and generally does not emphasize the conceptual content of the citations and its ideational impact. To address this problem, we develop Deep Content-enriched Ideational Impact Classification (Deep-CENIC) as the first automated approach for ideational impact classification to support researchers' literature search practices. We evaluate Deep-CENIC on 1256 papers citing 24 information systems review articles from the IT business value domain. We show that Deep-CENIC significantly outperforms state-of-the-art benchmark models. We contribute to information systems research by operationalizing the concept of ideational impact, designing a recommender system for academic papers based on deep learning techniques, and empirically exploring the ideational impact in the IT business value domain.

Putting Crossref Metadata into Context

At Crossref, we see a future where society benefits from an open scholarly infrastructure in which all research is connected and preserved. This will enable researchers to build upon knowledge, ensure greater discoverability and dissemination of research, and aid decision-making for funders, with better evaluation and tracking of research impact. This short talk aims to deepen the understanding of the power of good metadata, by explaining how it can be harnessed to improve discoverability of content, and how it is used in the creation of collaborative tools and initiatives for the research community.&#x0D; Persistent identifiers (PIDs) enable clear disambiguation and persistent links to be made between researchers, their affiliations, funding and contributions. However, membership at Crossref is not just about creating identifiers for content. It is about placing that content into context and a DOI is simply one small, but important, piece of the puzzle. Crossref works in collaboration with individuals and organizations to collect extensive metadata, both bibliographic and non-bibliographic. The breadth of the metadata Crossref supports is capable of expanding over time to respond to community needs e.g. incorporating CRediT for contributor roles. It’s also key that the metadata is made openly available; comprehensive, open metadata guarantees interoperability across borders, disciplines, research outputs and organizations. It is also essential for building scalability and improving efficiency via collaborative initiatives such as the Research Organizations Registry (ROR), ORCID auto-update, and the new DataCite Commons. &#x0D; Crossref has minimal requirements in order to support a variety of publication practises and the schema can support a diverse range of content types and associated metadata. All of the metadata received is standardized and machine readable; the more complete this metadata is, the more likely content can be discovered and disseminated. Whenever you submit metadata to Crossref, DataCite, DOAJ or other organizations, the metadata you deposit should be accurate, complete and up to date. Accuracy is important as misspelled author names, incorrect dates, missing license information or bad URLs, undermine the usefulness of shared services, as well as being a pain for readers and authors. &#x0D; There remains a lack of knowledge about the value and importance of quality metadata. Machine-readable metadata and identifiers underpin both Crossref tools and services, and those developed by the wider community. I’ll detail examples of the applications of scholarly metadata to make this case, show how publishers can see the metadata they’re registering and suggest some quick fixes to getting more from your Crossref metadata.

WITHDRAWN: Evaluating impact from research: A methodological framework

Abstract Background Interest in impact evaluation has grown rapidly as research funders increasingly demand evidence that their investments lead to public benefits. Aims This paper analyses literature to provide a new definition of research impact and impact evaluation, develops a typology of research impact evaluation designs, and proposes a methodological framework to guide evaluations of the significance and reach of impact that can be attributed to research. Method An adapted Grounded Theory Analysis of research impact evaluation frameworks drawn from cross-disciplinary peer-reviewed and grey literature. Results • Recognizing the subjective nature of impacts as they are perceived by different groups in different times, places and cultures, we define research impact evaluation as the process of assessing the significance and reach of both positive and negative effects of research. • Five types of impact evaluation design are identified encompassing a range of evaluation methods and approaches: i) experimental and statistical methods; ii) textual, oral and arts-based methods; iii) systems analysis methods; iv) indicator-based approaches; and v) evidence synthesis approaches. • Our guidance enables impact evaluation design to be tailored to the aims and context of the evaluation, for example choosing a design to establish a body of research as a necessary (e.g. a significant contributing factor among many) or sufficient (e.g. sole, direct) cause of impact, and choosing the most appropriate evaluation design for the type of impact being evaluated. Conclusion Using the proposed definitions, typology and methodological framework, researchers, funders and other stakeholders working across multiple disciplines can select a suitable evaluation design and methods to evidence the impact of research from any discipline.

Impact of surgical intervention trials on healthcare: A systematic review of assessment methods, healthcare outcomes, and determinants.

BackgroundFrameworks used in research impact evaluation studies vary widely and it remains unclear which methods are most appropriate for evaluating research impact in the field of surgical research. Therefore, we aimed to identify and review the methods used to assess the impact of surgical intervention trials on healthcare and to identify determinants for surgical impact.MethodsWe searched journal databases up to March 10, 2020 for papers assessing the impact of surgical effectiveness trials on healthcare. Two researchers independently screened the papers for eligibility and performed a Risk of Bias assessment. Characteristics of both impact papers and trial papers were summarized. Univariate analyses were performed to identify determinants for finding research impact, which was defined as a change in healthcare practice.ResultsSixty-one impact assessments were performed in 37 included impact papers. Some surgical trial papers were evaluated in more than one impact paper, which provides a total of 38 evaluated trial papers. Most impact papers were published after 2010 (n = 29). Medical records (n = 10), administrative databases (n = 22), and physician’s opinion through surveys (n = 5) were used for data collection. Those data were analyzed purely descriptively (n = 3), comparing data before and after publication (n = 29), or through time series analyses (n = 5). Significant healthcare impact was observed 49 times and more often in more recent publications. Having impact was positively associated with using medical records or administrative databases (ref.: surveys), a longer timeframe for impact evaluation and more months between the publication of the trial paper and the impact paper, data collection in North America (ref.: Europe), no economic evaluation of the intervention, finding no significant difference in surgical outcomes, and suggesting de-implementation in the original trial paper.Conclusions and implicationsResearch impact evaluation receives growing interest, but still a small number of impact papers per year was identified. The analysis showed that characteristics of both surgical trial papers and impact papers were associated with finding research impact. We advise to collect data from either medical records or administrative databases, with an evaluation time frame of at least 4 years since trial publication.