Evaluation Of Research Impact Research Articles

Bibliometric study on the sustainability of EU FP projects: Methodological aspects

The influence of EU Framework Programmes on European and even global research policies is increasing every year. One of the examples of this trend is the fast development of research impact evaluation. In particular, this concerns studies that are dealing with long-term effects of R&D funding programmes. Bibliometric data have been one type of indicators of research funding. In case of EU Framework Programmes there are numerous questions and options as to a) the choice of reliable data sources, b) compatibility of different datasets, c) reliability of referencing practices, d) time consuming data cleaning, e) the additional workload due to the overlap between the various framework programmes (if the goal is to make a survey on the results of a specific framework programme). The objective of the current paper is to analyse, using the example of the FPs, to what extent the existing sources of information (WoS) make it possible to form an adequate picture of the impact of FP funding, and what are the shortcomings. A characteristic feature of FPs is their long-term durability, both in terms of signed contracts and in terms of the output timeline. The peak of the publication is reached eight to nine years after the start of the Framework Programme. Despite the clear referencing rules, grant recipients are not following them. The reliability of data is becoming a key issue.

An Evaluation of Research Impact in Accounting, Economics, Finance, Management, Marketing, Psychology, and the Natural Sciences

An Evaluation of Research Impact in Accounting, Economics, Finance, Management, Marketing, Psychology, and the Natural Sciences

Evaluating the non-academic impact of academic research: design considerations

ABSTRACTEvaluation of academic research plays a significant role in government efforts to steer public universities. The scope of such evaluation is now being extended to include the ‘relevance’ or ‘impact’ of academic research outside the academy. We address how evaluation of non-academic research impact can promote more such impact without undermining academic freedom and research excellence. Five questions on evaluation design are considered: (1) What should be the object of measurement? (2) What should be the timeframe? (3) How should non-academic users of research inform evaluation processes? (4) How should controversial impacts be managed? (5) When in funding cycles should impact evaluation occur? We conclude that non-academic impact should be selectively promoted and evaluated. This is how greater gains from research might be best captured without imposing misguided and onerous reporting requirements on individuals and institutions.

Business and Management Impact Assessment in Research Excellence Framework 2014: Analysis and Reflection

The evaluation of research impact is likely to remain an important element of research quality audits in the UK for the foreseeable future. With this paper, we contribute to debates on impact and relevance of business and management studies research through an analysis of Research Excellence Framework 2014 impact scores within the business and management unit of assessment. We offer insights into the organizational contexts of UK business schools within which impact is produced, drawing attention to the issues of linkages with research intensity, grant income generation, research team size, career stage and gender of academics, and whether impact activity is focused on private or public sector organizations and national or international reach. We put forward recommendations for managers responsible for business schools and higher education policymakers regarding management and organizational policies and processes, as well as possible changes to the rules guiding future research excellence audits.

Evaluating Research Impact: The Development of a Research for Impact Tool.

This paper examines the process of developing a Research for Impact Tool in the contexts of general fiscal constraint, increased competition for funding, perennial concerns about the over-researching of Aboriginal and Torres Strait Islander issues without demonstrable benefits as well as conceptual and methodological difficulties of evaluating research impact. The aim is to highlight the challenges and opportunities involved in evaluating research impact to serve as resource for potential users of the research for impact tool and others interested in assessing the impact of research. A combination of literature reviews, workshops with researchers, and reflections by project team members and partners using participatory snowball techniques. Assessing research impact is perceived to be difficult, akin to the so-called "wicked problem," but not impossible. Heuristic and collaborative approach to research that takes the expectations of research users, research participants and the funders of research offers a pragmatic solution to evaluating research impact. The logic of the proposed Research for Impact Tool is based on the understanding that the value of research is to create evidence and/or products to support smarter decisions so as to improve the human condition. Research is, therefore, of limited value unless the evidence created is used to make smarter decisions for the betterment of society. A practical way of approaching research impact is, therefore, to start with the decisions confronting decision makers whether they are government policymakers, industry, professional practitioners, or households and the extent to which the research supports them to make smarter policy and practice decisions and the knock-on consequences of doing so. Embedded at each step in the impact planning and tracking process is the need for appropriate mix of expertise, capacity enhancement, and collaborative participatory learning-by-doing approaches. The tool was developed in the context of Aboriginal and Torres Strait Islander research but the basic idea that the way to assess research impact is to start upfront with the information needs of decisions makers is equally applicable to research in other settings, both applied (horizontal) and basic (vertical) research. The tool will be further tested and evaluated with researchers over the next 2 years (2016/17). The decision by the Australian Government to include 'industry engagement' and 'impact' as additions to the Excellence in Research for Australia (ERA) quality measures from 2018 makes the Research for Impact Tool a timely development. The wider challenge is to engage with major Australian research funding agencies to ensure consistent alignment and approaches across research users, communities, and funders in evaluating impact.

When will we ever learn … to change policy: current state of impact evaluation

This article explores the recent past, current status and future of impact evaluation (IE) of development interventions. We both review the existing literature and conduct key informant interviews with Evaluators and Implementers. We provide an overview of the field of IE of development interventions, both in terms of conducting IE studies and in terms of using IEs for policy change. We also examine the challenges for both Evaluators and Implementers in combining IE research with efforts to change policy.

A08 The evaluation of research impact on transport and health in five non-communicable disease areas as reported in a UK newspaper

BackgroundNewspaper reports of research on transport and health impact can influence ministers, senior officials, medical personnel, other researchers and the general public. This project explores the reporting of research activities in a widely-circulated UK newspaper, the Daily Mail, on various forms of transport and their effects on five non-communicable diseases (NCDs) namely oncology, cardiovascular diseases, diabetes, respiratory and mental disorders. MethodsThe Daily Mail was examined for articles citing research, i.e. peer-reviewed journal articles, from 2002 to 2013. In total 28,254 stories were retrieved from the online archive using a systematic search of keywords and these were screened for relevance. The keywords used were based upon a combination of 58 identified themes such as walking or diesel or train, with another four namely research, study, science and policy. In total 232 searches were performed under the health section of the e-archive. Data such as headline and journalist, details of the cited papers, any commentators and a brief synopsis were transferred to an Excel spreadsheet and the types of research and the NCDs affected were recorded by means of codes. After exclusion of personal stories not involving research, and of duplicates, a total of 300 studies fitted the inclusion criteria. Details of the cited papers were obtained from the Web of Science. These were then used for statistical analysis with a range of VBA macros. ResultsPreliminary analysis showed that a third of the stories were about the positive effects of walking and cycling on fitness, and hence on cardiovascular disease and diabetes. The other two-thirds were negative, and concerned the effects of noise and air pollution on mental health, respiratory diseases and cancer. Geographically, the USA dominated the cited research papers with 30%, followed by the UK (21%) and Canada (11%). These papers were very clinical and described the effects on health rather than basic research. ConclusionsTransport affects human health in positive and negative ways. For example, recent legislation (December 2014) that provides for a smoking ban in cars in the presence of children has been receiving repeated reporting in the press since 2009 by experts seeking to create this law. Transport policy can thus affect any of these five NCDs and influence individuals on making choices about using different forms of transport or choosing where they live.

La valutazione d’impatto socio-economico e culturale della ricerca nelle scienze umane e sociali: uno studio di caso in una Università del Regno Unito

The aim of this paper is to describe the impact of research on society, economy and culture in the Humanities and Social sciences. In the first part of the paper, we describe the principal characteristics of the evaluation of research impact in the United Kingdom (UK), with a special focus on the new national exercise of research evaluation: the ‘Research Excellence Framework’ (REF2014). In the second part of the paper we present a case study in a British university campus where good practices of impact evaluation have been recently introduced. The main results of this study show that impact evaluation of Social sciences and Humanities research is a today’s key challenge for the academia, especially due to the difficulty in identifying the appropriate indicators for research impact measurement. Finally, we present the main themes of the current debate about the evaluation of the impacts in different fields of research.

Developing a Research Agenda for Impact Evaluation in Development

This article sets out what would be required to develop a research agenda for impact evaluation. It begins by explaining why it is needed and what process it would involve. It outlines four areas where research is needed - the enabling environment, practice, products and impacts. It reviews the different research methods that can be used to research impact evaluation and argues for particular attention to detailed, theory-informed, mixed-method comparative case studies of the actual processes and impacts of impact evaluation. It explores some examples of research questions that would be valuable to focus on and how they might be addressed. Finally, it makes some suggestions about the process that is needed to create a formal and collaborative research agenda.

Income Transfer Policies and Nutritional Condition of Children: An Evaluation of “Bolsa Família”

This paper investigates the impact of the Bolsa Familia program, a conditional cash transfer program focused on Brazilian poor families, on children‘s nutritional status in a context of low monitoring of conditionalities. The analysis is carried out using data from a baseline survey conducted in 2005, the “Bolsa Família” Impact Evaluation Research. The evaluation is made using the Propensity Score Matchingtechnique. Besides considering observational differences between beneficiaries and non-beneficiaries of Bolsa Família Program, our empirical strategy also addresses some concerns about informational issues in order to take into account potential endogeneity of the decision to participate in the Program. Our final results show a positive effect of Bolsa Família Program on children nutritional status only whencontrolling for the informational bias and for those children fulfilling educational requirements; however, this positive impact on nutritional status is restricted to BMIfor-age but does not affect height-for-age.

How can research contribute to peacebuilding?

How do we know whether or not research contributes to peacebuilding? And, what kind of positive or negative impacts may research have? These two questions are conspicuous by their absence despite the large number of peacebuilding-specific research institutes, academic programmes, publishers and funders. Some of the reasons for the current state of affairs are examined: the ambiguous, elastic and politicised nature of ‘peacebuilding’ as a concept and as a practice; the inappropriateness of current approaches to the evaluation of research impact; and the disconnection between the world of evaluators and the world of peace researchers. The current article addresses these questions through an exploration of the intersection of research, evaluation methodology and politics. The article concludes with preliminary framework for teasing out the specific kinds of peacebuilding impacts catalysed through research.

Supply chain management research impact: an evidence‐based perspective

PurposeThe purpose of this paper, using an evidence‐based management theoretical lens, is to examine research impact to provide guidance to supply chain management academics in evidencing and exploiting the outputs, outcomes and impact of their research.Design/methodology/approachEvidence‐based management theory is examined and applied to types of academic research impact. The distinction between academic and non‐academic impact is developed into a supply chain framework of research outputs, transfer, outcomes, impact and national/international benefits. Impact of supply chain management research is explored through a case study in the English National Health Service. Future opportunities and challenges for supply chain management researchers arising from increasing demand for and supply of evidence are discussed.FindingsAuthor academic impact and citations are found to be increasingly important building blocks of evidence‐based evaluations of individual academics, journals, research quality assessments of groups and universities, and global rankings of universities. Supply chain management researchers can compare their impact with other areas of academia. Non‐academic impact of research has been assessed by funders of research projects and has spread to research quality assessments of universities.Social implicationsBibliometrics provide evidence of author and journal impact that can be used in human resource decisions, research quality assessments and global rankings of universities; this availability enables a debate on appropriate use of academic impact evidence. Supply chain management academics evidencing non‐academic research impact on business, society and economy will enable governments and funders of research to evaluate value for money return on their investment.Originality/ valueThis perspective of evidence‐based evaluation of research impact and its implications might encourage debate on academic and non‐academic impact and encourage supply chain researchers to consider evidencing impact in their research design and methodology.

Citation time window choice for research impact evaluation

This paper aims to inform choice of citation time window for research evaluation, by answering three questions: (1) How accurate is it to use citation counts in short time windows to approximate total citations? (2) How does citation ageing vary by research fields, document types, publication months, and total citations? (3) Can field normalization improve the accuracy of using short citation time windows? We investigate the 31-year life time non-self-citation processes of all Thomson Reuters Web of Science journal papers published in 1980. The correlation between non-self-citation counts in each time window and total non-self-citations in all 31 years is calculated, and it is lower for more highly cited papers than less highly cited ones. There are significant differences in citation ageing between different research fields, document types, total citation counts, and publication months. However, the within group differences are more striking; many papers in the slowest ageing field may still age faster than many papers in the fastest ageing field. Furthermore, field normalization cannot improve the accuracy of using short citation time windows. Implications and recommendations for choosing adequate citation time windows are discussed.

Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews

BackgroundHow to assess the impact of research is of growing interest to funders, policy makers and researchers mainly to understand the value of investments and to increase accountability. Broadly speaking the term "research impact" refers to the contribution of research activities to achieve desired societal outcomes. The aim of this overview is to identify the most common approaches to research impact assessment, categories of impact and their respective indicators.MethodsWe systematically searched the relevant literature (PubMed, The Cochrane Library (1990-2009)) and funding agency websites. We included systematic reviews, theoretical and methodological papers, and empirical case-studies on how to evaluate research impact. We qualitatively summarised the included reports, as well the conceptual frameworks.ResultsWe identified twenty-two reports belonging to four systematic reviews and 14 primary studies. These publications reported several theoretical frameworks and methodological approaches (bibliometrics, econometrics, ad hoc case studies). The "payback model" emerged as the most frequently used. Five broad categories of impact were identified: a) advancing knowledge, b) capacity building, c) informing decision-making, d) health benefits, e) broad socio-economic benefits. For each proposed category of impact we summarized a set of indicators whose pros and cons are presented and briefly discussed.ConclusionsThis overview is a comprehensive, yet descriptive, contribution to summarize the conceptual framework and taxonomy of an heterogeneous and evolving area of research. A shared and comprehensive conceptual framework does not seem to be available yet and its single components (epidemiologic, economic, and social) are often valued differently in different models.

What shapes research impact on policy? Understanding research uptake in sexual and reproductive health policy processes in resource poor contexts.

Assessing the impact that research evidence has on policy is complex. It involves consideration of conceptual issues of what determines research impact and policy change. There are also a range of methodological issues relating to the question of attribution and the counter-factual. The dynamics of SRH, HIV and AIDS, like many policy arenas, are partly generic and partly issue- and context-specific. Against this background, this article reviews some of the main conceptualisations of research impact on policy, including generic determinants of research impact identified across a range of settings, as well as the specificities of SRH in particular. We find that there is scope for greater cross-fertilisation of concepts, models and experiences between public health researchers and political scientists working in international development and research impact evaluation. We identify aspects of the policy landscape and drivers of policy change commonly occurring across multiple sectors and studies to create a framework that researchers can use to examine the influences on research uptake in specific settings, in order to guide attempts to ensure uptake of their findings. This framework has the advantage that distinguishes between pre-existing factors influencing uptake and the ways in which researchers can actively influence the policy landscape and promote research uptake through their policy engagement actions and strategies. We apply this framework to examples from the case study papers in this supplement, with specific discussion about the dynamics of SRH policy processes in resource poor contexts. We conclude by highlighting the need for continued multi-sectoral work on understanding and measuring research uptake and for prospective approaches to receive greater attention from policy analysts.

H-indices in a university department of anaesthesia: an evaluation of their feasibility, reliability, and validity as an assessment of academic performance

BackgroundThe h-index is a tool that is increasingly used to measure individual research productivity. It is unknown whether its use as an evaluation of individual research impact is reliable and valid within the context of anaesthesia. MethodsWe calculated the h-indices of 268 faculty members of a university department of anaesthesia using Scopus™ and Web of Science®. Agreement between the databases was investigated with a Bland–Altman plot. The construct validity was examined by comparing the h-indices for faculty grouped by academic rank. ResultsThe mean bias between the Scopus™ and Web of Science®h-indices was 0.09 but 1.96 sd limits of agreement were −5.7 to 5.9. The Web of Science®-derived h-indices showed a statistically significant difference between the different academic ranks (P<0.001): median h-indices were 0 for lecturers, 2 for assistant professors, 9 for associate professors, and 16 for full professors. The Scopus™-derived h-indices also showed a statistically significant difference between the different academic ranks (P<0.001): median h-indices were 0 for lecturers, 1 for assistant professors, 9 for associate professors, and 17 for full professors. Post hoc testing found statistically significant differences in all comparisons between academic ranks (all P<0.01). Ignoring self-citations did not affect construct validity of the h-index. We found no evidence that the h-index is superior to counting the total number of publications. ConclusionsAgreement between the two databases was problematic. There was evidence of construct validity; however, the overlap between academic ranks limits the discriminative power of a low h-index.

Does nursing research impact on policy? A case study of health visiting research and UK health policy

The aim of this study was to critically examine the impact of nursing research on the development of health care policy using UK health visiting research as an example. We used established methods to evaluate research impact. This included a documentary review of over 30 policy documents, citation analyses on 19 papers and interviews with health visiting researchers. Although there were examples of policy documents being informed by health visiting research it was not always clear what role research had played in the development of recommendations. Information from researchers provided examples of local, national and international impact, although the extent to which papers may have impacted upon policy was less clear from the citation analyses. Many of the UK studies cited in policy documents were qualitative, observational or reflexive and a lack of evaluative research, in particular randomised controlled trials and other controlled evaluations, may limit the impact of health visiting research on health care policy in the UK. There is evidence that health visiting research has influenced health care policy but this has been limited and there is a need for more research to underpin and inform the role of the health visitor.

Defining and evaluating the impact of cross-disciplinary conservation research

SUMMARYCross-disciplinary research is advocated as a way of improving understanding of the complexity of environmental problems; cross-disciplinary projects, centres and academic institutes have increased. However, there is confusion over the nature of cross-disciplinary research. Through review of papers defining themselves as cross-disciplinary that aim to address conservation problems, and by standardizing the definition of cross-disciplinary research, it is possible to evaluate the potential research impact on peers and practitioners. When papers were reclassified by authors, those reclassified as transdisciplinary were perceived to have a greater impact on practitioners, and those reclassified as non cross-disciplinary had the greatest impact on colleagues. Having clear definitions for types of cross-disciplinary research would help establish a firm foundation, not only for improving research quality, but also for evaluating research impact. While the number of cross-disciplinary studies is increasing, cross-disciplinary research falls short of integrating disciplinary methods in much depth and does not have much impact on participants outside of academia.

Beyond the “Toolkit Approach”: Arts Impact Evaluation Research and the Realities of Cultural Policy‐Making

This article presents a reflection on the possibility and potential advantages of the development of a humanities‐based approach to assessing the impact of the arts, which attempts to move away from a paradigm of evaluation based on a one‐size‐fits‐all model usually reliant on empirical methodologies borrowed from the social sciences. A “toolkit approach” to arts impact assessment, as the article argues, demands excessive simplifications, and its popularity is linked to its perceived advocacy potential rather than to any demonstrable contribution it may make to a genuine understanding of the nature and potential effects of artistic engagement. The article also explores the relationship between research, advocacy and the actual realities of policy‐making with a view to proposing a critical research agenda for impact evaluation based on Carol Weiss’s notion of the “enlightenment” function of policy‐oriented research. In particular, the article attempts to highlight the contribution that cultural policy scholars working within the humanities could make to this area of policy research.

Urban Poverty in Developing Countries: A Scoping Study for Future Research

This paper is a scoping exercise to explore options for research on urban poverty in developing countries. Based on a review of the literature and experience, the first part of the paper reviews the changing nature of the urban poverty agenda over the last 30 years by comparing today’s issues with those addressed in World Development Report 1979. The issues are categorized by new issues, old issues still relevant, and forgotten issues; and are grouped by challenges, instruments, analytical and planning tools, data and benchmarks, regions/countries/cities, and institutions. The conclusion of this part of the paper is that while some of the old issues and prescriptions are still valid today, and some important ones apparently forgotten, there are many new issues and approaches that need now be considered. The second part of the paper takes the large array of issues identified in the first section and selects 10 for more detailed consideration as part of a menu of potential high-priority research initiatives on urban poverty. They include slums, employment, small- and medium-size cities, and women/children/youth (from among the challenges); comprehensive slum upgrading programs, conditional cash transfer programs, and incentive-based approaches (from among instruments); happiness and life satisfaction research approaches, impact evaluation, and strategic planning tools (from among analytical and planning tools). Among regions, countries and cities, Africa is recommended as a regional focus, China and India for country studies, and Bogota (or possibly other large cities) for specific city studies.