Declaration On Research Assessment Research Articles

Are Narrative CVs contributing towards shifting research culture? Workshop Report from the 2023 Recognition and Rewards Festival

Abstract* Background Over the past decade, calls for research assessment reform have grown, led by initiatives such as the Declaration on Research Assessment (DORA) and the Leiden Manifesto, and, more recently, the Coalition for Advancing Research Assessment (CoARA). A key element being discussed as part of research assessment reform is a shift towards more qualitative assessments, focussed on the content of research and the broad skills and competencies of researchers, and the array of contributions they make to knowledge creation and innovation. Narrative CV formats have emerged as a good practice example for enabling qualitative assessments of research projects and researchers, and are becoming more widely piloted and implemented. Methods As part of the 2023 Dutch Recognition and Rewards Festival, the authors hosted a workshop to gather perspectives on Narrative CVs, including whether and how they may contribute to shifts in research culture that are needed to support research assessment reform. Results Participants, representing research organisations and the research community, discussed both beneficial and critical aspects of narrative-style CV implementations from their experiences. The effects observed since narrative CVs have been implemented were discussed, with perspectives provided on career prospects and the empowerment of the research community to direct change. Finally, the discussion turned to expectations for the future, with workshop participants calling for focus on the roles that narrative-style CVs can play in improving research careers, recognition of collaborative work, and equality, diversity, and inclusion. A short informal survey exploring levels of implementation of narrative CVs across different research organisations was run prior to the workshop, the results of which are also presented as part of this report. Discussion The authors intend to expand this discussion to other scientific and policy conferences, and this report serves as a basis for a wider and deeper dialogue in the community.

A broader view of research contributions: Necessary adjustments to DORA for hiring and promotion in psychology.

Recently Schönbrodt et al. (2022) released recommendations for improving how psychologists could be evaluated for recruitment, retention, and promotion. Specifically, they provided four principles of responsible research assessment in response to current methods that rely heavily on bibliometric indices of journal quality and research impact. They build their case for these principles on the San Francisco Declaration on Research Assessment (DORA) perspective that decries reliance on invalid quantitative metrics of research quality and productivity in hiring and promotion. The paper makes clear the tension panels have to address in evaluating applications—too little time to do an in-depth evaluation of an individual’s career and contribution, so reliance on easy to understand, but perhaps invalid, metrics. This dilemma requires an alternative mechanism rather than simply a rejection of metrics. To that end, the authors are to be congratulated for operationalising what those alternatives might look like. Nonetheless, the details embedded in the principles seem overly narrow and restrictive.

Practicing responsible research assessment: Qualitative study of faculty hiring, promotion, and tenure assessments in the United States

Abstract Recent times have seen the growth in the number and scope of interacting professional reform movements in science, centered on themes such as open research, research integrity, responsible research assessment, and responsible metrics. The responsible metrics movement identifies the growing influence of quantitative performance indicators as a major problem and seeks to steer and improve practices around their use. It is a multi-actor, multi-disciplinary reform movement premised upon engendering a sense of responsibility among academic evaluators to approach metrics with caution and avoid certain poor practices. In this article we identify how academic evaluators engage with the responsible metrics agenda, via semi-structured interview and open-text survey responses on professorial hiring, tenure and promotion assessments among senior academics in the United States—a country that has so far been less visibly engaged with the responsible metrics reform agenda. We explore how notions of ‘responsibility’ are experienced and practiced among the very types of professionals international reform initiatives such as the San Francisco Declaration on Research Assessment (DORA) are hoping to mobilize into their cause. In doing so, we draw on concepts from science studies, including from literatures on Responsible Research and Innovation and ‘folk theories’ of citation. We argue that literature on citation folk theories should extend its scope beyond simply asking researchers how they view the role and validity of these tools as performance measures, by asking them also what they consider are their professional obligations to handle bibliometrics appropriately.

Research Assessment Reform, Non-Traditional Research Outputs, and Digital Repositories: An Analysis of the Declaration on Research Assessment (DORA) Signatories in the United Kingdom

Objective – The goal of this study was to better understand to what extent digital repositories at academic libraries are active in promoting the collection of non-traditional research outputs. To achieve this goal, the researcher examined the digital repositories of universities in the United Kingdom who are signatories of the Declaration on Research Assessment (DORA), which recommends broadening the range of research outputs included in assessment exercises. Methods – The researcher developed a list of 77 universities in the UK who are signatories to DORA and have institutional repositories. Using this list, the researcher consulted the public websites of these institutions using a structured protocol and collected data to 1) characterize the types of outputs collected by research repositories at DORA-signatory institutions and their ability to provide measures of potential impact, and 2) assess whether university library websites promote repositories as a venue for hosting non-traditional research outputs. Finally, the researcher surveyed repository managers to understand the nature of their involvement with supporting the aims of DORA on their campuses. Results – The analysis found that almost all (96%) of the 77 repositories reviewed contained a variety of non-traditional research outputs, although the proportion of these outputs was small compared to traditional outputs. Of these 77 repositories, 82% featured usage metrics of some kind. Most (67%) of the same repositories, however, were not minting persistent identifiers for items. Of the universities in this sample, 53% also maintained a standalone data repository. Of these data repositories, 90% featured persistent identifiers, and all of them featured metrics of some kind. In a review of university library websites promoting the use of repositories, 47% of websites mentioned non-traditional research outputs. In response to survey questions, repository managers reported that the library and the unit responsible for the repository were involved in implementing DORA, and managers perceived it to be influential on their campus. Conclusion – Repositories in this sample are relatively well positioned to support the collection and promotion of non-traditional research outputs. However, despite this positioning, and repository managers’ belief that realizing the goals of DORA is important, most libraries in this sample do not appear to be actively collecting non-traditional outputs, although they are active in other areas to promote research assessment reform.

Editorial

Ten Years of ChemBioEng Reviews Dear readers, authors, and reviewers, Welcome to 2023 – the year that marks the tenth anniversary of ChemBioEng Reviews. No one of us could have anticipated what the decade ahead had in store for us when Barbara Boeck and Susanne Mueller launched the journal in 2014. Back then we were already at the beat of the time with topics ranging from process development and optimization over separation technologies to sustainable production and renewable resources. And we kept on improving ChemBioEng Reviews. We started right in the first year with being the trailblazer for the chemical engineering family receiving enhanced HTML coverage, which is also known as “Anywhere Article”. This feature offered the advantages of availability and adaptability of the content to various end devices for making the readers' experience interactive and more flexible. The next step in improving our availability and the visibility of our articles was the introduction of Virtual Special Issues. Those special issues were growing over the years and can be accessed on our website (www.ChemBioEngRev.de). In 2016 we benefitted from being part of Wiley when the agreement between the Association of Universities in the Netherlands (VSNU) and Wiley was signed as the first transitional agreement for open access publishing. Since then, several other agreements followed, e. g., with Germany, Austria, Hungary, Norway, Sweden, Finland, and American consortia, giving researchers the chance to publish open access with us easily. When everyone was forced to pause and the world felt like it stood still in 2020 due to the pandemic, we finally received our first impact factor. That was great news for us as well as for you, our authors and readers. Being indexed in the Web of Science helped us to gain even more visibility and with that more people were accessing our published review articles. But as we all know, the impact factor is not always the best way to describe the worth and novelty of scientific work. That is why Wiley signed the Declaration on Research Assessment (DORA), for a barrier-free, divers, and bias-free publishing of research. We are happy to start with you into a new year. The hot topics back in 2014 are still relevant and thus, we also start in 2023 with a range of subjects from biofuels over wastewater treatment to CO2 capture. Stay tuned for the rest of the year when we will bring you an even broader variety of topics covering machine learning, industrial fouling, and sodium-ion batteries, amongst others. Stay safe and healthy! Yours sincerely Arno Knappschneider (Associate Editor) and Barbara Boeck (Editor-in-Chief) Arno Knappschneider Barbara Boeck

Progress and Changes

Dear Crystal Research & Technology readers, The Editorial of issue 1 is always the perfect opportunity to recap the last year, report on the progress of the journal, and announce some changes. Since the journal was founded back in 1966, we have a long journey to look back on (more details about the history can be found in the “55 Years of Crystal Research & Technology” editorial). For 57 years now, Crystal Research & Technology has been publishing original research and reviews in experimental, industrial and theoretical crystallography, crystal growth and modern materials science. In the very first Editorial, the editor wrote: “It is the conviction of the editors and the publisher that a ‘Journal of Experimental and Technical Crystallography’ thematically oriented in the manner indicated, will find a lively response because it is particularly committed to both research and application and thus to the connection between the two.” We can state that this, indeed, is still true and proves the success story of the journal. We are pleased to have published 110 articles in 2022 and would like to take this opportunity to mention some of the highlights here. As such, Table 1 shows a list of the most cited articles according to Web of Science in 2022, while Table 2 gives an overview of the most accessed articles in 2022. First-Principles Calculations of Electronic Structure and Optical Properties of Si-Doped and Vacancy β-Ga2O3 (Research Article) We are confident that Crystal Research & Technology has solidified, and will continue to strengthen, its position as an important journal in the field of crystal research. We are proud that Crystal Research & Technology has received one of the highest Impact Factors in its history, but in addition to this journal metric, Wiley signed the Declaration on Research Assessment (DORA) in May to include further metrics that measure levels of impact beyond citation value, including usage, re-use, reproducibility, peer review assessment, geographic reach, and public recognition through references in the media. That will enhance responsible research assessment practices and support our mission around open practices and transparency. With DORA, we will deliver more ways to assess and recognize research outputs, which in turn supports healthy scholarship and allows more researchers to thrive in their careers. Furthermore, the transition to open access (OA) continues to be an important driver for the publication of research results. For authors, there are various benefits, such as more downloads, more citations, and better Altmetric attention of their articles. Wiley has several agreements with Funders and Institutions to help authors publish open access and cover the associated Article Publication Charges. Authors who receive funding from an agency or institution with a Wiley Open Access Account do not pay directly. The charge is paid by the institution or funder. Authors whose institutions have paid the Wiley Open Access partner fee are eligible for a discount on the open-access publication charge, and on acceptance a discounted fee is payable by the author. If you are not sure about your institution or funding agency open-access policy: find out more with Wiley's Author Compliance Tool. Moreover, to make submitting as easy as possible, we now offer free format submission. Your time is valuable and should be spent on your research, not re-typing reference lists. Free format submission helps make getting your manuscript ready to submit easier and faster. During 2023, we look forward to publishing several Review articles on recent topics. We also look forward to a combined special issue of the 3rd International Symposium on Modeling of Crystal Growth Processes and Devices (MCGPD-2023) and the 2nd Indo-Japan Joint Workshop on Photovoltaics (IJWP-2023). Last but not least, we would like to welcome onboard our new Crystal Research & Technology Advisory Board members: Hanna A. Dabkowska, McMaster University, Canada Philomela Komninou, Aristotle University of Thessaloniki, Greece Zulipiya Shadike, Shanghai Jiao Tong University, China Jiayue Xu, Shanghai Institute of Technology, China Wenkun Zhu, Southwest University of Science and Technology Mianyang, China. We are well prepared and highly motivated to take the next steps into the future. With the help of our Advisory Board Members, Guest Editors, Reviewers, and of course, our Authors, we are confident that Crystal Research & Technology will have a great 2023 – and we already look forward to celebrating its 60th anniversary in three years. Best wishes to you all for a healthy and productive year 2023! On behalf of the editorial team, Marc Zastrow, Katy Lally, Heike Höpcke, Pejun Tu

Annalen der Physik – Wiley's Physics Forum and Center of Excellence

<i>Annalen der Physik</i> – Wiley's Physics Forum and Center of Excellence

Covering the Broad Spectrum of Intelligent Systems: From Neuromorphic Engineering to Robotics

Covering the Broad Spectrum of Intelligent Systems: From Neuromorphic Engineering to Robotics

Article‐level Metrics, Transparency and Open Research

Article‐level Metrics, Transparency and Open Research

News and perspectives for Annals of Applied Biology in 2023: Looking forward to more and more submissions

News and perspectives for <scp><i>Annals of Applied Biology</i></scp> in 2023: Looking forward to more and more submissions

Open Science: Challenges, Possible Solutions and the Way Forward

Everyone agrees that scientific communication should be free for all. Unfortunately, accessing publications from many reputed journals comes at a high cost—a cost that many researchers and institutions cannot afford. Although, open-access publication model is considered by many as a possible route to ensure that science is free for all; however, it is fraught with its own challenges. This review attempts at exploring the possibilities of keeping science accessible. Firstly, we re-visit the meaning of “open science” as a comprehensive concept which includes open source, data, access, resources, peer review etc. and not merely open access publication model. Next, we have discussed the global initiatives towards open access—the Budapest Open Access Initiative, Bethesda Statement on Open Access Publishing, Berlin Declaration on Open Access to Knowledge in the Sciences and Humanities, cOAlition S and its Plan S initiative, UNESCO Recommendation on Open Science and the San Francisco Declaration on Research Assessment (DORA). Following this we have included the various open access initiatives from India. In the next part, we have focused on problems with dissemination of scientific outcomes and the challenges associated with existing publication models. Finally, we explore the possible solutions to the existing challenges, which include promotion of pre-print servers and other ideas that we have detailed in the manuscript.

The intersections between DORA, open research and equity

The San Francisco Declaration on Research Assessment (DORA) is a campaigning initiative to improve the ways that we evaluate research and researchers. It aims particularly to help people understand the problems associated with over-reliance on aggregate metrics like the journal impact factor or H-index in assessment processes. Such metrics have enduring appeal because they appear to offer the simplicity and objectivity of numerical analyses. However, we need to be mindful of the subjective nature of decisions that lead to citation counts – the raw material of many performance metrics – and the biases that perturb them. The challenge now, if we are to move to more robust and equitable forms of evaluation, is to ensure that these are as effective and as efficient as possible. Embracing this challenge will also help to clear the way for more open and impactful science, and for a more inclusive academy.

Health Care Science —Why another journal and why we will be different?

<i>Health Care Science</i> —Why another journal and why we will be different?

Implementing the Declaration on Research Assessment: a publisher case study

There has been much debate around the role of metrics in scholarly communication, with particular focus on the misapplication of journal metrics, such as the impact factor in the assessment of research and researchers. Various initiatives have advocated for a change in this culture, including the Declaration on Research Assessment (DORA), which invites stakeholders throughout the scholarly communication ecosystem to sign up and show their support for practices designed to address the misuse of metrics. This case study provides an overview of the process undertaken by a large academic publisher (Taylor &amp; Francis Group) in signing up to DORA and implementing some of its key practices in the hope that it will provide some guidance to others considering becoming a signatory. Our experience suggests that research, consultation and flexibility are crucial components of the process. Additionally, approaching signing with a project mindset versus a ‘sign and forget’ mentality can help organizations to understand the practical implications of signing, to anticipate and mitigate potential obstacles and to support cultural change.

The San Francisco Declaration on Research Assessment (DORA) in Nursing Science

The San Francisco Declaration on Research Assessment (DORA) in Nursing Science

Capturing and Articulating Visual Media as Scholarship.

Recent advancements in social media and educational technologies have facilitated dissemination of information in visual formats to broad audiences. When academic and creative visual media work is of high quality and accessible via various platforms (eg, in journals, on websites), it can meet the definition of scholarship.However, visual scholarship is sometimes viewed as an afterthought. For example, even venues that include visual abstracts with manuscripts may encourage authors to create visual abstracts post-submission to promote the article rather than as an independent scholarly pursuit. Typically, annual performance reviews, accreditation requirements, and promotion and tenure processes do not recognize or reward visual media as scholarly activity, despite the considerable technical and pedagogical skills required for its creation.The updated Accreditation Council for Graduate Medical Education (ACGME) Common Program Requirements define scholarship as including discovery, application, and teaching.1 This position builds on antecedent work by Boyer and Glassick.2,3The Declaration on Research Assessment (DORA) has encouraged scholars to equitably acknowledge each form of scholarship by using specific quality metrics, rather than relying on surrogate measures such as impact factor.4 DORA explores how we can value various forms of visual scholarship as meaningful contributions to the medical and medical education communities. For example, infographics and tweetorials can be examined by dissemination measures using journal-based or platform-based metrics such as Altmetric or Twitter analytics, respectively.5

Research on Covid-19: a disruptive phenomenon for bibliometrics.

The Covid-19 pandemic has been the highest disruptive event in the world recent history. Worldwide academic research on this topic has led to an explosion of scientific literature, never seen before. Bibliometrics provide methods to illustrate this exceptional phenomenon in academic publications. The objective of this paper is to analyze the Covid-19 research from a bibliometric perspective and to study the impact of the publication explosion on bibliometric indicators. The present study shows how an exceptional phenomenon has a disruptive impact on bibliometric indicators, such as the h-index and the Journal Impact Factor. The higher the specialization, the higher the possible impact of a disruptive phenomenon. In applied sciences, more important than the research or the discipline, the specific theme of the research is crucial for citations of articles and for their impact. The salience of the topic, the magnitude of the problem at study and the urgency to find solutions are drivers for citations. The study of the Covid-19 research illustrates the relativity of indicators and the need for context. The present study also confirms the plead for responsible metrics of the San Francisco Declaration on Research Assessment (DORA).

Implementing the Global University Publications Licence: a new open scholarship model for advocating change

Universities want a voluntary, non-exclusive licence from authors to disseminate publications. This practitioner case study explores an innovative model to communicate and advance open and equitable scholarship through the implementation of the Global University Publications Licence at the University of Nottingham Ningbo China. This article explains the licensing policy and key influences, including, the copyright law of the People’s Republic of China and the Declaration on Research Assessment (DORA). The University approved the Global University Publications Licence, with implementation from 1 August 2019. It is available in Chinese and English. Since implementation, the University has retained rights for 74% of research publications submitted. 100% of those publications are available through the University with a CC-BY licence and zero embargo. The open scholarship model provides an equitable approach to versions and citation. The article concludes by suggesting university libraries can exploit copyright law in China to progress open scholarship strategies, including recognition of employers as authors of works, a priority right to the exploitation of works and an embargo protection of two years after the completion of the work. The author’s final version of publications can be open, discoverable, cited and preserved through trusted universities with global reputations for high-quality research.

Journal- or article-based citation measure? A study of academic promotion at a Swiss university.

In academia, decisions on promotions are influenced by the citation impact of the works published by the candidates. The Medical Faculty of the University of Bern used a measure based on the journal impact factor (JIF) for this purpose: the JIF of the papers submitted for promotion should rank in the upper third of journals in the relevant discipline (JIF rank >0.66). The San Francisco Declaration on Research Assessment (DORA) aims to eliminate the use of journal-based metrics in academic promotion. We examined whether the JIF rank could be replaced with the relative citation ratio (RCR), an article-level measure of citation impact developed by the National Institutes of Health (NIH). An RCR percentile >0.66 corresponds to the upper third of citation impact of articles from NIH-sponsored research. We examined 1525 publications submitted by 64 candidates for academic promotion at University of Bern. There was only a moderate correlation between the JIF rank and RCR percentile (Pearson correlation coefficient 0.34, 95% CI 0.29-0.38). Among the 1,199 articles (78.6%) published in journals ranking >0.66 for the JIF, less than half (509, 42.5%) were in the upper third of the RCR percentile. Conversely, among the 326 articles published in journals ranking <0.66 regarding the JIF, 72 (22.1%) ranked in the upper third of the RCR percentile. Our study demonstrates that the rank of the JIF is a bad proxy measure for the actual citation impact of individual articles. The Medical Faculty of University of Bern has signed DORA and replaced the JIF rank with the RCR percentile to assess the citation impact of papers submitted for academic promotion.

Ethical Theories in Research Evaluation: An Exploratory Approach

Research evaluation encompasses the practices of assessing research quality and impact at various stages of research. The processes and criteria of research evaluation vary depending on the nature and objectives of the assessment. Different research evaluation systems influence the research strategies of universities and institutes. There are, however, some known issues of research evaluation with regards to the peer review and, most prominently, the use of citation-based metrics, which lead to recent calls for responsible use of metrics. In this paper, we argue that there is a need for ethical theories for considering research evaluation and that research evaluation ethics, as an overlapping area between research ethics and evaluation ethics, deserve its own treatment. The core of the article consists of a discussion of the most influential ethical theories in the context of the research evaluation, including the deontological ethics, the consequentialist ethics and the virtue ethics. The aim is to highlight the need to assume an ethical view that combines the deontological and the consequentialist concepts, adopting ‘common good’ as the most likely pillar for the research evaluation procedures. We propose that the mixed approach would be useful for developing a framework for research evaluation ethics and for analysing ethical approaches and ethical dilemmas in research evaluation. Policy HighlightsThe misuses and abuse of evaluative metrics have been discussed and debated in many high-profile publications including the San Francisco Declaration on Research Assessment (DORA), The Metrics Tide, the Leiden Manifesto, and the Hong Kong Principles. There are also many studies stating the limitations of and bias in peer review.The debates and discussions, however, have not been explored in light of ethical theories. The article considers also good practices in evaluation, including the American Evaluation Association Guiding Principles for Evaluators (AEA 2018), the Australasian Evaluation Society Guidelines for the ethical conduct of evaluations (AES 2000, 2010, 2013) the UK Evaluation Society Guidelines for Good Practices in Evaluation (UK 2019) and the United Nations Ethical Guidelines for Evaluation (UNEG 2008).The paper argues that ethical theories are useful in understanding ethical assumptions and ethical dilemmas in research evaluation and are pertinent in future design and development of research evaluation processes and criteria.Ethical theories that can construct ethical principles for research evaluation, including deontological and consequentialist ethics, taking into account the Mertonian normative theory, have been examined.In order to address the issues of research evaluation, we propose a mixed approach that combines the deontological and the consequentialist concepts that is able to infringe the boundaries of the rivaling theories and provide basis needed for research evaluation ethics.