Using Peer Review to Evaluate the Societal Relevance of Humanities Research

Early career researchers have questions about peer review—we asked the <scp><i>ASLO</i></scp> editors for answers

How the Biochemical Society and Portland Press are engaging with and supporting early career researchers

Purpose: This paper describes the impact of external crises, i.e. the pandemic, the war in Ukraine, inflation and the reform of science and higher education on the work of early career researchers. Methods: The respondent group comprised early career researchers in humanities, theology and art. A qualitative analysis of free statements provided in the first round of in-depth interviews was performed. The Polish project (2023-) is a continuation of two international Harbingers projects, which included longitudinal interviews with early career researchers and dealt with changes in technology (2016-2018) and the pandemic (2020-2022). Results: Quotations from the interviews conducted in spring 2023 were used to describe the impact of external crises on the work of early career researchers in the three fields of science. The interviewees usually mentioned being overworked, sometimes – undervalued and, less frequently – the feeling of professional burnout. Financial problems of the young generation of Polish scientists were also brought up. Value: An attempt was made to characterise the conditions of work of Polish early career researchers in art, humanities and theology.

The article reports results that compare the self‐perceptions of Chinese early career researchers (ECRs) with those of senior career researchers (SCRs) as peer reviewers, aiming to show the characteristics of the increasing number of Chinese ECRs as reviewers. An investigation was conducted with 207 Chinese researchers (including 116 ECRs and 91 SCRs) using an online questionnaire from October to November 2021. The results show the differences and similarities between ECRs and SCRs in five dimensions: views, motivations, preferences, behaviours and self‐confidence. It was found that ECRs are more inclined than SCRs to regard peer review as an extra‐role responsibility. ECRs' self‐focused motivations for peer review are stronger than SCRs' self‐focused motivations, while other‐focused motivations of ECRs and SCRs are equally strong. ECRs are more willing to accept review invitations from high‐quality indexed journals (such as SCI/SSCI/A&amp;HCI journals) than SCRs. In terms of peer review behaviour, ECRs and SCRs have a similar degree of conscientiousness and agreeableness. In addition, ECRs are more lacking in self‐confidence as reviewers than SCRs.

Scientific journals, like the Scandinavian Journal of Work, Environment and Health, are dependent on their peer reviewers. Without a rigorous peer review of submitted papers, there are no publications and no journals. As researchers, we know that a critical and constructive evaluation of research in the review process is essential to move the science in our field forward. While worldwide the academic peer review system is under pressure (1–3), a rigorous peer review is more important than ever before for three reasons we highlight below: (i) an exponential increase in scholarly journals; (ii) the reduced reliance on facts; and (iii) the intensification of work. In the past years, researchers have received increasingly review requests from what seems a sheer flood of journals from all over the world. As shown in figure 1, the number of scientific journals has increased exponentially (4). Within this increase, we see robust scientific journals but also predatory ones, often characterized by high article processing charges, false claims of peer review, and the unethical listing of academics (or even fake scholars) on editorial boards without their consent (5). To date, www.predatoryjournals.org – a website that provides information and resources on predatory publishing practices – lists 1363 publishers and 2780 journals. Moreover, artificial intelligence (AI) technologies make it increasingly difficult for researchers and society to distinguish between facts (ie, evidence) and fiction. Robust peer reviews and high-quality journals are thus extra important nowadays. The proper use of AI in the writing, reviewing and publishing process is for all of us, editors, reviewers and researchers alike, a new challenge we have to carefully address. Clear guidelines of when AI can or cannot be used are in continuous development (see among others the BMJ www.bmj.com/content/ai-use). Researchers use AI tools for support in writing introductory paragraphs and perhaps in other parts of studies. This proliferation of AI tools will likely result in an increased number of papers. At the same time, we would like to stress that the Scandinavian Journal of Work, Environment and Health prohibits reviewers from uploading manuscripts to AI platforms for review purposes as this clearly breach confidentiality, thus the availability of AI tools will likely be only of limited help in the review process. In addition to the tsunami of submitted papers, finding peer reviewers is another bottleneck in the academic peer review system. This is likely related to the intensification of work over the past decades. Society has become more globalized and digitalized, both increasing work pace, with major implications for the 24/7 work environment, making the working life more demanding and intense (6). Academics are experiencing high levels of work pressure (7), which can reduce their availability and willingness to participate in peer reviews (2). Yet, at a time when science is under increasing global pressure—from the rise of misinformation to sweeping policy changes and budget cuts—the need for skilled peer reviewers has never been greater. A robust and thorough review process is essential to uphold research quality in these changing times (8). To achieve this, we must build a large and diverse pool of reviewers that reflects the full breadth of researchers and research in our field. With this editorial, we would like to send out a call for peer review skills development in a mentor–mentee relationship. We at the Journal, but also the entire research community, need the next generation(s) of researchers to learn and apply peer review skills as we strongly believe they are crucial for good science. Senior researchers have many years of valued reviewing experience, and it is exactly this expertise they can pass on to their early-career colleagues. We propose to prioritize mentor–mentee relationships and approach performing a peer review as a necessary cornerstone of the competence and skill development of emerging researchers. As the peer-review process is often not taught by graduate schools or PhD programs, it is the responsibility of the individual PhD supervisor or research leader to teach or promote the important skill of providing constructive feedback which is really different from writing a scientific paper. We recognize that mentoring a new reviewer requires time and commitment from both the mentor and the mentee, as it is a learning process. However, this investment will pay off by helping to cultivate the next generation of thoughtful, critical, and constructive peer reviewers. Reviews conducted through a mentor–mentee collaboration may also lead to higher-quality outcomes, as each reviewer brings a fresh perspective—and no single reviewer can catch everything. In a time of growing societal misinformation, peer review must remain a high priority to safeguard the integrity of research. When it comes to the peer-review process itself, numerous checklists and guidelines exist, but our main message and call for action is to prioritize peer review tasks. In particular, we encourage early-career researchers to collaborate with their senior colleagues in the peer-review process. In 2024, Fagher &amp; Verhagen (9) insightfully summarized the advantages of early-career researchers contributing to peer review. Peer review is an important academic skill that should be incorporated into eg, PhD portfolios. The Scandinavian Journal of Work, Environment and Health provides early-career researchers with the opportunity to conduct peer reviews under the responsibility of a senior researcher. We hope this editorial contributes to our peer review system by motivating senior researchers to mentor early-career researchers, and by stimulating early-career researchers to develop their peer review skills. Especially, we hope that all our readers are now extra eager to prioritize peer review tasks and designate their precious time for peer review in good-standard journals. Together we can make it work!

ABSTRACTThe Harbingers study of early career researchers (ECRs) and their work life and scholarly communications began by studying generational—Millennial—change (H‐1), then moved to pandemic change (H‐2) and is now investigating another change agent—artificial intelligence (AI). This paper from the study constitutes a deep dive into the peer review attitudes and practices of 91 international ECRs from all disciplines. Depth interviews were the main means by which data was collected, and questions covered ECRs as reviewers, authors and readers, and are described in their own words. Main findings are: (1) ECRs proved to be a highly experienced in peer review; (2) There is more trust in peer review than distrust in it, but there are concerns; (3) Peer review is something that arts and humanities ECRs are unfamiliar with or much concerned about; (4) A sizeable majority of ECRs thought peer review could be improved, with anonymity/double‐blind reviewing topping the list; (5) The majority view was that AI will have an impact on peer review and that it would be beneficial; (6) little has changed since the last Harbingers study, except for AI, which is seen to be transformative. We believe that few studies have drilled down so deeply and widely in respect to ECRs.

BackgroundThe peer review process has been questioned as it may fail to allow the publication of high-quality articles. This study aimed to evaluate the accuracy in identifying inadequate reporting in RCT reports by early career researchers (ECRs) using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process.MethodsWe performed a cross-sectional diagnostic study of 119 manuscripts, from BMC series medical journals, BMJ, BMJ Open, and Annals of Emergency Medicine reporting the results of two-arm parallel-group RCTs. One hundred and nineteen ECRs who had never reviewed an RCT manuscript were recruited from December 2017 to January 2018. Each ECR assessed one manuscript. To assess accuracy in identifying inadequate reporting, we used two tests: (1) ECRs assessing a manuscript using the COBPeer tool (after completing an online training module) and (2) the usual peer-review process. The reference standard was the assessment of the manuscript by two systematic reviewers. Inadequate reporting was defined as incomplete reporting or a switch in primary outcome and considered nine domains: the eight most important CONSORT domains and a switch in primary outcome(s). The primary outcome was the mean number of domains accurately classified (scale from 0 to 9).ResultsThe mean (SD) number of domains (0 to 9) accurately classified per manuscript was 6.39 (1.49) for ECRs using COBPeer versus 5.03 (1.84) for the journal’s usual peer-review process, with a mean difference [95% CI] of 1.36 [0.88–1.84] (p < 0.001). Concerning secondary outcomes, the sensitivity of ECRs using COBPeer versus the usual peer-review process in detecting incompletely reported CONSORT items was 86% [95% CI 82–89] versus 20% [16–24] and in identifying a switch in primary outcome 61% [44–77] versus 11% [3–26]. The specificity of ECRs using COBPeer versus the usual process to detect incompletely reported CONSORT domains was 61% [57–65] versus 77% [74–81] and to identify a switch in primary outcome 77% [67–86] versus 98% [92–100].ConclusionsTrained ECRs using the COBPeer tool were more likely to detect inadequate reporting in RCTs than the usual peer review processes used by journals. Implementing a two-step peer-review process could help improve the quality of reporting.Trial registrationClinical.Trials.govNCT03119376 (Registered April, 18, 2017).

A Review of:&#x0D; Nicholas, D., Boukacem-Zeghmouri, C., Rodríguez-Bravo, B., Xu, J., Watkinson, A., Abrizah, A., Herman, E., &amp; Świgoń, M. (2017). Where and how early career researchers find scholarly information. Learned Publishing, 30(1), 19-29. http://dx.doi.org/10.1002/leap.1087&#x0D; &#x0D; Abstract &#x0D; &#x0D; Objective – To examine the attitudes and information behaviours of early career researchers (ECRs) when locating scholarly information. &#x0D; &#x0D; Design – Qualitative longitudinal study.&#x0D; &#x0D; Setting – Research participants from the United Kingdom, United States of America, China, France, Malaysia, Poland, and Spain.&#x0D; &#x0D; Subjects – A total 116 participants from various disciplines, aged 35 and younger, who were holding or had previously held a research position, but not in a tenured position. All participants held a doctorate or were in the process of earning one.&#x0D; &#x0D; Methods – Using structured interviews of 60-90 minutes, researchers asked 60 questions of each participant via face-to-face, Skype, or telephone interviews. The interview format and questions were formed via focus groups. &#x0D; &#x0D; Main Results – As part of a longitudinal project, results reported are limited to the first year of the study, and focused on three primary questions identified by the authors: where do ECRs find scholarly information, whether they use their smartphones to locate and read scholarly information, and what social media do they use to find scholarly information. Researchers describe how ECRs themselves interpreted the phrase scholarly information to primarily mean journal articles, while the researchers themselves had a much expanded definition to include professional and “scholarly contacts, ideas, and data” (p. 22). &#x0D; &#x0D; This research shows that Google and Google Scholar are widely used by ECRs for locating scholarly information regardless of discipline, language, or geography. Their analysis by country points to currency and the combined breadth-and-depth search experience that Google provides as prime reasons for the popularity of Google and Google Scholar. Of particular interest is the popularity and use of Google Scholar in China, where it is officially blocked but accessed by ECRs via proxy services. Other general indexes, such as Web of Science and Scopus, are also popular but not universally used by ECRs, and regional differences again point to pros and cons of these services. Some specialized services are emphasized, including regional tools such as the China National Knowledge Infrastructure, as well as certain broad disciplinary resources, such as PubMed for its coverage of sciences and biomedical information. &#x0D; &#x0D; Researchers report that ECRs participating in this study were less concerned about how they gained access to full-text scholarly information, only that they could access full-text sources. In particular, ECRs do not take much notice of libraries and their platforms, seemingly unaware of the steps libraries take to acquire and ensure access to scholarly information, while viewing physical libraries themselves primarily as study spaces for undergraduate students and not places for the ECR to visit or work. While ECRs occasionally acknowledge library portals and login interfaces, researchers found that these participants mostly ignored these, and that they found discovery services to be confusing or difficult. &#x0D; &#x0D; Concerning social media use, participants identified 11 different platforms used but only ResearchGate was mentioned and used by participants from all seven countries represented. Social media tends to be used directly for keeping track of research trends and opinions and also the work specific researchers are publishing, and indirectly when referred to sites such as ResearchGate to find full-text of a specific article. Facebook, Twitter, and LinkedIn are used occasionally or moderately, but not universally. Researchers highlight regional differences of social media use in China, where ECRs are more likely to connect with other researchers and receive notifications when those researchers publish.&#x0D; &#x0D; The study reports limited information ECRs’ use of smartphones for information seeking. About half of ECR participants reported use of their smartphone for discovering scholarly sources. The advantage smartphones provide includes near-ubiquitous Internet access and therefore the ability to access scholarly materials on the go, though ECRs are less likely to download or read full-text articles via their smartphones. The rate of adoption of smartphone use for scholarly materials varies by country. &#x0D; &#x0D; Conclusion – Early career researchers access scholarly information in a wide variety of ways, with Google and Google Scholar as the preferred starting location, and with social media also proving useful. Ease-of-use and full-text availability are paramount concerns; the spread of open access materials helps fuel the availability of materials, and Google makes these easy to find. Though physical libraries are perceived to be of limited use, the digital access they provide to full-text scholarly sources is still vital even if ECRs do not make the connection between having that important access and the fact that libraries act as buyers and providers of access

Using preprints in evidence synthesis: Commentary on experience during the COVID-19 pandemic

Decision-making approaches to grant funding allocation: insights from a realist synthesis

Peer-reviewed journal publication is the main means for academic researchers in the life sciences to create a permanent public record of their work. These publications are also the de facto currency for career progress, with a strong link between journal brand recognition and perceived value. The current peer-review process can lead to long delays between submission and publication, with cycles of rejection, revision, and resubmission causing redundant peer review. This situation creates unique challenges for early career researchers (ECRs), who rely heavily on timely publication of their work to gain recognition for their efforts. Today, ECRs face a changing academic landscape, including the increased interdisciplinarity of life sciences research, expansion of the researcher population, and consequent shifts in employer and funding demands. The publication of preprints, publicly available scientific manuscripts posted on dedicated preprint servers prior to journal-managed peer review, can play a key role in addressing these ECR challenges. Preprinting benefits include rapid dissemination of academic work, open access, establishing priority or concurrence, receiving feedback, and facilitating collaborations. Although there is a growing appreciation for and adoption of preprints, a minority of all articles in life sciences and medicine are preprinted. The current low rate of preprint submissions in life sciences and ECR concerns regarding preprinting need to be addressed. We provide a perspective from an interdisciplinary group of ECRs on the value of preprints and advocate their wide adoption to advance knowledge and facilitate career development.

This article reports on the findings of an international online survey of early career researchers (ECRs) with regard to their authorship and peer review, attitudes, and practices, which sought to discover how the new wave of researchers were utilizing these key aspects of the scholarly communications system. A questionnaire was developed on the back of a 3‐year longitudinal, qualitative study and was distributed through publisher lists, social media networks, university networks, and specialist ECR membership organizations. Identical English, Polish, Russian, Chinese, Spanish, and French versions of the questionnaire were used. Results from 1,600 respondents demonstrated that 82.7% had co‐authored a paper, and most had performed a variety of authorship tasks. Almost half the respondents reported being subject to various authorship policies, although a quarter said they were not aware of any such policies. Almost all Chinese ECRs reported being subject to authorship policies, but only a third of UK ECRs reported the same. Three‐quarters of ECRs had experience in responding to peer review, and half had been peer reviewers. Half the respondents had a good experience of review and viewed it as a valuable way to improve their authorship skills. However, there was some criticism of some shortcoming such as lengthy peer review and superficial or uninformed comments by reviewers. Double‐blind review was the preferred methodology, and there were few suggestions for how to improve the review process.

Economic perspectives on the future of academic publishing: Introduction to the special issue

The successful publication of peer reviewed academic journal articles is an essential achievement for early career researchers (ECRs) seeking to establish themselves in their profession. However, this journey can pose several significant challenges for ECRs. We use an autoethnographic approach that draws deeply on our lived experience as ECRs to capture our recent and current experiences of negotiating the academic journal article publication journey to explore the tensions, contradictions, and benefits encountered in the journey. We critically examine challenges we experienced in choosing a target journal and negotiating the follow‐up process; undertaking revisions; and our experiences of limitations and possibilities in peer review and editorial support. While the peer review journal writing process has played a significant role in supporting us to become more effective ECRs, we also highlight challenges we faced negotiating ethical quandaries in this space, as well as illustrate how our preconceptions of a simple publication journey were confounded by subsequent experience of the complex realities of the space. We also suggest that educational interventions are indicated to provide ECRs support in foundational knowledge about what constitutes valuable revisions, an effective paper, and the scope of issues that can be addressed to make a paper more effective, with reference to the possibility of academic mentoring to support this need. Finally, we explore our findings in light of the tensions imposed by the relative inexperience and lack of power yielded by ECRs.

This report details a skills development process for early career researchers (ECRs) associated with ten Arts and Humanities Research Council (AHRC)-funded research networks that focus on disability-inclusive development. The process involved both the formation of an ECR peer support group and a face-to-face workshop. For the peer support group, the principal investigators (PIs) from the networks were asked to nominate associated ECRs to participate. The definition of an ECR was kept broad, with anyone who self-defines as an ECR being welcome to join. Members were not expected to have a postgraduate degree but either be working in a research position or have strong aspirations to do so in the near future. The group attracted 22 members from 12 countries. Online meetings were held every month over a period of eight months in 2023 and 2024. These meetings involved discussions, presentations, and networking opportunities for the ECRs. A face-to-face workshop was the next phase of the skills development work. The PIs were asked to select ECRs to attend a face-to-face skills development workshop. Various skills domains for development were identified, including collegiality, mentoring, publishing research, and collaboration. The ECRs also expressed a desire to learn more about effective approaches to enhance public engagement with research, techniques for understanding and integrating policy implications into research, and about innovative ways to disseminate research to diverse audiences. The workshop was delivered in partnership with Stellenbosch University, South Africa, in October 2024, with seven ECRs participating. This report provides a detailed account of the entire ECR skills development process, including reflections from participants on their experiences of being involved.