Lack of methodological rigor and limited coverage of generative artificial intelligence in existing artificial intelligence reporting guidelines: a scoping review.

How Can <i>IJDS</i> Authors, Reviewers, and Editors Use (and Misuse) Generative AI?

The integration of artificial intelligence (AI) has revolutionized medical research, offering innovative solutions for data collection, patient engagement, and information dissemination. Powerful generative AI (GenAI) tools and other similar chatbots have emerged, facilitating user interactions with virtual conversational agents. However, the increasing use of GenAI tools in medical research presents challenges, including ethical concerns, data privacy issues, and the potential for generating false content. These issues necessitate standardization of reporting to ensure transparency and scientific rigor. The development of the Generative Artificial Intelligence Tools in Medical Research (GAMER) reporting guidelines aims to establish comprehensive, standardized guidelines for reporting the use of GenAI tools in medical research. The GAMER guidelines are being developed following the methodology recommended by the Enhancing the Quality and Transparency of Health Research (EQUATOR) Network, involving a scoping review and expert Delphi consensus. The scoping review searched PubMed, Web of Science, Embase, CINAHL, PsycINFO, and Google Scholar (for the first 200 results) using keywords like "generative AI" and "medical research" to identify reporting elements in GenAI-related studies. The Delphi process involves 30-50 experts with ≥3 years of experience in AI applications or medical research, selected based on publication records and expertise across disciplines (eg, clinicians and data scientists) and regions (eg, Asia and Europe). A 7-point-scale survey will establish consensus on checklist items. The testing phase invites authors to apply the GAMER checklist to GenAI-related manuscripts and provide feedback via a questionnaire, while experts assess reliability (κ statistic) and usability (time taken, 7-point Likert scale). The study has been approved by the Ethics Committee of the Institute of Health Data Science at Lanzhou University (HDS-202406-01). The GAMER project was launched in July 2023 by the Evidence-Based Medicine Center of Lanzhou University and the WHO Collaborating Centre for Guideline Implementation and Knowledge Translation, and it concluded in July 2024. The scoping review was completed in November 2023. The Delphi process was conducted from October 2023 to April 2024. The testing phase began in March 2025 and is ongoing. The expected outcome of the GAMER project is a reporting checklist accompanied by relevant terminology, examples, and explanations to guide stakeholders in better reporting the use of GenAI tools. GAMER aims to guide researchers, reviewers, and editors in the transparent and scientific application of GenAI tools in medical research. By providing a standardized reporting checklist, GAMER seeks to enhance the clarity, completeness, and integrity of research involving GenAI tools, thereby promoting collaboration, comparability, and cumulative knowledge generation in AI-driven health care technologies. DERR1-10.2196/64640.

Enhancing the quality and transparency of health research

Response to M. Trengove & coll regarding "Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine".

during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see The Effect of Open Access).

Getting AI Right: Introductory Notes on AI &amp; Society

Reporting Guidelines for Artificial Intelligence Use in Orthopaedic Surgery Research.

Introducing a new series on effective writing and publishing of scientific papers

Applications of artificial intelligence (AI) are pervasive in modern biomedical science. In fact, research results suggesting algorithms and AI models for different target diseases and conditions are continuously increasing. While this situation undoubtedly improves the outcome of AI models, health care providers are increasingly unsure which AI model to use due to multiple alternatives for a specific target and the "black box" nature of AI. Moreover, the fact that studies rarely use guidelines in developing and reporting AI models poses additional challenges in trusting and adapting models for practical implementation. This review protocol describes the planned steps and methods for a review of the synthesized evidence regarding the quality of available guidelines and frameworks to facilitate AI applications in medicine. We will commence a systematic literature search using medical subject headings terms for medicine, guidelines, and machine learning (ML). All available guidelines, standard frameworks, best practices, checklists, and recommendations will be included, irrespective of the study design. The search will be conducted on web-based repositories such as PubMed, Web of Science, and the EQUATOR (Enhancing the Quality and Transparency of Health Research) network. After removing duplicate results, a preliminary scan for titles will be done by 2 reviewers. After the first scan, the reviewers will rescan the selected literature for abstract review, and any incongruities about whether to include the article for full-text review or not will be resolved by the third and fourth reviewer based on the predefined criteria. A Google Scholar (Google LLC) search will also be performed to identify gray literature. The quality of identified guidelines will be evaluated using the Appraisal of Guidelines, Research, and Evaluation (AGREE II) tool. A descriptive summary and narrative synthesis will be carried out, and the details of critical appraisal and subgroup synthesis findings will be presented. The results will be reported using the PRISMA (Preferred Reporting Items for Systematic Review and Meta-Analyses) reporting guidelines. Data analysis is currently underway, and we anticipate finalizing the review by November 2023. Guidelines and recommended frameworks for developing, reporting, and implementing AI studies have been developed by different experts to facilitate the reliable assessment of validity and consistent interpretation of ML models for medical applications. We postulate that a guideline supports the assessment of an ML model only if the quality and reliability of the guideline are high. Assessing the quality and aspects of available guidelines, recommendations, checklists, and frameworks-as will be done in the proposed review-will provide comprehensive insights into current gaps and help to formulate future research directions. DERR1-10.2196/47105.

Our study aimed to amplify and explain the items of statistical reporting requirements proposed by medical journals, and to improve the statistical reporting quality of medical articles. Statistical reporting requirements were obtained from the reporting standards published by the International Committee of Medical Journal Editors (ICMJE), the Enhancing the QUAlity and Transparency of Health Research (EQUATOR) network, and the editorial board of Chinese Medical Journal, etc. The items involved in statistical reporting requirements were summarized as issues of study design, statistical analysis, and interpretation of results. Each item was amplified based on cases of original articles. It is noticeable that the statistical reporting requirements of English medical journals generally referring to guidance documents, including "Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals" proposed by the ICMJE, or the statements for different study types published by the EQUATOR network, where the statistical reporting of medical articles had been detailed specified. The statistical reporting requirements of Chinese medical journals, however, were usually stated by the editorial boards. Although the formats and contents of statistical analysis had been regulated, the requirements of Chinese medical journals were to some extent insufficient and should be enhanced in accordance with the international standards. In conclusion, the amplification and explanation of statistical reporting requirements were expected to help investigators understand the requirements for statistical reporting in medical researches, so as to effectively improve the quality of medical articles.

Objectives/Goals: The Standards for Reporting Implementation Studies (StaRI) are the Enhancing the Quality and Transparency of Health Research (EQUATOR) Network 27-item checklist for Implementation Science. This study quantifies StaRI adherence among self-defined Implementation Science studies in published Learning Health Systems (LHS) research. Methods/Study Population: A medical librarian-designed a search strategy identified original Implementation Science research published in one of the top 20 Implementation Science journals between 2017 and 2021. Inclusion criteria included studies or protocols describing the implementation of any intervention in healthcare settings. Exclusion criteria included concept papers, non-implementation research, or editorials. Full-text documents were reviewed by two investigators to abstract and judge StaRI implementation and intervention adherence, partial adherence, or non-adherence. Results/Anticipated Results: A total of 330 documents were screened, 97 met inclusion criteria, and 47 were abstracted including 30 research studies and 17 protocols. Adherence to individual StaRI reporting items ranged from 13% to 100%. Most StaRI items were reported in >60% of manuscripts and protocols. The lowest adherence in research studies was noted around economic evaluation reporting for implementation (16%) or intervention (13%) strategies, harms (13%), contextual changes (30%), or fidelity of either the intervention (34%) or implementation (53%) approach. Subgroup analyses were infrequently contemplated or reported (43%). In protocols, the implications of the implementation strategy (41%) or intervention approach (47%) were not commonly reported. Discussion/Significance of Impact: When leveraging implementation science to report reproducible and sustainable practice change initiatives, LHS researchers will need to include assessments of economics, harms, context, and fidelity in order to attain higher levels of adherence to EQUATOR’s StaRI checklist.

182 Background: The notion of sex and gender is constantly evolving through many disciplines. In medical oncology, disease behavior and treatment modalities have been shown to impact men and women differently. Sex and gender-tailored clinical research in oncology could promote a better understanding of disease progression and interpretation of clinical trials. The Guidelines for Sex and Gender Equity in Research (SAGER) promote the specification of sex and gender in any given manuscript. The Enhancing the Quality and Transparency of Health Research (EQUATOR) Network promotes the wider use of 575 guidelines, which SAGER is a part of. We sought to determine the frequency in which SAGER Guidelines and the EQUATOR network are referenced in the instructions to authors of the top 100 medical oncology journals with the highest impact factor. Methods: We identified the top 100 medical oncology journals with the highest impact factor. For each, we revised their instructions to authors' material and recorded the frequency with which they referenced SAGER guidelines, the EQUATOR network, or sex and gender recommendations. Results: Median value of the (N=100) journal´s impact factor was 7.47 (range 4.8 – 286.1). Overall, 28 journals mentioned SAGER guidelines, 31 mentioned the EQUATOR network, and 37 mentioned sex and gender. Of the 28 journals that mention SAGER guidelines, 27 also mention sex and gender in their instructions to authors. A total of 12 journals mentioned both SAGER and the EQUATOR network in their instructions to authors. We dichotomized the IF value into “journals with a high impact factor” (≥10 IF) (n=31) and “journals with a low impact factor” (&lt;10) (n=69). SAGER was mentioned in 7 (22.5%) high-impact factor journals and 21 (30.4%) low-impact factor journals. The EQUATOR network was mentioned in 8 (25.8%) high-impact factor journals and 23 (33.3%) low-impact factor journals. Sex and gender was mentioned in 12 (38.7%) high-impact factor journals and 25 (36.2%) low-impact factor journals. Conclusions: Most oncology journals still warrant consideration for appropriate sex and gender recommendations. Guidelines like SAGER and EQUATOR are still commonly overlooked in instructions to authors. Consideration for this is paramount in oncology studies, as sex and gender-tailored research could impact disease biology and treatment understanding in oncology.

IntroductionReporting guidelines are important tools for improving the quality of medical research. The Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network’s Library contains a comprehensive and up-to-date database...

The Enhancing the Quality and Transparency of Health Research (EQUATOR) Network indexes over 600 reporting guidelines designed to improve the reproducibility of manuscripts across medical fields and study designs. Although...

IntroductionWhile there are guidelines for reporting on observational studies (eg, Strengthening the Reporting of Observational Studies in Epidemiology, Reporting of Studies Conducted Using Observational Routinely Collected Health Data Statement), estimation...