Assessing the Transparency of Methods in Scientific Reporting

Abstract

Enhancing transparency in scientific reports is crucial to foster trust, facilitate reproducibility, and ensure the integrity of research, ultimately advancing the progress of knowledge and innovation. To devise strategies for enhancing transparency in scientific reports, the initial step is to assess the current state: to objectively measure the current level of transparency, identifying its shortcomings and associated factors, and to gauge improvements, for instance, following interventions. Here we present a new tool and a proof of concept to this endeavor. Using a checklist, we evaluated the methods transparency of a corpus of 180 papers published in 2011 and 2021 in five top-tier psychology journals. We specifically focused on the materials, procedures, and characteristics of population and sample. We summarized the level of transparency in the methods of each paper with the Transparency Of Methods (TOM) score. This score consists of the proportion of relevant information regarding the method that is available to the reader of a scientific report, either in the main text of the paper, or the appendixes, supplementary materials, and online open repository. It ranges from 0 (i.e., no transparency in the relevant aspects of the methods and materials) to 1 (i.e., the scientific report is fully transparent in all the relevant materials and methods). The results affirm TOM’s potential for assessing the transparency of scientific reporting and offer two snapshots of transparency in the methods of published papers, a decade apart. While they indicate progress has been made, there remains room for further enhancements and highlight specific areas that require attention. In conclusion, this work underscores the ongoing need for improvement in methods’ transparency and introduces a valuable tool, demonstrating its applicability as a means to evaluate the transparency of scientific reports.