Reproducible Science Research Articles

A design-thinking approach to therapeutic translation: tympanic regeneration.

Clinician researchers face the pressures of meeting academic benchmarks combined with advancing new therapies to patients. The vast majority of drug discoveries fail in translation. A new method of meeting the challenges of preclinical therapeutic translation is presented using the example of tympanic regeneration. The key to a design-thinking approach to therapeutic translation is to 'begin with the end in mind' by widening the scope of the problem, with multiple points of view, to not only understand the disease but the context for the patient and the health system in which it occurs. Idea for therapeutics should be tested in relevant models early and once proof of efficacy is established, translational milestones that represent the greatest risk, such as safety and toxicity should be addressed first. It is important to seek the feedback of industry early to understand what milestones should be best addressed next with limited academic resources. Whenever proceeding, guidelines for maintaining scientific reproducibility should be followed to minimize risk of failure during transfer into industry. A Design-thinking approach addresses the potential failures in drug discovery and preclinical translation.

Toward a global and reproducible science for brain imaging in neurotrauma: the ENIGMA adult moderate/severe traumatic brain injury working group

The global burden of mortality and morbidity caused by traumatic brain injury (TBI) is significant, and the heterogeneity of TBI patients and the relatively small sample sizes of most current neuroimaging studies is a major challenge for scientific advances and clinical translation. The ENIGMA (Enhancing NeuroImaging Genetics through Meta-Analysis) Adult moderate/severe TBI (AMS-TBI) working group aims to be a driving force for new discoveries in AMS-TBI by providing researchers world-wide with an effective framework and platform for large-scale cross-border collaboration and data sharing. Based on the principles of transparency, rigor, reproducibility and collaboration, we will facilitate the development and dissemination of multiscale and big data analysis pipelines for harmonized analyses in AMS-TBI using structural and functional neuroimaging in combination with non-imaging biomarkers, genetics, as well as clinical and behavioral measures. Ultimately, we will offer investigators an unprecedented opportunity to test important hypotheses about recovery and morbidity in AMS-TBI by taking advantage of our robust methods for large-scale neuroimaging data analysis. In this consensus statement we outline the working group’s short-term, intermediate, and long-term goals.

Publishing reproducible dynamic kinetic models.

Publishing repeatable and reproducible computational models is a crucial aspect of the scientific method in computational biology and one that is often forgotten in the rush to publish. The pressures of academic life and the lack of any reward system at institutions, granting agencies and journals means that publishing reproducible science is often either non-existent or, at best, presented in the form of an incomplete description. In the article, we will focus on repeatability and reproducibility in the systems biology field where a great many published models cannot be reproduced and in many cases even repeated. This review describes the current landscape of software tooling, model repositories, model standards and best practices for publishing repeatable and reproducible kinetic models. The review also discusses possible future remedies including working more closely with journals to help reviewers and editors ensure that published kinetic models are at minimum, repeatable. Contact: hsauro@uw.edu.

Barriers to reproducibility: misalignment of career incentives and open science best practices

We are on the precipice of a sea change in how research is conducted. Funder mandates and changing expectations around scientific reproducibility have resulted in a new set of open science best practices (OSBP) for research. These practices include, but are not limited to, data and code sharing, hypothesis preregistration, registered reports, and open peer review. With these changes come challenges and potentially unanticipated side-effects. In particular, here I will highlight what I see as the biggest mismatches between the career incentives of the field of neuroscience and OSBP, from the perspective of an early career researcher.

Automated Water Supply Model (AWSM): Streamlining and standardizing application of a physically based snow model for water resources and reproducible science

Reproducible science requires a shift in thinking and application for how data, code and analysis are shared. Now, scientists must act more like software engineers to design models and perform analysis that use principles and techniques pioneered by software developers. Creating reproducible models that are easy to use and understand is in the best interest for the snow and hydrology community, enabling studies by other researchers and facilitating technology transfer to operational applications. Here, we present the Automated Water Supply Model (AWSM) that streamlines and standardizes the workflow of a physically based snow model to create fully reproducible model simulations that can be utilized by researchers and operational water resource managers. AWSM orchestrates four core components that historically required significant, ad-hoc modeler interaction to load the input data, spatially interpolate to the modeling domain, run the models and process the outputs. Because AWSM was developed using principles and techniques from software engineering, users can quickly perform reproducible simulations on any operating system, from a laptop to the cloud. The three fully reproducible example case studies showcase the simplicity and flexibility of using AWSM to perform simulations from small research catchments to simulations that aid in real time water management decisions.

APRICOT: Advanced Platform for Reproducible Infrastructures in the Cloud via Open Tools.

Background Scientific publications are meant to exchange knowledge among researchers but the inability to properly reproduce computational experiments limits the quality of scientific research. Furthermore, bibliography shows that irreproducible preclinical research exceeds 50%, which produces a huge waste of resources on nonprofitable research at Life Sciences field. As a consequence, scientific reproducibility is being fostered to promote Open Science through open databases and software tools that are typically deployed on existing computational resources. However, some computational experiments require complex virtual infrastructures, such as elastic clusters of PCs, that can be dynamically provided from multiple clouds. Obtaining these infrastructures requires not only an infrastructure provider, but also advanced knowledge in the cloud computing field. Objectives The main aim of this paper is to improve reproducibility in life sciences to produce better and more cost-effective research. For that purpose, our intention is to simplify the infrastructure usage and deployment for researchers. Methods This paper introduces Advanced Platform for Reproducible Infrastructures in the Cloud via Open Tools (APRICOT), an open source extension for Jupyter to deploy deterministic virtual infrastructures across multiclouds for reproducible scientific computational experiments. To exemplify its utilization and how APRICOT can improve the reproduction of experiments with complex computation requirements, two examples in the field of life sciences are provided. All requirements to reproduce both experiments are disclosed within APRICOT and, therefore, can be reproduced by the users. Results To show the capabilities of APRICOT, we have processed a real magnetic resonance image to accurately characterize a prostate cancer using a Message Passing Interface cluster deployed automatically with APRICOT. In addition, the second example shows how APRICOT scales the deployed infrastructure, according to the workload, using a batch cluster. This example consists of a multiparametric study of a positron emission tomography image reconstruction. Conclusion APRICOT's benefits are the integration of specific infrastructure deployment, the management and usage for Open Science, making experiments that involve specific computational infrastructures reproducible. All the experiment steps and details can be documented at the same Jupyter notebook which includes infrastructure specifications, data storage, experimentation execution, results gathering, and infrastructure termination. Thus, distributing the experimentation notebook and needed data should be enough to reproduce the experiment.

A Multilevel Bayesian Approach to Improve Effect Size Estimation in Regression Modeling of Metabolomics Data Utilizing Imputation with Uncertainty.

To ensure scientific reproducibility of metabolomics data, alternative statistical methods are needed. A paradigm shift away from the p-value toward an embracement of uncertainty and interval estimation of a metabolite’s true effect size may lead to improved study design and greater reproducibility. Multilevel Bayesian models are one approach that offer the added opportunity of incorporating imputed value uncertainty when missing data are present. We designed simulations of metabolomics data to compare multilevel Bayesian models to standard logistic regression with corrections for multiple hypothesis testing. Our simulations altered the sample size and the fraction of significant metabolites truly different between two outcome groups. We then introduced missingness to further assess model performance. Across simulations, the multilevel Bayesian approach more accurately estimated the effect size of metabolites that were significantly different between groups. Bayesian models also had greater power and mitigated the false discovery rate. In the presence of increased missing data, Bayesian models were able to accurately impute the true concentration and incorporating the uncertainty of these estimates improved overall prediction. In summary, our simulations demonstrate that a multilevel Bayesian approach accurately quantifies the estimated effect size of metabolite predictors in regression modeling, particularly in the presence of missing data.

Research Data Management for Master’s Students: From Awareness to Action

This article provides an analysis of how sixteen recently graduated master’s students from the Netherlands perceive research data management. It is important to study the master’s students’ attitudes towards this, as students in this phase prepare themselves for their career. Some of them might become future academics or policymakers, thus, potentially, the future advocates of good data management and reproducible science. In general, students were rather unsure what ‘data management’ meant and would often confuse it with data analysis, study design or methodology, or ethics and privacy. When students defined the concept, they focussed on privacy aspects. Concepts such as open data and the ‘FAIR’ principles were rarely mentioned, even though these are the cornerstones of contemporary data management efforts. In practice, the students managed their own data in an ad hoc way, and only a few of them worked with a clear data management plan. Illustrative of this is that half of the interviewees did not know where to find their data anymore. Furthermore, their study programmes had diverse approaches to data management education. Most of the classes offered were limited in scope. Nevertheless, the students seemed to be aware of the importance of data management and were willing to learn more about good data management practices. This report helps to catch an important first glimpse of how master’s students (from different scientific backgrounds) think about research data management. Only by knowing this, accurate measures can be taken to improve data management awareness and skills. The article also provides some useful recommendations on what such measures might be, and introduces some of the steps already taken by the Delft University of Technology (TU Delft).

WRF‐TEB: Implementation and Evaluation of the Coupled Weather Research and Forecasting (WRF) and Town Energy Balance (TEB) Model

AbstractUrban land surface processes need to be represented to inform future urban climate and building energy projections. Here, the single layer urban canopy model Town Energy Balance (TEB) is coupled to the Weather Research and Forecasting (WRF) model to create WRF‐TEB. The coupling method is described generically, implemented into software, and the code and data are released with a Singularity image to address issues of scientific reproducibility. The coupling is implemented modularly and verified by an integration test. Results show no detectable errors in the coupling. Separately, a meteorological evaluation is undertaken using observations from Toulouse, France. The latter evaluation, during an urban canopy layer heat island episode, shows reasonable ability to estimate turbulent heat flux densities and other meteorological quantities. We conclude that new model couplings should make use of integration tests as meteorological evaluations by themselves are insufficient, given that errors are difficult to attribute because of the interplay between observational errors and multiple parameterization schemes (e.g., radiation, microphysics, and boundary layer).

Design and Use of Semantic Resources: Findings from the Section on Knowledge Representation and Management of the 2020 International Medical Informatics Association Yearbook.

Summary Objective : To select, present, and summarize the best papers in the field of Knowledge Representation and Management (KRM) published in 2019. Methods : A comprehensive and standardized review of the biomedical informatics literature was performed to select the most interesting papers of KRM published in 2019, based on PubMed and ISI Web Of Knowledge queries. Results : Four best papers were selected among 1,189 publications retrieved, following the usual International Medical Informatics Association Yearbook reviewing process. In 2019, research areas covered by pre-selected papers were represented by the design of semantic resources (methods, visualization, curation) and the application of semantic representations for the integration/enrichment of biomedical data. Besides new ontologies and sound methodological guidance to rethink knowledge bases design, we observed large scale applications, promising results for phenotypes characterization, semantic-aware machine learning solutions for biomedical data analysis, and semantic provenance information representations for scientific reproducibility evaluation. Conclusion : In the KRM selection for 2019, research on knowledge representation demonstrated significant contributions both in the design and in the application of semantic resources. Semantic representations serve a great variety of applications across many medical domains, with actionable results.

How robust are sleep-mediated memory benefits?

Memories benefit from a retention interval filled with sleep. Current theories assume that this beneficial effect relies on consolidation processes occurring during slow-wave sleep (SWS). However, in the last years, several key findings supporting these theories could not be replicated or occurred only under certain conditions, suggesting that effects of sleep on memory are smaller, more task-dependent, less SWS-related, less robust and less long-lasting than previously assumed. In this review, we summarize recent replication failures, null-findings, meta-analyses and studies reporting important boundary conditions for the effect of sleep on declarative memory. We argue that more attempts to replicate and meta-analytic approaches together with higher standards for reproducible science are critical to advance the field of sleep and memory.

Responsible, practical genomic data sharing that accelerates research.

Data sharing anchors reproducible science, but expectations and best practices are often nebulous. Communities of funders, researchers and publishers continue to grapple with what should be required or encouraged. To illuminate the rationales for sharing data, the technical challenges and the social and cultural challenges, we consider the stakeholders in the scientific enterprise. In biomedical research, participants are key among those stakeholders. Ethical sharing requires considering both the value of research efforts and the privacy costs for participants. We discuss current best practices for various types of genomic data, as well as opportunities to promote ethical data sharing that accelerates science by aligning incentives.

The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research*

Reproducible science requires transparent reporting. The ARRIVE guidelines (Animal Research: Reporting of In Vivo Experiments) were originally developed in 2010 to improve the reporting of animal research. They consist of a checklist of information to include in publications describing in vivo experiments to enable others to scrutinise the work adequately, evaluate its methodological rigour, and reproduce the methods and results. Despite considerable levels of endorsement by funders and journals over the years, adherence to the guidelines has been inconsistent, and the anticipated improvements in the quality of reporting in animal research publications have not been achieved. Here, we introduce ARRIVE 2.0. The guidelines have been updated and information reorganised to facilitate their use in practice. We used a Delphi exercise to prioritise and divide the items of the guidelines into 2 sets, the “ARRIVE Essential 10,” which constitutes the minimum requirement, and the “Recommended Set,” which describes the research context. This division facilitates improved reporting of animal research by supporting a stepwise approach to implementation. This helps journal editors and reviewers verify that the most important items are being reported in manuscripts. We have also developed the accompanying Explanation and Elaboration document, which serves (1) to explain the rationale behind each item in the guidelines, (2) to clarify key concepts, and (3) to provide illustrative examples. We aim, through these changes, to help ensure that researchers, reviewers, and journal editors are better equipped to improve the rigour and transparency of the scientific process and thus reproducibility.

The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research

Reproducible science requires transparent reporting. The ARRIVE guidelines (Animal Research: Reporting of In Vivo Experiments) were originally developed in 2010 to improve the reporting of animal research. They consist of a checklist of information to include in publications describing in vivo experiments to enable others to scrutinise the work adequately, evaluate its methodological rigour, and reproduce the methods and results. Despite considerable levels of endorsement by funders and journals over the years, adherence to the guidelines has been inconsistent, and the anticipated improvements in the quality of reporting in animal research publications have not been achieved. Here, we introduce ARRIVE 2.0. The guidelines have been updated and information reorganised to facilitate their use in practice. We used a Delphi exercise to prioritise and divide the items of the guidelines into 2 sets, the “ARRIVE Essential 10,” which constitutes the minimum requirement, and the “Recommended Set,” which describes the research context. This division facilitates improved reporting of animal research by supporting a stepwise approach to implementation. This helps journal editors and reviewers verify that the most important items are being reported in manuscripts. We have also developed the accompanying Explanation and Elaboration document, which serves (1) to explain the rationale behind each item in the guidelines, (2) to clarify key concepts, and (3) to provide illustrative examples. We aim, through these changes, to help ensure that researchers, reviewers, and journal editors are better equipped to improve the rigour and transparency of the scientific process and thus reproducibility.

The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research.

Reproducible science requires transparent reporting. The ARRIVE guidelines (Animal Research: Reporting of In Vivo Experiments) were originally developed in 2010 to improve the reporting of animal research. They consist of a checklist of information to include in publications describing in vivo experiments to enable others to scrutinise the work adequately, evaluate its methodological rigour, and reproduce the methods and results. Despite considerable levels of endorsement by funders and journals over the years, adherence to the guidelines has been inconsistent, and the anticipated improvements in the quality of reporting in animal research publications have not been achieved. Here, we introduce ARRIVE 2.0. The guidelines have been updated and information reorganised to facilitate their use in practice. We used a Delphi exercise to prioritise and divide the items of the guidelines into 2 sets, the “ARRIVE Essential 10,” which constitutes the minimum requirement, and the “Recommended Set,” which describes the research context. This division facilitates improved reporting of animal research by supporting a stepwise approach to implementation. This helps journal editors and reviewers verify that the most important items are being reported in manuscripts. We have also developed the accompanying Explanation and Elaboration (E&E) document, which serves (1) to explain the rationale behind each item in the guidelines, (2) to clarify key concepts, and (3) to provide illustrative examples. We aim, through these changes, to help ensure that researchers, reviewers, and journal editors are better equipped to improve the rigour and transparency of the scientific process and thus reproducibility.

The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research.

Reproducible science requires transparent reporting. The ARRIVE guidelines (Animal Research: Reporting of In Vivo Experiments) were originally developed in 2010 to improve the reporting of animal research. They consist of a checklist of information to include in publications describing in vivo experiments to enable others to scrutinise the work adequately, evaluate its methodological rigour, and reproduce the methods and results. Despite considerable levels of endorsement by funders and journals over the years, adherence to the guidelines has been inconsistent, and the anticipated improvements in the quality of reporting in animal research publications have not been achieved. Here, we introduce ARRIVE 2.0. The guidelines have been updated and information reorganised to facilitate their use in practice. We used a Delphi exercise to prioritise and divide the items of the guidelines into 2 sets, the “ARRIVE Essential 10,” which constitutes the minimum requirement, and the “Recommended Set,” which describes the research context. This division facilitates improved reporting of animal research by supporting a stepwise approach to implementation. This helps journal editors and reviewers verify that the most important items are being reported in manuscripts. We have also developed the accompanying Explanation and Elaboration (E&E) document, which serves (1) to explain the rationale behind each item in the guidelines, (2) to clarify key concepts, and (3) to provide illustrative examples. We aim, through these changes, to help ensure that researchers, reviewers, and journal editors are better equipped to improve the rigour and transparency of the scientific process and thus reproducibility.

Big data requirements for artificial intelligence.

To summarize how big data and artificial intelligence technologies have evolved, their current state, and next steps to enable future generations of artificial intelligence for ophthalmology. Big data in health care is ever increasing in volume and variety, enabled by the widespread adoption of electronic health records (EHRs) and standards for health data information exchange, such as Digital Imaging and Communications in Medicine and Fast Healthcare Interoperability Resources. Simultaneously, the development of powerful cloud-based storage and computing architectures supports a fertile environment for big data and artificial intelligence in health care. The high volume and velocity of imaging and structured data in ophthalmology and is one of the reasons why ophthalmology is at the forefront of artificial intelligence research. Still needed are consensus labeling conventions for performing supervised learning on big data, promotion of data sharing and reuse, standards for sharing artificial intelligence model architectures, and access to artificial intelligence models through open application program interfaces (APIs). Future requirements for big data and artificial intelligence include fostering reproducible science, continuing open innovation, and supporting the clinical use of artificial intelligence by promoting standards for data labels, data sharing, artificial intelligence model architecture sharing, and accessible code and APIs.

ReprohackNL 2019: how libraries can promote research reproducibility through community engagement

University Libraries play a crucial role in moving towards Open Science, contributing to more transparent, reproducible and reusable research. The Center for Digital Scholarship (CDS) at Leiden University (LU) library is a scholarly lab that promotes open science literacy among Leiden’s scholars by two complementary strategies: existing top-down structures are used to provide training and services, while bottom-up initiatives from the research community are actively supported by offering the CDS’s expertise and facilities. An example of how bottom-up initiatives can blossom with the help of library structures such as the CDS is ReproHack. ReproHack – a reproducibility hackathon – is a grass-root initiative by young scholars with the goal of improving research reproducibility in three ways. First, hackathon attendees learn about reproducibility tools and challenges by reproducing published results and providing feedback to authors on their attempt. Second, authors can nominate their work and receive feedback on their reproducibility efforts. Third, the collaborative atmosphere helps building a community interested in making their own research reproducible.
 A first ReproHack in the Netherlands took place on November 30th, 2019, co-organised by the CDS at the LU Library with 44 participants from the fields of psychology, engineering, biomedicine, and computer science. For 19 papers, 24 feedback forms were returned and five papers were reported as successfully reproduced. Besides the researchers’ learning experience, the event led to recommendations on how to enhance research reproducibility. The ReproHack format therefore provides an opportunity for libraries to improve scientific reproducibility through community engagement.

The ARRIVE guidelines 2.0: updated guidelines for reporting animal research

Reproducible science requires transparent reporting. The ARRIVE guidelines (Animal Research: Reporting of In Vivo Experiments) were originally developed in 2010 to improve the reporting of animal research. They consist of...

Cloud Computing for Climate Modelling: Evaluation, Challenges and Benefits

Cloud computing is a mature technology that has already shown benefits for a wide range of academic research domains that, in turn, utilize a wide range of application design models. In this paper, we discuss the use of cloud computing as a tool to improve the range of resources available for climate science, presenting the evaluation of two different climate models. Each was customized in a different way to run in public cloud computing environments (hereafter cloud computing) provided by three different public vendors: Amazon, Google and Microsoft. The adaptations and procedures necessary to run the models in these environments are described. The computational performance and cost of each model within this new type of environment are discussed, and an assessment is given in qualitative terms. Finally, we discuss how cloud computing can be used for geoscientific modelling, including issues related to the allocation of resources by funding bodies. We also discuss problems related to computing security, reliability and scientific reproducibility.