Reproducible Science Research Articles

Geomasking to Safeguard Geoprivacy in Geospatial Health Data

Geomasking is a set of techniques that introduces noise or intentional errors into geospatial data to minimize the risk of identifying exact location information related to individuals while preserving the utility of the data to a controlled extent. It protects the geoprivacy of the data contributor and mitigates potential harm from data breaches while promoting safer data sharing. The development of digital health technologies and the extensive use of individual geospatial data in health studies have raised concerns about geoprivacy. The individual tracking data and health information, if accessed by unauthorized parties, may lead to privacy invasions, criminal activities, and discrimination. These risks underscore the importance of robust protective measures in the collection, management, and sharing of sensitive data. Geomasking techniques have been developed to safeguard geoprivacy in geospatial health data, addressing the risks and challenges associated with data sharing. This entry paper discusses the importance of geoprivacy in geospatial health data and introduces various kinds of geomasking methods and their applications in balancing the protection of individual privacy with the need for data sharing to ensure scientific reproducibility, highlighting the urgent need for more effective geomasking techniques and their applications.

Birthing a new maternal cognition literature: 10 recommendations for future research.

Whilst the field of maternal cognition is gaining interest, with a recent increase in publications, there are still only a handful of existing studies. This presents a unique opportunity for reflection and growth, advancing scientific rigor to ensure that future interpretations of maternal cognitive functioning are based on robust, generalizable data. With this in mind, we offer ten recommendations for future cognitive research in motherhood, with a focus on intentional study design. A study's design dictates the questions that can be asked, and the answers that can be gleaned from collected data, making study design a cornerstone of robust and reproducible science. These recommendations are intended as a resource for study conceptualization and design, participant recruitment, result interpretation, and peer review.

Comparison of datasets citation coverage in Google Scholar, Web of Science, Scopus, Crossref, and DataCite

The rapid increase of Earth science data from remote sensing, models, and ground-based observations highlights an urgent need for effective data management practices. Data repositories track provenance and usage metrics which are crucial for ensuring data integrity and scientific reproducibility. Although the introduction of Digital Object Identifiers (DOIs) for datasets in the late 1990s has significantly aided in crediting creators and enhancing dataset discoverability (akin to traditional research citations), considerable challenges persist in establishing linkage of datasets used with scholarly documents. This study evaluates the citation coverage of datasets from NASA’s Earth Observing System Data and Information System (EOSDIS) across several major bibliographic sources ‒ namely Google Scholar (GS), Web of Science (WoS), Scopus, Crossref, and DataCite—which helps data managers in making informed decisions when selecting bibliographic sources. We provide a robust and comprehensive understanding of the citation landscape, crucial for advancing data management practices and advancing open science. Our study searched and analyzed temporal trends across the bibliographic sources for publications that cite approximately 11,000 DOIs associated with EOSDIS datasets, yielding 17,000 unique journal and conference articles, reports, and book records linked to 3,000 dataset DOIs. GS emerged as the most comprehensive source while Crossref lagged significantly behind the other major sources. Crossref’s record references revealed that the absence of dataset DOIs and shortcomings in the Crossref Event data interface likely contributed to its underperformance. Scopus initially outperformed WoS until 2020, after which WoS began to show superior performance. Overall, our study underscores the necessity of utilizing multiple bibliographic sources for citation analysis, particularly for exploring dataset-to-document connections.

Enhancing Systematic Reviews in Health Science Using Integrated Search Interfaces, Discovery Layers and Federated Searching

ABSTRACT This article highlights the importance of integrated search interfaces (ISI) in systematic reviews and examines the current guidelines on their use. ISI can help researchers determine the scope of their review topic and identify relevant search terms and subject headings. They can also assist in developing and documenting search strategies, as well as rapidly expanding searches to additional databases, registers, and gray literature. ISI can also be used for ongoing literature surveillance and the identification of retractions and errata. Limitations and challenges associated with the use of ISI in systematic reviews include; difficulties in translating search syntax between databases, limitations in the extensibility of controlled subject classifications, the need to select relevant databases, the commercialization of ISI hindering scientific reproducibility, challenges in maintaining the reproducibility of search strategies over time, and skills and knowledge deficits in the workforce. This article also discusses opportunities for vendors to enhance ISI to better support systematic review workflows. Features to improve ISI may include duplicate citation identification, cross-platform integration with citation management tools and screening platforms, transparency of search interface configuration, integration of bibliometrics and semantic mapping, and the use of artificial intelligence (AI) to enhance search strategies. Advancements in Large Language Models in the realm of AI, along with APIs designed to incorporate these models into various software tools, have the capability to significantly enhance the effectiveness and comprehensiveness of systematic reviews.

Open instrumentation, like open data, is key to reproducible science. Yet, without incentives it won't thrive.

There is far more to open science than simply not shutting away your work. For example, a significant time investment must be made to collate, curate, explain and document precisely how an experiment can be reproduced or data can be reused. This time investment is currently poorly rewarded in our current model of open science where a bare minimum of openness is mandated, but further work is not recognized. As the open science movement looks beyond open access publications and open data towards ongoing detailed work such as open source software and open source hardware, it needs to consider how to properly encourage the extra work that is needed to properly document these projects. Without detailed documentation, the work cannot be replicated, reused and continually improved. If the work cannot be replicated or reused, is it really even open? This article is part of the Theo Murphy meeting issue 'Open, reproducible hardware for microscopy'.

Avoiding a reproducibility crisis in regulatory toxicology—on the fundamental role of ring trials

The ongoing transition from chemical hazard and risk assessment based on animal studies to assessment relying mostly on non-animal data, requires a multitude of novel experimental methods, and this means that guidance on the validation and standardisation of test methods intended for international applicability and acceptance, needs to be updated. These so-called new approach methodologies (NAMs) must be applicable to the chemical regulatory domain and provide reliable data which are relevant to hazard and risk assessment. Confidence in and use of NAMs will depend on their reliability and relevance, and both are thoroughly assessed by validation. Validation is, however, a time- and resource-demanding process. As updates on validation guidance are conducted, the valuable components must be kept: Reliable data are and will remain fundamental. In 2016, the scientific community was made aware of the general crisis in scientific reproducibility—validated methods must not fall into this. In this commentary, we emphasize the central importance of ring trials in the validation of experimental methods. Ring trials are sometimes considered to be a major hold-up with little value added to the validation. Here, we clarify that ring trials are indispensable to demonstrate the robustness and reproducibility of a new method. Further, that methods do fail in method transfer and ring trials due to different stumbling blocks, but these provide learnings to ensure the robustness of new methods. At the same time, we identify what it would take to perform ring trials more efficiently, and how ring trials fit into the much-needed update to the guidance on the validation of NAMs.

Evaluating Glyph Design for Showing Large-Magnitude-Range Quantum Spins.

We present experimental results to explore a form of bivariate glyphs for representing large-magnitude-range vectors. The glyphs meet two conditions: (1) two visual dimensions are separable; and (2) one of the two visual dimensions uses a categorical representation (e.g., a categorical colormap). We evaluate how much these two conditions determine the bivariate glyphs' effectiveness. The first experiment asks participants to perform three local tasks requiring reading no more than two glyphs. The second experiment scales up the search space in global tasks when participants must look at the entire scene of hundreds of vector glyphs to get an answer. Our results support that the first condition is necessary for local tasks when a few items are compared. But it is not enough for understanding a large amount of data. The second condition is necessary for perceiving global structures of examining very complex datasets. Participants' comments reveal that the categorical features in the bivariate glyphs trigger emergent optimal viewers' behaviors. This work contributes to perceptually accurate glyph representations for revealing patterns from large scientific results. We release source code, quantum physics data, training documents, participants' answers, and statistical analyses for reproducible science at https://osf.io/4xcf5/?view_only=94123139df9c4ac984a1e0df811cd580.

Five creative ways to promote reproducible science.

Five creative ways to promote reproducible science.

Computational reproducibility of Jupyter notebooks from biomedical publications.

Jupyter notebooks facilitate the bundling of executable code with its documentation and output in one interactive environment, and they represent a popular mechanism to document and share computational workflows, including for research publications. The reproducibility of computational aspects of research is a key component of scientific reproducibility but has not yet been assessed at scale for Jupyter notebooks associated with biomedical publications. We address computational reproducibility at 2 levels: (i) using fully automated workflows, we analyzed the computational reproducibility of Jupyter notebooks associated with publications indexed in the biomedical literature repository PubMed Central. We identified such notebooks by mining the article's full text, trying to locate them on GitHub, and attempting to rerun them in an environment as close to the original as possible. We documented reproduction success and exceptions and explored relationships between notebook reproducibility and variables related to the notebooks or publications. (ii) This study represents a reproducibility attempt in and of itself, using essentially the same methodology twice on PubMed Central over the course of 2 years, during which the corpus of Jupyter notebooks from articles indexed in PubMed Central has grown in a highly dynamic fashion. Out of 27,271 Jupyter notebooks from 2,660 GitHub repositories associated with 3,467 publications, 22,578 notebooks were written in Python, including 15,817 that had their dependencies declared in standard requirement files and that we attempted to rerun automatically. For 10,388 of these, all declared dependencies could be installed successfully, and we reran them to assess reproducibility. Of these, 1,203 notebooks ran through without any errors, including 879 that produced results identical to those reported in the original notebook and 324 for which our results differed from the originally reported ones. Running the other notebooks resulted in exceptions. We zoom in on common problems and practices, highlight trends, and discuss potential improvements to Jupyter-related workflows associated with biomedical publications.

Quantifying network behavior in the rat prefrontal cortex.

The question of how consciousness and behavior arise from neural activity is fundamental to understanding the brain, and to improving the diagnosis and treatment of neurological and psychiatric disorders. There is significant murine and primate literature on how behavior is related to the electrophysiological activity of the medial prefrontal cortex and its role in working memory processes such as planning and decision-making. Existing experimental designs, specifically the rodent spike train and local field potential recordings during the T-maze alternation task, have insufficient statistical power to unravel the complex processes of the prefrontal cortex. We therefore examined the theoretical limitations of such experiments, providing concrete guidelines for robust and reproducible science. To approach these theoretical limits, we applied dynamic time warping and associated statistical tests to data from neuron spike trains and local field potentials. The goal was to quantify neural network synchronicity and the correlation of neuroelectrophysiology with rat behavior. The results show the statistical limitations of existing data, and the fact that making meaningful comparison between dynamic time warping with traditional Fourier and wavelet analysis is impossible until larger and cleaner datasets are available.

ExpFlow: a graphical user interface for automated reproducible electrochemistry

ExpFlow, a software that allows the systematic encoding of laboratory workflows through a graphical user interface, facilitates the translation of human-developed laboratory procedures to robotic experimentation.

Reproducible Spinal Cord Quantitative MRI Analysis with the Spinal Cord Toolbox.

The spinal cord plays a pivotal role in the central nervous system, providing communication between the brain and the body and containing critical motor and sensory networks. Recent advancements in spinal cord MRI data acquisition and image analysis have shown a potential to improve the diagnostics, prognosis, and management of a variety of pathological conditions. In this review, we first discuss the significance of standardized spinal cord MRI acquisition protocol in multi-center and multi-manufacturer studies. Then, we cover open-access spinal cord MRI datasets, which are important for reproducible science and validation of new methods. Finally, we elaborate on the recent advances in spinal cord MRI data analysis techniques implemented in the open-source software package Spinal Cord Toolbox (SCT).

A metascience investigation of inclusive, open, and reproducible science practices in research posters at the 2021 SRCD biennial meeting.

Over the past decade, there has been a growing appreciation of metascience issues in psychological science. Using data collected from 2615 posters presented at the 2021 biennial meeting of the Society for Research in Child Development, this article examines the use of transparent research practices to increase rigor and reproducibility as well as generalizability through greater inclusivity of diverse samples. Research presented through poster presentations was heavily skewed toward quantitative studies featuring American researchers using Western hemisphere samples. Sharing of data/materials, preregistrations, and replications were uncommon. During a time when governments are increasingly requiring more open practices and access, this research provides an important baseline by which developmental science can benchmark progress toward the goals of greater inclusivity and openness.

Replacement of Dietary Fish Protein with Bacterial Protein Results in Decreased Adiposity Coupled with Liver Gene Expression Changes in Female Danio rerio

BackgroundEffective use of Danio rerio as a preclinical model requires standardization of macronutrient sources to achieve scientific reproducibility across studies and labs. ObjectiveOur objective was to evaluate a bacterial-based single-cell protein (SCP) for the production of open-source standardized diets with defined health characteristics for the zebrafish research community. MethodsWe completed a 16-wk feeding trial using juvenile D. rerio 31 d postfertilization (10 tanks per diet and 14 D. rerio per tank) with formulated diets containing either a typical fish protein ingredient [standard reference (SR) diet] or a novel bacterial SCP source [bacterial protein (BP) diet]. At the end of the feeding trial, growth metrics, body composition, reproductive success, and bulk transcriptomics of the liver (RNAseq on female D. rerio with confirmatory rtPCR) were performed for each diet treatment. ResultsD. rerio fed the BP diet had body weight gains equivalent to the D. rerio fed fish protein, and females had significantly lower total carcass lipid, indicating reduced adiposity. Reproductive success was similar between treatments, suggesting normal physiological function. Genes differentially expressed in female D. rerio fed the BP diet compared with females fed the SR diet were overrepresented in the gene ontologies of metabolism, biosynthesis of cholesterol precursors and products, and protein unfolding responses. ConclusionProtein source substantially affected body growth metrics and composition as well as gene expression. These data support the development of an open-source diet utilizing an ingredient that correlates with improved health profiles and reduced variability in notable outcomes.

How Do Isothermal Calorimeters Age? Repeatability and Reproducibility Across Calorimeter Generations

AbstractReliable experimental data is essential for reproducible science. As our lab has access to five instruments of the isothermal (heat conduction) type of different age and product generations, we became interested in the quality of experimental data obtained from the different instruments. Each instrument consists of 8 individual calorimeters. An intralaboratory study of the first 72 hours of cement hydration was conducted with all five instruments. The objective was to determine the differences between the available instruments with respect to repeatability (precision of the 8 calorimeters within one instrument) and reproducibility (precision between the different instruments). The experimental program comprises 120 calorimetry runs in total (5 instruments each with 8 calorimeters, triple measurement performed). Results show that the oldest instrument version with its 8 calorimeters has a stronger higher standard deviation of the heat released after 72 hours compared to the latest state‐of‐the‐art instrument model (246.7 ± 2.7 vs. 232.3 ± 0.5 Jg–1). This is attributed to the technical design differences of the instruments and continuous use over time. In terms of reproducibility, an increasing standard deviation for the heat released from 1 day to 3 days (136.2 ± 2.6 and 236.4 ± 5.9 J g–1) is observed. From our point of view, this test study indicates that isothermal heat conduction calorimetry is a robust and precise measurement method to determine the heat of cement hydration for 72 hours.

MotifHub: Detection of trans-acting DNA motif group with probabilistic modeling algorithm

BackgroundTrans-acting factors are of special importance in transcription regulation, which is a group of proteins that can directly or indirectly recognize or bind to the 8–12 bp core sequence of cis-acting elements and regulate the transcription efficiency of target genes. The progressive development in high-throughput chromatin capture technology (e.g., Hi-C) enables the identification of chromatin-interacting sequence groups where trans-acting DNA motif groups can be discovered. The problem difficulty lies in the combinatorial nature of DNA sequence pattern matching and its underlying sequence pattern search space. MethodHere, we propose to develop MotifHub for trans-acting DNA motif group discovery on grouped sequences. Specifically, the main approach is to develop probabilistic modeling for accommodating the stochastic nature of DNA motif patterns. ResultsBased on the modeling, we develop global sampling techniques based on EM and Gibbs sampling to address the global optimization challenge for model fitting with latent variables. The results reflect that our proposed approaches demonstrate promising performance with linear time complexities. ConclusionMotifHub is a novel algorithm considering the identification of both DNA co-binding motif groups and trans-acting TFs. Our study paves the way for identifying hub TFs of stem cell development (OCT4 and SOX2) and determining potential therapeutic targets of prostate cancer (FOXA1 and MYC). To ensure scientific reproducibility and long-term impact, its matrix-algebra-optimized source code is released at http://bioinfo.cs.cityu.edu.hk/MotifHub.

Towards a framework for interoperability and reproducibility of predictive models

The development and deployment of machine learning (ML) models for biomedical research and healthcare currently lacks standard methodologies. Although tools for model replication are numerous, without a unifying blueprint it remains difficult to scientifically reproduce predictive ML models for any number of reasons (e.g., assumptions regarding data distributions and preprocessing, unclear test metrics, etc.) and ultimately, questions around generalizability and transportability are not readily answered. To facilitate scientific reproducibility, we built upon the Predictive Model Markup Language (PMML) to capture essential information. As a key component of the PREdictive Model Index and Exchange REpository (PREMIERE) platform, we present the Automated Metadata Pipeline (AMP) for conversion of a given predictive ML model into an extended PMML file that autocompletes an ML-based checklist, assessing model elements for interoperability and reproducibility. We demonstrate this pipeline on multiple test cases with three different ML algorithms and health-related datasets, providing a foundation for future predictive model reproducibility, sharing, and comparison.

Characterizing Families of Spectral Similarity Scores and Their Use Cases for Gas Chromatography-Mass Spectrometry Small Molecule Identification.

Metabolomics provides a unique snapshot into the world of small molecules and the complex biological processes that govern the human, animal, plant, and environmental ecosystems encapsulated by the One Health modeling framework. However, this "molecular snapshot" is only as informative as the number of metabolites confidently identified within it. The spectral similarity (SS) score is traditionally used to identify compound(s) in mass spectrometry approaches to metabolomics, where spectra are matched to reference libraries of candidate spectra. Unfortunately, there is little consensus on which of the dozens of available SS metrics should be used. This lack of standard SS score creates analytic uncertainty and potentially leads to issues in reproducibility, especially as these data are integrated across other domains. In this work, we use metabolomic spectral similarity as a case study to showcase the challenges in consistency within just one piece of the One Health framework that must be addressed to enable data science approaches for One Health problems. Here, using a large cohort of datasets comprising both standard and complex datasets with expert-verified truth annotations, we evaluated the effectiveness of 66 similarity metrics to delineate between correct matches (true positives) and incorrect matches (true negatives). We additionally characterize the families of these metrics to make informed recommendations for their use. Our results indicate that specific families of metrics (the Inner Product, Correlative, and Intersection families of scores) tend to perform better than others, with no single similarity metric performing optimally for all queried spectra. This work and its findings provide an empirically-based resource for researchers to use in their selection of similarity metrics for GC-MS identification, increasing scientific reproducibility through taking steps towards standardizing identification workflows.

A dynamic web resource for robust and reproducible genomics in nonmodel species: marineomics.io

Abstract Genomic methods are becoming increasingly valuable and established in ecological research, particularly in nonmodel species. Supporting their progress and adoption requires investment in resources that promote (i) reproducibility of genomic analyses, (ii) accessibility of learning tools and (iii) keeping pace with rapidly developing methods and principles. We introduce marineomics.io, an open‐source, living document to disseminate tutorials, reproducibility tools and best principles for ecological genomic research in marine and nonmodel systems. The website's existing content spans population and functional genomics, including current recommendations for whole‐genome sequencing, RAD‐seq, Pool‐seq and RNA‐seq. With the goal to facilitate the development of new, similar resources, we describe our process for aggregating and synthesizing methodological principles from the ecological genomics community to inform website content. We also detail steps for authorship and submission of new website content, as well as protocols for providing feedback and topic requests from the community. These web resources were constructed with guidance for doing rigorous, reproducible science. Collaboration and contributions to the website are encouraged from scientists of all skill sets and levels of expertise.

Ten quick tips for building FAIR workflows.

Research data is accumulating rapidly and with it the challenge of fully reproducible science. As a consequence, implementation of high-quality management of scientific data has become a global priority. The FAIR (Findable, Accesible, Interoperable and Reusable) principles provide practical guidelines for maximizing the value of research data; however, processing data using workflows-systematic executions of a series of computational tools-is equally important for good data management. The FAIR principles have recently been adapted to Research Software (FAIR4RS Principles) to promote the reproducibility and reusability of any type of research software. Here, we propose a set of 10 quick tips, drafted by experienced workflow developers that will help researchers to apply FAIR4RS principles to workflows. The tips have been arranged according to the FAIR acronym, clarifying the purpose of each tip with respect to the FAIR4RS principles. Altogether, these tips can be seen as practical guidelines for workflow developers who aim to contribute to more reproducible and sustainable computational science, aiming to positively impact the open science and FAIR community.