Reproducible Data Analysis Research Articles

Reproducible brain PET data analysis: easier said than done.

While a great deal of recent effort has focused on addressing a perceived reproducibility crisis within brain structural magnetic resonance imaging (MRI) and functional MRI research communities, this article argues that brain positron emission tomography (PET) research stands on even more fragile ground, lagging behind efforts to address MRI reproducibility. We begin by examining the current landscape of factors that contribute to reproducible neuroimaging data analysis, including scientific standards, analytic plan pre-registration, data and code sharing, containerized workflows, and standardized processing pipelines. We then focus on disparities in the current status of these factors between brain MRI and brain PET. To demonstrate the positive impact that further developing such reproducibility factors would have on brain PET research, we present a case study that illustrates the many challenges faced by one laboratory that attempted to reproduce a community-standard brain PET processing pipeline. We identified key areas in which the brain PET community could enhance reproducibility, including stricter reporting policies among PET dedicated journals, data repositories, containerized analysis tools, and standardized processing pipelines. Other solutions such as mandatory pre-registration, data sharing, code availability as a condition of grant funding, and online forums and standardized reporting templates, are also discussed. Bolstering these reproducibility factors within the brain PET research community has the potential to unlock the full potential of brain PET research, propelling it toward a higher-impact future.

Identification of a New and Effective Marker Combination for a Standardized and Automated Bin-Based Basophil Activation Test (BAT) Analysis.

(1) Background: The basophil activation test (BAT) is a functional whole blood-based ex vivo assay to quantify basophil activation after allergen exposure by flow cytometry. One of the most important prerequisites for the use of the BAT in the routine clinical diagnosis of allergies is a reliable, standardized and reproducible data analysis workflow. (2) Methods: We re-analyzed a public mass cytometry dataset from peanut (PN) allergic patients (n = 6) and healthy controls (n = 3) with our binning approach "pattern recognition of immune cells" (PRI). Our approach enabled a comprehensive analysis of the dataset, evaluating 30 markers to achieve optimal basophil identification and activation through multi-parametric analysis and visualization. (3) Results: We found FcεRIα/CD32 (FcγRII) as a new marker couple to identify basophils and kept CD63 as an activation marker to establish a modified BAT in combination with our PRI analysis approach. Based on this, we developed an algorithm for automated raw data processing, which enables direct data analysis and the intuitive visualization of the test results including controls and allergen stimulations. Furthermore, we discovered that the expression pattern of CD32 correlated with FcεRIα, anticorrelated with CD63 and was detectable in both the re-analyzed public dataset and our own flow cytometric results. (4) Conclusions: Our improved BAT, combined with our PRI procedure (bin-BAT), provides a reliable test with a fully reproducible analysis. The advanced bin-BAT enabled the development of an automated workflow with an intuitive visualization to discriminate allergic patients from non-allergic individuals.

Detailed delineation of the fetal brain in diffusion MRI via multi-task learning.

Diffusion-weighted MRI is increasingly used to study the normal and abnormal development of fetal brain inutero. Recent studies have shown that dMRI can offer invaluable insights into the neurodevelopmental processes in the fetal stage. However, because of the low data quality and rapid brain development, reliable analysis of fetal dMRI data requires dedicated computational methods that are currently unavailable. The lack of automated methods for fast, accurate, and reproducible data analysis has seriously limited our ability to tap the potential of fetal brain dMRI for medical and scientific applications. In this work, we developed and validated a unified computational framework to (1) segment the brain tissue into white matter, cortical/subcortical gray matter, and cerebrospinal fluid, (2) segment 31 distinct white matter tracts, and (3) parcellate the brain's cortex and delineate the deep gray nuclei and white matter structures into 96 anatomically meaningful regions. We utilized a set of manual, semi-automatic, and automatic approaches to annotate 97 fetal brains. Using these labels, we developed and validated a multi-task deep learning method to perform the three computations. Our evaluations show that the new method can accurately carry out all three tasks, achieving a mean Dice similarity coefficient of 0.865 on tissue segmentation, 0.825 on white matter tract segmentation, and 0.819 on parcellation. The proposed method can greatly advance the field of fetal neuroimaging as it can lead to substantial improvements in fetal brain tractography, tract-specific analysis, and structural connectivity assessment.

MS-PyCloud: A Cloud Computing-Based Pipeline for Proteomic and Glycoproteomic Data Analyses.

Rapid development and wide adoption of mass spectrometry-based glycoproteomic technologies have empowered scientists to study proteins and protein glycosylation in complex samples on a large scale. This progress has also created unprecedented challenges for individual laboratories to store, manage, and analyze proteomic and glycoproteomic data, both in the cost for proprietary software and high-performance computing and in the long processing time that discourages on-the-fly changes of data processing settings required in explorative and discovery analysis. We developed an open-source, cloud computing-based pipeline, MS-PyCloud, with graphical user interface (GUI), for proteomic and glycoproteomic data analysis. The major components of this pipeline include data file integrity validation, MS/MS database search for spectral assignments to peptide sequences, false discovery rate estimation, protein inference, quantitation of global protein levels, and specific glycan-modified glycopeptides as well as other modification-specific peptides such as phosphorylation, acetylation, and ubiquitination. To ensure the transparency and reproducibility of data analysis, MS-PyCloud includes open-source software tools with comprehensive testing and versioning for spectrum assignments. Leveraging public cloud computing infrastructure via Amazon Web Services (AWS), MS-PyCloud scales seamlessly based on analysis demand to achieve fast and efficient performance. Application of the pipeline to the analysis of large-scale LC-MS/MS data sets demonstrated the effectiveness and high performance of MS-PyCloud. The software can be downloaded at https://github.com/huizhanglab-jhu/ms-pycloud.

IMG-29. IMPLICATIONS OF IMAGE HARMONIZATION FOR BRAIN TUMOR CLINICAL TRIALS

Abstract BACKGROUND Clinical trials for rare pediatric conditions like optic pathway gliomas associated with NF1 (NF1-OPGs) require numerous sites that utilize different MRI scanners and protocols. These MRIs must be harmonized for reliable comparison using machine learning tools. We propose a deep-learning based MRI harmonization that enables reproducible data analysis across multiple sites and MRI platforms. METHODS One-hundred eighty clinical T1-weighted-MRIs of pediatric NF1-OPG subjects were acquired from three sites [Children’s National Hospital (site A, N=60), Children’s Hospital Colorado (site B, N=60), and Children’s Hospital of Philadelphia (site C, N=60)] using different MR scanners (GE, Phillips, and Siemens, respectively). MRIs from A and B were used to train a branched neural network and C were used for testing. Using unsupervised learning, the neural network identified differences in imaging protocols and harmonized data to a chosen protocol, e.g., that of site A, while preserving the patient anatomy. The harmonization was evaluated using Wasserstein distance (nWD), which measured similarity between two gray scale appearances of MRI intensity histograms. Anatomy preservation metric (AP) measured relative volume changes in gray matter (GM) segmented using Freesurfer, before and after harmonization. For both metrics, 100% means the best possible harmonization was achieved. RESULTS Before harmonization, nWD between grayscale appearance of MRIs from sites A and B was 27.3±3.1%. After harmonization, the similarity became 94.2±2.0% with a GM volume error of 3.11±0.56ml which corresponded to AP of 93.1±4.7%. For the test data, nWD similarity post harmonization improved from 38.2±3.9% to 93.4±2.2% with GM volume error of 4.12±0.47ml which corresponded to AP of 90.3±3.5%. CONCLUSIONS Our approach enables MRI harmonization while preserving anatomy for precise analysis using machine learning tools. It scales to new sites to support multisite clinical trials that utilize imaging outcomes for brain tumors and other rare diseases.

MetaboAnalystR 4.0: a unified LC-MS workflow for global metabolomics.

The wide applications of liquid chromatography - mass spectrometry (LC-MS) in untargeted metabolomics demand an easy-to-use, comprehensive computational workflow to support efficient and reproducible data analysis. However, current tools were primarily developed to perform specific tasks in LC-MS based metabolomics data analysis. Here we introduce MetaboAnalystR 4.0 as a streamlined pipeline covering raw spectra processing, compound identification, statistical analysis, and functional interpretation. The key features of MetaboAnalystR 4.0 includes an auto-optimized feature detection and quantification algorithm for LC-MS1 spectra processing, efficient MS2 spectra deconvolution and compound identification for data-dependent or data-independent acquisition, and more accurate functional interpretation through integrated spectral annotation. Comprehensive validation studies using LC-MS1 and MS2 spectra obtained from standards mixtures, dilution series and clinical metabolomics samples have shown its excellent performance across a wide range of common tasks such as peak picking, spectral deconvolution, and compound identification with good computing efficiency. Together with its existing statistical analysis utilities, MetaboAnalystR 4.0 represents a significant step toward a unified, end-to-end workflow for LC-MS based global metabolomics in the open-source R environment.

Guidelines for reproducible analysis of adaptive immune receptor repertoire sequencing data.

Enhancing the reproducibility and comprehension of adaptive immune receptor repertoire sequencing (AIRR-seq) data analysis is critical for scientific progress. This study presents guidelines for reproducible AIRR-seq data analysis, and a collection of ready-to-use pipelines with comprehensive documentation. To this end, ten common pipelines were implemented using ViaFoundry, a user-friendly interface for pipeline management and automation. This is accompanied by versioned containers, documentation and archiving capabilities. The automation of pre-processing analysis steps and the ability to modify pipeline parameters according to specific research needs are emphasized. AIRR-seq data analysis is highly sensitive to varying parameters and setups; using the guidelines presented here, the ability to reproduce previously published results is demonstrated. This work promotes transparency, reproducibility, and collaboration in AIRR-seq data analysis, serving as a model for handling and documenting bioinformatics pipelines in other research domains.

WALIS dashboard: An online tool to explore a global paleo sea-level database.

In this paper, we present WALIS Dashboard, an open-access interface to the World Atlas of Last Interglacial Shorelines (WALIS), which was developed and compiled thanks to funding from the European Research Council. WALIS is a database that includes thousands of samples (dated with different radiometric methods) and sea-level indicators formed during the Last Interglacial (~80 to 130 ka). The WALIS Dashboard was coded in R (shiny app), and allows querying a simplified version of WALIS by either geographic extent or by attributes. The user can then download the queried data and perform simple and reproducible data analysis. The WALIS Dashboard can be used both online and offline.

Single-cell multi-parametric characterization of microbiota by flow cytometry.

It has become evident, that the microbes colonizing the human body have a great impact on health and disease. Investigations of microbiota currently primarily rely on culturomics, high-throughput sequencing and metaproteomics which have considerably advanced our knowledge regarding the role of the microbiota in our environment and for our health. While single-cell phenotyping of immune cells and other somatic cells by flow cytometry has become widely used, the detailed analysis of bacterial cells such as the human microbiota on the single-cell level, is lagging behind. Here, we outline a protocol for the single-cell characterization of bacterial cells from complex microbiota samples, such as stool, by multi-parametric flow cytometry. Our protocol describes the isotype-specific detection of host-antibody coating of intestinal bacteria ex vivo, which together with quantitative DNA staining and light scatter detection comprise an individual's microbiota fingerprint. Cryoconservation and appropriate staining controls ensure reliable, reproducible data generation and analysis. We have automated the analysis of the multi-dimensional data using a segmentation approach by self-organizing map (SOM) algorithm for downstream comparative analyses. Our protocol can be adapted to integrate further phenotypic markers and uses the power of analytical cytometry for the characterization of bacteria on the single-cell level.

What is integrative medicine? Establishing the decision criteria for an operational definition of integrative medicine for general practice health services research in Australia

BackgroundOperational definitions outline how a conceptual definition will be measured for consistent, reproducible data collection and analysis. This article reports the decision criteria that will be used for an operational definition of integrative medicine (IM) in a secondary analysis of an Australian national survey of general practitioner activity. MethodsA multidisciplinary team applied an iterative approach, informed by expert knowledge and literature reviews to establish decision criteria for categorizing the terms in the Australian clinical interface terminology of the International Classification of Primary Care, second edition (ICPC-2 PLUS) and the Coding Atlas for Pharmaceutical Substances, according to whether they reflected IM, conventional/mainstream medicine (MM), or both IM and MM (IM/MM). ResultsThe final decision criteria categorized all terms for examinations, investigations, advice/counselling, and drugs with synthetic ingredients, and terms for referrals to secondary care services and healthcare practitioners that are not a traditional or complementary medicine practitioner as MM. Terms that could apply to both styles of clinical practice (e.g., preventive health, lifestyle medicine, psychosocial and some drugs with natural ingredients) were categorised as IM/MM. The remaining terms, that mostly reflected the World Health Organization's theoretical definitions of traditional and complementary medicine, were categorized as IM. ConclusionDifferentiating between integrative and conventional/mainstream medicine in general practice is context specific and not always possible. The category IM/MM proposes integrative medicine as an extension, rather than an alternative. The rationale for the integrative medicine operational definition has relevance for researchers and health services in Australia, and internationally.

Data provenance - from experimental data to trustworthy simulation models and standards

FAIR (findable, accessible, interoperable and reusable) data usage is one of the main principals that many of the research and funding organizations include in their strategic plans, which means that following the main principals of FAIR data is required in many research projects. The definition of data being FAIR is very general. When implementing that for a specific application or project or even setting a standardized procedure within a working group, a company or a research community, many challenges arise. In this contribution, an overview about our experience with different methods and tools is outlined.We begin with a motivation on potential use cases for the application of FAIR data with increasing complexity starting from a reproducible research paper over collaborative projects with multiple participants such as Round-Robin tests up to data-based models within standardization codes, applications in machine learning or parameter estimation of physics-based simulation models.In a second part, different options for structuring the data (including metadata schema) are discussed. The first one is the openBIS system, which is an open-source lab notebook and PostgreSQL based data management system. A second option is a semantic representation using RDF based on ontologies for the domain of interest.In a third section, requirements for workflow tools to automate data processing are discussed and their integration into reproducible data analysis is presented with an outlook on required information to be stored as metadata in the database.Finally, the presented procedures are exemplarily demonstrated for the calibration of a temperature dependent constitutive model for additively manufactured mortar. A metadata schema for a rheological measurement setup is derived and implemented in an openBIS database. After a short review of a potential numerical model predicting the structural build-up behavior, the automatic workflow to use the stored data for model parameter estimation is demonstrated.

WALIS dashboard: An online tool to explore a global paleo sea-level database

This paper describes the WALIS dashboard, an open-access interface to the World Atlas of Last Interglacial Shorelines (WALIS), which was developed and compiled thanks to funding from the European Research Council. WALIS is a database that includes thousands of samples (dated with different radiometric methods) and sea-level indicators formed during the Last Interglacial (125 ka). The WALIS dashboard was coded in R (shiny app) and is connected with a simplified version of the WALIS database. The app allows querying data by geographic extent or by filtering metadata. It then allows the user to download the queried data and perform simple reproducible data analysis. The WALIS dashboard can be used both online and offline.

MetaPro: a scalable and reproducible data processing and analysis pipeline for metatranscriptomic investigation of microbial communities

BackgroundWhole microbiome RNASeq (metatranscriptomics) has emerged as a powerful technology to functionally interrogate microbial communities. A key challenge is how best to process, analyze, and interpret these complex datasets. In a typical application, a single metatranscriptomic dataset may comprise from tens to hundreds of millions of sequence reads. These reads must first be processed and filtered for low quality and potential contaminants, before being annotated with taxonomic and functional labels and subsequently collated to generate global bacterial gene expression profiles.ResultsHere, we present MetaPro, a flexible, massively scalable metatranscriptomic data analysis pipeline that is cross-platform compatible through its implementation within a Docker framework. MetaPro starts with raw sequence read input (single-end or paired-end reads) and processes them through a tiered series of filtering, assembly, and annotation steps. In addition to yielding a final list of bacterial genes and their relative expression, MetaPro delivers a taxonomic breakdown based on the consensus of complementary prediction algorithms, together with a focused breakdown of enzymes, readily visualized through the Cytoscape network visualization tool. We benchmark the performance of MetaPro against two current state-of-the-art pipelines and demonstrate improved performance and functionality.ConclusionsMetaPro represents an effective integrated solution for the processing and analysis of metatranscriptomic datasets. Its modular architecture allows new algorithms to be deployed as they are developed, ensuring its longevity. To aid user uptake of the pipeline, MetaPro, together with an established tutorial that has been developed for educational purposes, is made freely available at https://github.com/ParkinsonLab/MetaPro. The software is freely available under the GNU general public license v3.9LNNdXLsCA96o8Y-9qmMqKVideo

NeuroWRAP: integrating, validating, and sharing neurodata analysis workflows.

Multiphoton calcium imaging is one of the most powerful tools in modern neuroscience. However, multiphoton data require significant pre-processing of images and post-processing of extracted signals. As a result, many algorithms and pipelines have been developed for the analysis of multiphoton data, particularly two-photon imaging data. Most current studies use one of several algorithms and pipelines that are published and publicly available, and add customized upstream and downstream analysis elements to fit the needs of individual researchers. The vast differences in algorithm choices, parameter settings, pipeline composition, and data sources combine to make collaboration difficult, and raise questions about the reproducibility and robustness of experimental results. We present our solution, called NeuroWRAP (www.neurowrap.org), which is a tool that wraps multiple published algorithms together, and enables integration of custom algorithms. It enables development of collaborative, shareable custom workflows and reproducible data analysis for multiphoton calcium imaging data enabling easy collaboration between researchers. NeuroWRAP implements an approach to evaluate the sensitivity and robustness of the configured pipelines. When this sensitivity analysis is applied to a crucial step of image analysis, cell segmentation, we find a substantial difference between two popular workflows, CaImAn and Suite2p. NeuroWRAP harnesses this difference by introducing consensus analysis, utilizing two workflows in conjunction to significantly increase the trustworthiness and robustness of cell segmentation results.

Robustness and reproducibility of simple and complex synthetic logic circuit designs using a DBTL loop.

Computational tools addressing various components of design-build-test-learn (DBTL) loops for the construction of synthetic genetic networks exist but do not generally cover the entire DBTL loop. This manuscript introduces an end-to-end sequence of tools that together form a DBTL loop called Design Assemble Round Trip (DART). DART provides rational selection and refinement of genetic parts to construct and test a circuit. Computational support for experimental process, metadata management, standardized data collection and reproducible data analysis is provided via the previously published Round Trip (RT) test-learn loop. The primary focus of this work is on the Design Assemble (DA) part of the tool chain, which improves on previous techniques by screening up to thousands of network topologies for robust performance using a novel robustness score derived from dynamical behavior based on circuit topology only. In addition, novel experimental support software is introduced for the assembly of genetic circuits. A complete design-through-analysis sequence is presented using several OR and NOR circuit designs, with and without structural redundancy, that are implemented in budding yeast. The execution of DART tested the predictions of the design tools, specifically with regard to robust and reproducible performance under different experimental conditions. The data analysis depended on a novel application of machine learning techniques to segment bimodal flow cytometry distributions. Evidence is presented that, in some cases, a more complex build may impart more robustness and reproducibility across experimental conditions. Graphical Abstract.

EXPLORE: a novel deep learning-based analysis method for exploration behaviour in object recognition tests

Object recognition tests are widely used in neuroscience to assess memory function in rodents. Despite the experimental simplicity of the task, the interpretation of behavioural features that are counted as object exploration can be complicated. Thus, object exploration is often analysed by manual scoring, which is time-consuming and variable across researchers. Current software using tracking points often lacks precision in capturing complex ethological behaviour. Switching or losing tracking points can bias outcome measures. To overcome these limitations we developed “EXPLORE”, a simple, ready-to use and open source pipeline. EXPLORE consists of a convolutional neural network trained in a supervised manner, that extracts features from images and classifies behaviour of rodents near a presented object. EXPLORE achieves human-level accuracy in identifying and scoring exploration behaviour and outperforms commercial software with higher precision, higher versatility and lower time investment, in particular in complex situations. By labeling the respective training data set, users decide by themselves, which types of animal interactions on objects are in- or excluded, ensuring a precise analysis of exploration behaviour. A set of graphical user interfaces (GUIs) provides a beginning-to-end analysis of object recognition tests, accelerating a fast and reproducible data analysis without the need of expertise in programming or deep learning.

BIRCH: An Automated Workflow for Evaluation, Correction, and Visualization of Batch Effect in Bottom-Up Mass Spectrometry-Based Proteomics Data.

Recent surges in large-scale mass spectrometry (MS)-based proteomics studies demand a concurrent rise in methods to facilitate reliable and reproducible data analysis. Quantification of proteins in MS analysis can be affected by variations in technical factors such as sample preparation and data acquisition conditions leading to batch effects, which adds to noise in the data set. This may in turn affect the effectiveness of any biological conclusions derived from the data. Here we present Batch-effect Identification, Representation, and Correction of Heterogeneous data (BIRCH), a workflow for analysis and correction of batch effect through an automated, versatile, and easy to use web-based tool with the goal of eliminating technical variation. BIRCH also supports diagnosis of the data to check for the presence of batch effects, feasibility of batch correction, and imputation to deal with missing values in the data set. To illustrate the relevance of the tool, we explore two case studies, including an iPSC-derived cell study and a Covid vaccine study to show different context-specific use cases. Ultimately this tool can be used as an extremely powerful approach for eliminating technical bias while retaining biological bias, toward understanding disease mechanisms and potential therapeutics.

A Fiji process for quantifying fluorescent puncta in linear cellular structures.

Understanding the cell biology of protein trafficking and homeostasis requires reproducible methods for identifying and quantifying proteins within cells or cellular structures. Imaging protocols for measuring punctate protein accumulation in linear structures, for example the neurites of C. elegans, have relied on proprietary software for a full range of analysis capabilities. Here we describe a set of macros written for the NIH-supported imaging software ImageJ or Fiji (Fiji is Just ImageJ) that reliably identify protein puncta so that they can be analyzed with respect to intensity, density, and width at half-maximum intensity (Full-Width, Half-Maximum, FWHM). We provide an explanation of the workflow, data outputs, and limitations of the Fiji macro. As part of this integration, we also provide two independent data sets with side-by-side analyses using the proprietary IgorPro software and the Fiji macro (Hulsey-Vincent, et al. A, B., 2023 submitted). The Fiji macro is an important new tool because it provides robust, reproducible data analysis in a free, open-source format.

Nanopore Direct RNA Sequencing Data Processing and Analysis Using MasterOfPores.

This chapter describes MasterOfPores v.2 (MoP2), an open-source suite of pipelines for processing and analyzing direct RNA Oxford Nanopore sequencing data. The MoP2 relies on the Nextflow DSL2 framework and Linux containers, thus enabling reproducible data analysis in transcriptomic and epitranscriptomic studies. We introduce the key concepts of MoP2 and provide a step-by-step fully reproducible and complete example of how to use the workflow for the analysis of S. cerevisiae total RNA samples sequenced using MinION flowcells. The workflow starts with the pre-processing of raw FAST5 files, which includes basecalling, read quality control, demultiplexing, filtering, mapping, estimation of per-gene/transcript abundances, and transcriptome assembly, with support of the GPU computing for the basecalling and read demultiplexing steps. The secondary analyses of the workflow focus on the estimation of RNA poly(A) tail lengths and the identification of RNA modifications. The MoP2 code is available at https://github.com/biocorecrg/MOP2 and is distributed under the MIT license.

Better antimicrobial resistance data analysis and reporting in less time.

Insights about local antimicrobial resistance (AMR) levels and epidemiology are essential to guide decision-making processes in antimicrobial use. However, dedicated tools for reliable and reproducible AMR data analysis and reporting are often lacking. We aimed to compare traditional data analysis and reporting versus a new approach for reliable and reproducible AMR data analysis in a clinical setting. Ten professionals who routinely work with AMR data were provided with blood culture test results including antimicrobial susceptibility results. Participants were asked to perform a detailed AMR data analysis in a two-round process: first using their software of choice and next using our newly developed software tool. Accuracy of the results and time spent were compared between both rounds. Finally, participants rated the usability using the System Usability Scale (SUS). The mean time spent on creating the AMR report reduced from 93.7 to 22.4 min (P < 0.001). Average task completion per round changed from 56% to 96% (P < 0.05). The proportion of correct answers in the available results increased from 37.9% in the first to 97.9% in the second round (P < 0.001). Usability of the new tools was rated with a median of 83.8 (out of 100) on the SUS. This study demonstrated the significant improvement in efficiency and accuracy in standard AMR data analysis and reporting workflows through open-source software. Integrating these tools in clinical settings can democratize the access to fast and reliable insights about local microbial epidemiology and associated AMR levels. Thereby, our approach can support evidence-based decision-making processes in the use of antimicrobials.