Automated Data Analysis Pipeline Research Articles

Synchrotron Big Data Science.

The rapid development of synchrotrons has massively increased the speed at which experiments can be performed, while new techniques have increased the amount of raw data collected during each experiment. While this has created enormous new opportunities, it has also created tremendous challenges for national facilities and users. With the huge increase in data volume, the manual analysis of data is no longer possible. As a result, only a fraction of the data collected during the time- and money-expensive synchrotron beam-time is analyzed and used to deliver new science. Additionally, the lack of an appropriate data analysis environment limits the realization of experiments that generate a large amount of data in a very short period of time. The current lack of automated data analysis pipelines prevents the fine-tuning of beam-time experiments, further reducing their potential usage. These effects, collectively known as the "data deluge," affect synchrotrons in several different ways including fast data collection, available local storage, data management systems, and curation of the data. This review highlights the Big Data strategies adopted nowadays at synchrotrons, documenting this novel and promising hybridization between science and technology, which promise a dramatic increase in the number of scientific discoveries.

Machine Learning Methods for X-Ray Scattering Data Analysis from Biomacromolecular Solutions

Small-angle x-ray scattering (SAXS) of biological macromolecules in solutions is a widely employed method in structural biology. SAXS patterns include information about the overall shape and low-resolution structure of dissolved particles. Here, we describe how to transform experimental SAXS patterns to feature vectors and how a simple k-nearest neighbor approach is able to retrieve information on overall particle shape and maximal diameter (Dmax) as well as molecular mass directly from experimental scattering data. Based on this transformation, we develop a rapid multiclass shape-classification ranging from compact, extended, and flat categories to hollow and random-chain-like objects. This classification may be employed, e.g., as a decision block in automated data analysis pipelines. Further, we map protein structures from the Protein Data Bank into the classification space and, in a second step, use this mapping as a data source to obtain accurate estimates for the structural parameters (Dmax, molecular mass) of the macromolecule under study based on the experimental scattering pattern alone, without inverse Fourier transform for Dmax. All methods presented are implemented in a Fortran binary DATCLASS, part of the ATSAS data analysis suite, available on Linux, Mac, and Windows and free for academic use.

Standardized Immunophenotyping in the Canadian National Transplant Research Program

Background: Immune monitoring of post-transplant blood by flow cytometry (FCM) can identify features of immune tolerance and may help predict rejection. Discovery and validation of clinically useful biomarkers requires the analysis of large numbers of samples. Incorporating standardized FCM panels into clinical research enables the merging of data across time, site and study. We previously established standardized immunophenotyping at several sites across Canada using blood from healthy donors, and have now applied these protocols to the analysis of post hematopoietic stem cell transplant (HSCT) samples and added automated data analysis pipelines. Methods: Peripheral blood from 11 HSCT recipients 100d post-transplant was analysed within 4 h of collection using six DuraClone IM dry format panels originally developed by The ONE Study1. Blood was aged for 24 h prior to isolation and freezing of PBMCs, and distributed to three sites for comparative analysis. Relative biological variation was expressed as intraclass correlation coefficients (ICCs). Results from central manual analysis were compared to those obtained by supervised, flow Density-based automated gating pipelines. Results: FCM of identical, 24 h-aged and thawed PBMCs from post HSCT patients at 3 sites yielded very similar data (ICC > 0.75 for most values); ICC values were not significantly lower than those obtained with freshly frozen PBMCs from healthy donors, with the exception of some monocyte and B cell subtypes. Automated gating pipelines were created to extract data from healthy and post HSCT samples that fell within the range defined by three independent manual analyses. Finally, merging of phase 1 (healthy donors) and phase 2 (100 d post HSCT patients) studies revealed that monocytes were increased and lymphocytes (especially B cells) were decreased in the latter; B cells were shifted in favour of immature/transitional populations, while no significant differences were seen in the T cell populations. Conclusions: Standardized immunophenotyping can be successfully applied to notoriously difficult (aged & thawed post-transplant) samples. Whole blood-derived data obtained in separate, multi-site studies can be merged and automated gating pipelines can provide accurate and robust analysis of blood from very different donor types. Thus, this technology is suited to large-scale studies for the discovery and validation of clinically relevant biomarkers.

Flexible Data Analysis Pipeline for High-Confidence Proteogenomics.

Proteogenomics leverages information derived from proteomic data to improve genome annotations. Of particular interest are “novel” peptides that provide direct evidence of protein expression for genomic regions not previously annotated as protein-coding. We present a modular, automated data analysis pipeline aimed at detecting such “novel” peptides in proteomic data sets. This pipeline implements criteria developed by proteomics and genome annotation experts for high-stringency peptide identification and filtering. Our pipeline is based on the OpenMS computational framework; it incorporates multiple database search engines for peptide identification and applies a machine-learning approach (Percolator) to post-process search results. We describe several new and improved software tools that we developed to facilitate proteogenomic analyses that enhance the wealth of tools provided by OpenMS. We demonstrate the application of our pipeline to a human testis tissue data set previously acquired for the Chromosome-Centric Human Proteome Project, which led to the addition of five new gene annotations on the human reference genome.

Prospects for high-precision pulsar timing with the new Effelsberg PSRIX backend

The PSRIX backend is the primary pulsar timing instrument of the Effelsberg 100-m radio telescope since early 2011. This new ROACH-based system enables bandwidths up to 500 MHz to be recorded, significantly more than what was possible with its predecessor, the Effelsberg-Berkeley Pulsar Processor (EBPP). We review the first four years of PSRIX timing data for 33 pulsars collected as part of the monthly European Pulsar Timing Array (EPTA) observations. We describe the automated data analysis pipeline, CoastGuard, that we developed to reduce these observations. We also introduce TOASTER, the EPTA timing database used to store timing results, processing information and observation metadata. Using these new tools, we measure the phase-averaged flux densities at 1.4 GHz of all 33 pulsars. For 7 of these pulsars, our flux density measurements are the first values ever reported. For the other 26 pulsars, we compare our flux density measurements with previously published values. By comparing PSRIX data with EBPP data, we find an improvement of ~2-5 times in signal-to-noise ratio achievable, which translates to an increase of ~2-5 times in pulse time-of-arrival (TOA) precision. We show that such an improvement in TOA precision will improve the sensitivity to the stochastic gravitational wave background. Finally, we showcase the flexibility of the new PSRIX backend by observing several millisecond-period pulsars (MSPs) at 5 and 9 GHz. Motivated by our detections, we discuss the potential for complementing existing pulsar timing array data sets with MSP monitoring campaigns at these frequencies.

PMD: A Resource for Archiving and Analyzing Protein Microarray data.

Protein microarray is a powerful technology for both basic research and clinical study. However, because there is no database specifically tailored for protein microarray, the majority of the valuable original protein microarray data is still not publically accessible. To address this issue, we constructed Protein Microarray Database (PMD), which is specifically designed for archiving and analyzing protein microarray data. In PMD, users can easily browse and search the entire database by experimental name, protein microarray type, and sample information. Additionally, PMD integrates several data analysis tools and provides an automated data analysis pipeline for users. With just one click, users can obtain a comprehensive analysis report for their protein microarray data. The report includes preliminary data analysis, such as data normalization, candidate identification, and an in-depth bioinformatics analysis of the candidates, which include functional annotation, pathway analysis, and protein-protein interaction network analysis. PMD is now freely available at www.proteinmicroarray.cn.

Automated pipeline for purification, biophysical and x-ray analysis of biomacromolecular solutions.

Small angle X-ray scattering (SAXS), an increasingly popular method for structural analysis of biological macromolecules in solution, is often hampered by inherent sample polydispersity. We developed an all-in-one system combining in-line sample component separation with parallel biophysical and SAXS characterization of the separated components. The system coupled to an automated data analysis pipeline provides a novel tool to study difficult samples at the P12 synchrotron beamline (PETRA-3, EMBL/DESY, Hamburg).

Mapping differential interactomes by affinity purification coupled with data-independent mass spectrometry acquisition

Characterizing changes in protein-protein interactions associated with sequence variants (e.g. disease-associated mutations or splice forms) or following exposure to drugs, growth factors or hormones is critical to understanding how protein complexes are built, localized and regulated. Affinity purification (AP) coupled with mass spectrometry permits the analysis of protein interactions under near-physiological conditions, yet monitoring interaction changes requires the development of a robust and sensitive quantitative approach, especially for large-scale studies where cost and time are major considerations. To this end, we have coupled AP to data-independent mass spectrometric acquisition (SWATH), and implemented an automated data extraction and statistical analysis pipeline to score modulated interactions. Here, we use AP-SWATH to characterize changes in protein-protein interactions imparted by the HSP90 inhibitor NVP-AUY922 or melanoma-associated mutations in the human kinase CDK4. We show that AP-SWATH is a robust label-free approach to characterize such changes, and propose a scalable pipeline for systems biology studies.

B Cells With High Side Scatter Parameter by Flow Cytometry Correlate With Inferior Survival in Diffuse Large B-Cell Lymphoma

Despite advances in the understanding of diffuse large B-cell lymphoma (DLBCL) biology, only the clinically based International Prognostic Index (IPI) is used routinely for risk stratification at diagnosis. To find novel prognostic markers, we analyzed flow cytometric data from 229 diagnostic DLBCL samples using an automated multiparameter data analysis approach developed in our laboratory. By using the developed automated data analysis pipeline, we identified 71 of 229 cases as having more than 35% B cells with a high side scatter (SSC) profile, a parameter reflecting internal cellular complexity. This high SSC B-cell feature was associated with inferior overall and progression-free survival (P = .001 and P = .01, respectively) and remained a significant predictor of overall survival in multivariate Cox regression analysis (IPI, P = .001; high SSC, P = .004; rituximab, P = .53). This study suggests that high SSC among B cells may serve as a useful biomarker to identify patients with DLBCL at high risk for relapse. This is of particular interest because this biomarker is readily available in most clinical laboratories without significant alteration to existing routine diagnostic strategies or incurring additional costs.

Acoustic spectrum fitting for a large set of solar-like pulsators

Asteroseismology provides the means both to constrain the global properties and to probe the internal structures of stars. Asteroseismic data are now available on large numbers of solar-type stars, thanks in particular to the CoRoT and Kepler space missions, and automated data-analysis pipelines are needed to provide efficient and timely results. Here, we present an automated algorithm that is able to extract mode parameters under low signal-to-noise ratio conditions. We use a Bayesian framework to ensure the robustness of the algorithm. We discuss the efficiency of the method and test it using Variability of Solar Irradiance and Gravity Oscillations (VIRGO) Sun-as-a-star photometry data and the artificial Astero Fitting at Low Angular degree Group (asteroFLAG) Kepler ensemble. Analysis of the VIRGO data shows that it is possible to track variations of the individual mode parameters (frequency, height, width) through the solar cycle, using short time series (30 days). The present analysis also revealed a modulation of the degree l = 2 relative height through the solar cycle. Applied on asteroFLAG data, we show that the pipeline extracts accurately the central frequency and the large separation. It is also able to identify the degree of the modes in 78 per cent of stars.

Modeling and automation of sequencing-based characterization of RNA structure

Sequence census methods reduce molecular measurements such as transcript abundance and protein-nucleic acid interactions to counting problems via DNA sequencing. We focus on a novel assay utilizing this approach, called selective 2'-hydroxyl acylation analyzed by primer extension sequencing (SHAPE-Seq), that can be used to characterize RNA secondary and tertiary structure. We describe a fully automated data analysis pipeline for SHAPE-Seq analysis that includes read processing, mapping, and structural inference based on a model of the experiment. Our methods rely on the solution of a series of convex optimization problems for which we develop efficient and effective numerical algorithms. Our results can be easily extended to other chemical probes of RNA structure, and also generalized to modeling polymerase drop-off in other sequence census-based experiments.

Multiplexed RNA structure characterization with selective 2′-hydroxyl acylation analyzed by primer extension sequencing (SHAPE-Seq)

New regulatory roles continue to emerge for both natural and engineered noncoding RNAs, many of which have specific secondary and tertiary structures essential to their function. Thus there is a growing need to develop technologies that enable rapid characterization of structural features within complex RNA populations. We have developed a high-throughput technique, SHAPE-Seq, that can simultaneously measure quantitative, single nucleotide-resolution secondary and tertiary structural information for hundreds of RNA molecules of arbitrary sequence. SHAPE-Seq combines selective 2'-hydroxyl acylation analyzed by primer extension (SHAPE) chemistry with multiplexed paired-end deep sequencing of primer extension products. This generates millions of sequencing reads, which are then analyzed using a fully automated data analysis pipeline, based on a rigorous maximum likelihood model of the SHAPE-Seq experiment. We demonstrate the ability of SHAPE-Seq to accurately infer secondary and tertiary structural information, detect subtle conformational changes due to single nucleotide point mutations, and simultaneously measure the structures of a complex pool of different RNA molecules. SHAPE-Seq thus represents a powerful step toward making the study of RNA secondary and tertiary structures high throughput and accessible to a wide array of scientific pursuits, from fundamental biological investigations to engineering RNA for synthetic biological systems.

An Automated Data Analysis Pipeline for GC−TOF−MS Metabonomics Studies

Recent technological advances have made it possible to carry out high-throughput metabonomics studies using gas chromatography coupled with time-of-flight mass spectrometry. Large volumes of data are produced from these studies and there is a pressing need for algorithms that can efficiently process and analyze data in a high-throughput fashion as well. We present an Automated Data Analysis Pipeline (ADAP) that has been developed for this purpose. ADAP consists of peak detection, deconvolution, peak alignment, and library search. It allows data to flow seamlessly through the analysis steps without any human intervention and features two novel algorithms in the analysis. Specifically, clustering is successfully applied in deconvolution to resolve coeluting compounds that are very common in complex samples and a two-phase alignment process has been implemented to enhance alignment accuracy. ADAP is written in standard C++ and R and uses parallel computing via Message Passing Interface for fast peak detection and deconvolution. ADAP has been applied to analyze both mixed standards samples and serum samples and identified and quantified metabolites successfully. ADAP is available at http://www.du-lab.org .

Computerized image analysis for quantitative neuronal phenotyping in zebrafish

An integrated microscope image analysis pipeline is developed for automatic analysis and quantification of phenotypes in zebrafish with altered expression of Alzheimer's disease (AD)-linked genes. We hypothesize that a slight impairment of neuronal integrity in a large number of zebrafish carrying the mutant genotype can be detected through the computerized image analysis method. Key functionalities of our zebrafish image processing pipeline include quantification of neuron loss in zebrafish embryos due to knockdown of AD-linked genes, automatic detection of defective somites, and quantitative measurement of gene expression levels in zebrafish with altered expression of AD-linked genes or treatment with a chemical compound. These quantitative measurements enable the archival of analyzed results and relevant meta-data. The structured database is organized for statistical analysis and data modeling to better understand neuronal integrity and phenotypic changes of zebrafish under different perturbations. Our results show that the computerized analysis is comparable to manual counting with equivalent accuracy and improved efficacy and consistency. Development of such an automated data analysis pipeline represents a significant step forward to achieve accurate and reproducible quantification of neuronal phenotypes in large scale or high-throughput zebrafish imaging studies.

The FIRST Survey: Faint Images of the Radio Sky at Twenty Centimeters

The FIRST survey to produce Faint Images of the Radio Sky at Twenty centimeters is now underway using the NRAO Very Large Array. We describe here the scientific motivation for a large-area sky survey at radio frequencies which has a sensitivity and angular resolution comparable to the Palomar Observatory Sky Survey, and we recount the history that led to the current survey project. The technical design of the survey is covered in detail, including a description and justification of the grid pattern chosen, the rationale behind the integration time and angular resolution selected, and a summary of the other considerations which informed our planning for the project. A comprehensive description of the automated data analysis pipeline we have developed is presented. We also report here the results of the first year of FIRST observations. A total of 144 hr of time in 1993 April and May was used for a variety of tests, as well as to cover an initial strip of the survey extending between 07{sup h}15{sup m} and 16{sup h}30{sup m} in a 2{sq_bullet}8 wide declination zone passing through the local zenith (28.2{lt}{delta}{lt}31.0). A total of 2153 individual pointings yielded an image database containing 1039 merged images 46{sq_bullet}5{times}34{sq_bullet}5more » in extent with 1{center_dot}{double_prime}8 pixels and a typical rms of 0.13 mJy. A catalog derived from this 300 deg{sup 2} region contains 28,000 radio sources. We have performed extensive tests on the images and source list in order to establish the photometric and astrometric accuracy of these data products. We find systematic astrometric errors of {lt}0{center_dot}{double_prime}05; individual sources down to the 1 mJy survey flux density threshold have 90{percent} confidence error circles with radii of {lt}1{double_prime}. CLEAN bias introduces a systematic underestimate of point-source flux densities of {approximately}0.25 mJy; the bias is more severe for extended sources. (Abstract Truncated)« less