Ultra-large Datasets Research Articles

SCAMPP: Scaling Alignment-Based Phylogenetic Placement to Large Trees.

Phylogenetic placement, the problem of placing a "query" sequence into a precomputed phylogenetic "backbone" tree, is useful for constructing large trees, performing taxon identification of newly obtained sequences, and other applications. The most accurate current methods, such as pplacer and EPA-ng, are based on maximum likelihood and require that the query sequence be provided within a multiple sequence alignment that includes the leaf sequences in the backbone tree. This approach enables high accuracy but also makes these likelihood-based methods computationally intensive on large backbone trees, and can even lead to them failing when the backbone trees are very large (e.g., having 50,000 or more leaves). We present SCAMPP (SCaling AlignMent-based Phylogenetic Placement), a technique to extend the scalability of these likelihood-based placement methods to ultra-large backbone trees. We show that pplacer-SCAMPP and EPA-ng-SCAMPP both scale well to ultra-large backbone trees (even up to 200,000 leaves), with accuracy that improves on APPLES and APPLES-2, two recently developed fast phylogenetic placement methods that scale to ultra-large datasets. EPA-ng-SCAMPP and pplacer-SCAMPP are available at https://github.com/chry04/PLUSplacer.

Accurate and fast cell marker gene identification with COSG.

Accurate cell classification is the groundwork for downstream analysis of single-cell sequencing data, yet how to identify true marker genes for different cell types still remains a big challenge. Here, we report COSine similarity-based marker Gene identification (COSG) as a cosine similarity-based method for more accurate and scalable marker gene identification. COSG is applicable to single-cell RNA sequencing data, single-cell ATAC sequencing data and spatially resolved transcriptome data. COSG is fast and scalable for ultra-large datasets of million-scale cells. Application on both simulated and real experimental datasets showed that the marker genes or genomic regions identified by COSG have greater cell-type specificity, demonstrating the superior performance of COSG in terms of both accuracy and efficiency as compared with other available methods.

SaAlign: Multiple DNA/RNA sequence alignment and phylogenetic tree construction tool for ultra-large datasets and ultra-long sequences based on suffix array

Multiple DNA/RNA sequence alignment is an important fundamental tool in bioinformatics, especially for phylogenetic tree construction. With DNA-sequencing improvements, the amount of bioinformatics data is constantly increasing, and various tools need to be iterated constantly. Mitochondrial genome analyses of multiple individuals and species require bioinformatics software; therefore, their performances need to be optimized. To improve the alignment of ultra-large datasets and ultra-long sequences, we optimized a dynamic programming algorithm using longest common substring methods. Ultra-large test DNA datasets, containing sequences of different lengths, some over 300 kb (kilobase), revealed that the Multiple DNA/RNA Sequence Alignment Tool Based on Suffix Tree (SaAlign) saved time and computational space. It outperformed the existing technical tools, including MAFFT and HAlign-II. For mitochondrial genome datasets having limited numbers of sequences, MAFFT performed the required tasks, but it could not handle ultra-large mitochondrial genome datasets for core dump error. We implement a multiple DNA/RNA sequence alignment tool based on Center Star strategy and use suffix array algorithm to optimize the spatial and time efficiency. Nowadays, whole-genome research and NGS technology are becoming more popular, and it is necessary to save computational resources for laboratories. That software is of great significance in these aspects, especially in the study of the whole-mitochondrial genome of plants.

Advancing Genetic Selection and Behavioral Genomics of Working Dogs Through Collaborative Science.

The ancient partnership between people and dogs is struggling to meet modern day needs, with demand exceeding our capacity to safely breed high-performing and healthy dogs. New statistical genetic approaches and genomic technology have the potential to revolutionize dog breeding, by transitioning from problematic phenotypic selection to methods that can preserve genetic diversity while increasing the proportion of successful dogs. To fully utilize this technology will require ultra large datasets, with hundreds of thousands of dogs. Today, dog breeders struggle to apply even the tools available now, stymied by the need for sophisticated data storage infrastructure and expertise in statistical genetics. Here, we review recent advances in animal breeding, and how a new approach to dog breeding would address the needs of working dog breeders today while also providing them with a path to realizing the next generation of technology. We provide a step-by-step guide for dog breeders to start implementing estimated breeding value selection in their programs now, and we describe how genotyping and DNA sequencing data, as it becomes more widely available, can be integrated into this approach. Finally, we call for data sharing among dog breeding programs as a path to achieving a future that can benefit all dogs, and their human partners too.

Aggregation of information from diverse networks as the basis for training cyber security specialists on processing ultra large data sets

Aggregation of information from diverse networks as the basis for training cyber security specialists on processing ultra large data sets

Multiple Sequence Alignment for Large Heterogeneous Datasets Using SATé, PASTA, and UPP.

The estimation of very large multiple sequence alignments is a challenging problem that requires special techniques in order to achieve high accuracy. Here we describe two software packages-PASTA and UPP-for constructing alignments on large and ultra-large datasets. Both methods have been able to produce highly accurate alignments on 1,000,000 sequences, and trees computed on these alignments are also highly accurate. PASTA provides the best tree accuracy when the input sequences are all full-length, but UPP provides improved accuracy compared to PASTA and other methods when the input contains a large number of fragmentary sequences. Both methods are available in open source form on GitHub.

A novel feature representation: Aggregating convolution kernels for image retrieval

Activated hidden units in convolutional neural networks (CNNs), known as feature maps, dominate image representation, which is compact and discriminative. For ultra-large datasets, high dimensional feature maps in float format not only result in high computational complexity, but also occupy massive memory space. To this end, a new image representation by aggregating convolution kernels (ACK) is proposed, where some convolution kernels capturing certain patterns are activated. The top-n index numbers of the convolution kernels are extracted directly as image representation in discrete integer values, which rebuild relationship between convolution kernels and image. Furthermore, a distance measurement is defined from the perspective of ordered sets to calculate position-sensitive similarities between image representations. Extensive experiments conducted on Oxford Buildings, Paris, and Holidays, etc., manifest that the proposed ACK achieves competitive performance on image retrieval with much lower computational cost, outperforming the ones using feature maps for image representation.

Resolving single-cell heterogeneity from hundreds of thousands of cells through sequential hybrid clustering and NMF.

MotivationThe rapid proliferation of single-cell RNA-sequencing (scRNA-Seq) technologies has spurred the development of diverse computational approaches to detect transcriptionally coherent populations. While the complexity of the algorithms for detecting heterogeneity has increased, most require significant user-tuning, are heavily reliant on dimension reduction techniques and are not scalable to ultra-large datasets. We previously described a multi-step algorithm, Iterative Clustering and Guide-gene Selection (ICGS), which applies intra-gene correlation and hybrid clustering to uniquely resolve novel transcriptionally coherent cell populations from an intuitive graphical user interface.ResultsWe describe a new iteration of ICGS that outperforms state-of-the-art scRNA-Seq detection workflows when applied to well-established benchmarks. This approach combines multiple complementary subtype detection methods (HOPACH, sparse non-negative matrix factorization, cluster ‘fitness’, support vector machine) to resolve rare and common cell-states, while minimizing differences due to donor or batch effects. Using data from multiple cell atlases, we show that the PageRank algorithm effectively downsamples ultra-large scRNA-Seq datasets, without losing extremely rare or transcriptionally similar yet distinct cell types and while recovering novel transcriptionally distinct cell populations. We believe this new approach holds tremendous promise in reproducibly resolving hidden cell populations in complex datasets.Availability and implementationICGS2 is implemented in Python. The source code and documentation are available at http://altanalyze.org.Supplementary information Supplementary data are available at Bioinformatics online.

Deep Representation Learning for Complex Free-Energy Landscapes.

In this Letter, we analyzed the inductive bias underlying complex free-energy landscapes (FELs) and exploited it to train deep neural networks that yield reduced and clustered representation for the FEL. Our parametric method, called information distilling of metastability (IDM), is end-to-end differentiable and thus scalable to ultralarge data sets. IDM is able to perform clustering in the meantime of reducing the dimensionality. Besides, as an unsupervised learning method, IDM differs from many existing dimensionality reduction and clustering methods in that it requires neither a cherry-picked distance metric nor the ground-true number of clusters defined a priori, and it can be used to unroll and zoom in on the hierarchical FEL with respect to different time scales. Through multiple experiments, we show that IDM can achieve physically meaningful representations that partition the FEL into well-defined metastable states that hence are amenable for downstream tasks such as mechanism analysis and kinetic modeling.

Constrained incremental tree building: new absolute fast converging phylogeny estimation methods with improved scalability and accuracy

BackgroundAbsolute fast converging (AFC) phylogeny estimation methods are ones that have been proven to recover the true tree with high probability given sequences whose lengths are polynomial in the number of number of leaves in the tree (once the shortest and longest branch weights are fixed). While there has been a large literature on AFC methods, the best in terms of empirical performance was DCM_{NJ}, published in SODA 2001. The main empirical advantage of {DCM}_{NJ} over other AFC methods is its use of neighbor joining (NJ) to construct trees on smaller taxon subsets, which are then combined into a tree on the full set of species using a supertree method; in contrast, the other AFC methods in essence depend on quartet trees that are computed independently of each other, which reduces accuracy compared to neighbor joining. However, {DCM}_{NJ} is unlikely to scale to large datasets due to its reliance on supertree methods, as no current supertree methods are able to scale to large datasets with high accuracy.ResultsIn this study we present a new approach to large-scale phylogeny estimation that shares some of the features of {DCM}_{NJ} but bypasses the use of supertree methods. We prove that this new approach is AFC and uses polynomial time and space. Furthermore, we describe variations on this basic approach that can be used with leaf-disjoint constraint trees (computed using methods such as maximum likelihood) to produce other methods that are likely to provide even better accuracy. Thus, we present a new generalizable technique for large-scale tree estimation that is designed to improve scalability for phylogeny estimation methods to ultra-large datasets, and that can be used in a variety of settings (including tree estimation from unaligned sequences, and species tree estimation from gene trees).

MegaGTA: a sensitive and accurate metagenomic gene-targeted assembler using iterative de Bruijn graphs

BackgroundThe recent release of the gene-targeted metagenomics assembler Xander has demonstrated that using the trained Hidden Markov Model (HMM) to guide the traversal of de Bruijn graph gives obvious advantage over other assembly methods. Xander, as a pilot study, indeed has a lot of room for improvement. Apart from its slow speed, Xander uses only 1 k-mer size for graph construction and whatever choice of k will compromise either sensitivity or accuracy. Xander uses a Bloom-filter representation of de Bruijn graph to achieve a lower memory footprint. Bloom filters bring in false positives, and it is not clear how this would impact the quality of assembly. Xander does not keep track of the multiplicity of k-mers, which would have been an effective way to differentiate between erroneous k-mers and correct k-mers.ResultsIn this paper, we present a new gene-targeted assembler MegaGTA, which attempts to improve Xander in different aspects. Quality-wise, it utilizes iterative de Bruijn graphs to take full advantage of multiple k-mer sizes to make the best of both sensitivity and accuracy. Computation-wise, it employs succinct de Bruijn graphs (SdBG) to achieve low memory footprint and high speed (the latter is benefited from a highly efficient parallel algorithm for constructing SdBG). Unlike Bloom filters, an SdBG is an exact representation of a de Bruijn graph. It enables MegaGTA to avoid false-positive contigs and to easily incorporate the multiplicity of k-mers for building better HMM model.We have compared MegaGTA and Xander on an HMP-defined mock metagenomic dataset, and showed that MegaGTA excelled in both sensitivity and accuracy. On a large rhizosphere soil metagenomic sample (327Gbp), MegaGTA produced 9.7–19.3% more contigs than Xander, and these contigs were assigned to 10–25% more gene references. In our experiments, MegaGTA, depending on the number of k-mers used, is two to ten times faster than Xander.ConclusionMegaGTA improves on the algorithm of Xander and achieves higher sensitivity, accuracy and speed. Moreover, it is capable of assembling gene sequences from ultra-large metagenomic datasets. Its source code is freely available at https://github.com/HKU-BAL/megagta .

MO-FG-207B-02: Breast

State-of-the-Art in Radiomics in Radiology and Radiation Oncology Radiomics is the science of converting medical images into mineable data, data that are descriptive of “phenotypes,” which may provide diagnostic, prognostic, or therapeutic information. Genomics is the science of sequencing and analyzing the function and structure of genomes; the complete set of DNA in a single cell of an organism. In turn, imaging genomics (or radiogenomics) is concerned with the correlation between image-based features, as determined by radiomics, and gene expression, as determined by genomics. Imaging genomics arose from decades of work in at least three key areas: 1) sequencing of the human genome, 2) quantitative imaging, computer-aided diagnosis, therapy prognostics, and assessment of therapy response in preclinical and clinical research and practice; and 3) data science, including the burgeoning areas of genomics, preclinical and population-based disease modeling, individualized medicine, and big data. Imaging genomics may answer important questions in medicine by correlating validated, quantitative image phenotypes (through validated imaging biomarkers) with clinical data, histopathologic data, molecular classifications, genomic assays, and treatment outcomes. This approach could address some of the greatest health burdens, including cancer, cardiac disease, and arthritis. Participants will discuss the state-of-the-art of radiomics across multiple disease sites and modalities. Aspects of the presentations will include how to improve the quality of image-based phenotypes of normal and diseased tissue, how to better determine the relationships between these phenotypes and the underlying biology associated with the images, and how to create predictive models using the image-based phenotypes. Topics will also include: 1) creating, archiving, curating and sharing ultra-large datasets (“big data”); 2) standardizing image acquisition and processing methods; 3) standardizing and validating phenotype extraction methods and classifier designs; and 4) using high-throughput, robust, and validated phenotyping systems. H. Aerts, NIH; NCIH. Li, This research was funded in part by the University of Chicago Dean Bridge Fund,and by NCI U24-CA143848-05, P50- CA58223 Breast SPORE program. Hui Li received royalties from Hologic.W. Lu, This work was supported in part by the National Cancer Institute Grants R01CA172638.

MO-FG-207B-03: Brain

State-of-the-Art in Radiomics in Radiology and Radiation Oncology Radiomics is the science of converting medical images into mineable data, data that are descriptive of “phenotypes,” which may provide diagnostic, prognostic, or therapeutic information. Genomics is the science of sequencing and analyzing the function and structure of genomes; the complete set of DNA in a single cell of an organism. In turn, imaging genomics (or radiogenomics) is concerned with the correlation between image-based features, as determined by radiomics, and gene expression, as determined by genomics. Imaging genomics arose from decades of work in at least three key areas: 1) sequencing of the human genome, 2) quantitative imaging, computer-aided diagnosis, therapy prognostics, and assessment of therapy response in preclinical and clinical research and practice; and 3) data science, including the burgeoning areas of genomics, preclinical and population-based disease modeling, individualized medicine, and big data. Imaging genomics may answer important questions in medicine by correlating validated, quantitative image phenotypes (through validated imaging biomarkers) with clinical data, histopathologic data, molecular classifications, genomic assays, and treatment outcomes. This approach could address some of the greatest health burdens, including cancer, cardiac disease, and arthritis. Participants will discuss the state-of-the-art of radiomics across multiple disease sites and modalities. Aspects of the presentations will include how to improve the quality of image-based phenotypes of normal and diseased tissue, how to better determine the relationships between these phenotypes and the underlying biology associated with the images, and how to create predictive models using the image-based phenotypes. Topics will also include: 1) creating, archiving, curating and sharing ultra-large datasets (“big data”); 2) standardizing image acquisition and processing methods; 3) standardizing and validating phenotype extraction methods and classifier designs; and 4) using high-throughput, robust, and validated phenotyping systems. H. Aerts, NIH; NCIH. Li, This research was funded in part by the University of Chicago Dean Bridge Fund,and by NCI U24-CA143848-05, P50- CA58223 Breast SPORE program. Hui Li received royalties from Hologic.W. Lu, This work was supported in part by the National Cancer Institute Grants R01CA172638.

MO-FG-207B-01: Thorax/Lung

State-of-the-Art in Radiomics in Radiology and Radiation Oncology Radiomics is the science of converting medical images into mineable data, data that are descriptive of “phenotypes,” which may provide diagnostic, prognostic, or therapeutic information. Genomics is the science of sequencing and analyzing the function and structure of genomes; the complete set of DNA in a single cell of an organism. In turn, imaging genomics (or radiogenomics) is concerned with the correlation between image-based features, as determined by radiomics, and gene expression, as determined by genomics. Imaging genomics arose from decades of work in at least three key areas: 1) sequencing of the human genome, 2) quantitative imaging, computer-aided diagnosis, therapy prognostics, and assessment of therapy response in preclinical and clinical research and practice; and 3) data science, including the burgeoning areas of genomics, preclinical and population-based disease modeling, individualized medicine, and big data. Imaging genomics may answer important questions in medicine by correlating validated, quantitative image phenotypes (through validated imaging biomarkers) with clinical data, histopathologic data, molecular classifications, genomic assays, and treatment outcomes. This approach could address some of the greatest health burdens, including cancer, cardiac disease, and arthritis. Participants will discuss the state-of-the-art of radiomics across multiple disease sites and modalities. Aspects of the presentations will include how to improve the quality of image-based phenotypes of normal and diseased tissue, how to better determine the relationships between these phenotypes and the underlying biology associated with the images, and how to create predictive models using the image-based phenotypes. Topics will also include: 1) creating, archiving, curating and sharing ultra-large datasets (“big data”); 2) standardizing image acquisition and processing methods; 3) standardizing and validating phenotype extraction methods and classifier designs; and 4) using high-throughput, robust, and validated phenotyping systems. H. Aerts, NIH; NCIH. Li, This research was funded in part by the University of Chicago Dean Bridge Fund,and by NCI U24-CA143848-05, P50- CA58223 Breast SPORE program. Hui Li received royalties from Hologic.W. Lu, This work was supported in part by the National Cancer Institute Grants R01CA172638.

MO-FG-207B-04: Respond to Therapy

State-of-the-Art in Radiomics in Radiology and Radiation Oncology Radiomics is the science of converting medical images into mineable data, data that are descriptive of “phenotypes,” which may provide diagnostic, prognostic, or therapeutic information. Genomics is the science of sequencing and analyzing the function and structure of genomes; the complete set of DNA in a single cell of an organism. In turn, imaging genomics (or radiogenomics) is concerned with the correlation between image-based features, as determined by radiomics, and gene expression, as determined by genomics. Imaging genomics arose from decades of work in at least three key areas: 1) sequencing of the human genome, 2) quantitative imaging, computer-aided diagnosis, therapy prognostics, and assessment of therapy response in preclinical and clinical research and practice; and 3) data science, including the burgeoning areas of genomics, preclinical and population-based disease modeling, individualized medicine, and big data. Imaging genomics may answer important questions in medicine by correlating validated, quantitative image phenotypes (through validated imaging biomarkers) with clinical data, histopathologic data, molecular classifications, genomic assays, and treatment outcomes. This approach could address some of the greatest health burdens, including cancer, cardiac disease, and arthritis. Participants will discuss the state-of-the-art of radiomics across multiple disease sites and modalities. Aspects of the presentations will include how to improve the quality of image-based phenotypes of normal and diseased tissue, how to better determine the relationships between these phenotypes and the underlying biology associated with the images, and how to create predictive models using the image-based phenotypes. Topics will also include: 1) creating, archiving, curating and sharing ultra-large datasets (“big data”); 2) standardizing image acquisition and processing methods; 3) standardizing and validating phenotype extraction methods and classifier designs; and 4) using high-throughput, robust, and validated phenotyping systems. H. Aerts, NIH; NCIH. Li, This research was funded in part by the University of Chicago Dean Bridge Fund,and by NCI U24-CA143848-05, P50- CA58223 Breast SPORE program. Hui Li received royalties from Hologic.W. Lu, This work was supported in part by the National Cancer Institute Grants R01CA172638.

MO-FG-207B-00: State-of-the-Art in Radiomics in Radiology and Radiation Oncology

State-of-the-Art in Radiomics in Radiology and Radiation Oncology Radiomics is the science of converting medical images into mineable data, data that are descriptive of “phenotypes,” which may provide diagnostic, prognostic, or therapeutic information. Genomics is the science of sequencing and analyzing the function and structure of genomes; the complete set of DNA in a single cell of an organism. In turn, imaging genomics (or radiogenomics) is concerned with the correlation between image-based features, as determined by radiomics, and gene expression, as determined by genomics. Imaging genomics arose from decades of work in at least three key areas: 1) sequencing of the human genome, 2) quantitative imaging, computer-aided diagnosis, therapy prognostics, and assessment of therapy response in preclinical and clinical research and practice; and 3) data science, including the burgeoning areas of genomics, preclinical and population-based disease modeling, individualized medicine, and big data. Imaging genomics may answer important questions in medicine by correlating validated, quantitative image phenotypes (through validated imaging biomarkers) with clinical data, histopathologic data, molecular classifications, genomic assays, and treatment outcomes. This approach could address some of the greatest health burdens, including cancer, cardiac disease, and arthritis. Participants will discuss the state-of-the-art of radiomics across multiple disease sites and modalities. Aspects of the presentations will include how to improve the quality of image-based phenotypes of normal and diseased tissue, how to better determine the relationships between these phenotypes and the underlying biology associated with the images, and how to create predictive models using the image-based phenotypes. Topics will also include: 1) creating, archiving, curating and sharing ultra-large datasets (“big data”); 2) standardizing image acquisition and processing methods; 3) standardizing and validating phenotype extraction methods and classifier designs; and 4) using high-throughput, robust, and validated phenotyping systems. H. Aerts, NIH; NCIH. Li, This research was funded in part by the University of Chicago Dean Bridge Fund,and by NCI U24-CA143848-05, P50- CA58223 Breast SPORE program. Hui Li received royalties from Hologic.W. Lu, This work was supported in part by the National Cancer Institute Grants R01CA172638.

Ultra-large alignments using phylogeny-aware profiles.

Many biological questions, including the estimation of deep evolutionary histories and the detection of remote homology between protein sequences, rely upon multiple sequence alignments and phylogenetic trees of large datasets. However, accurate large-scale multiple sequence alignment is very difficult, especially when the dataset contains fragmentary sequences. We present UPP, a multiple sequence alignment method that uses a new machine learning technique, the ensemble of hidden Markov models, which we propose here. UPP produces highly accurate alignments for both nucleotide and amino acid sequences, even on ultra-large datasets or datasets containing fragmentary sequences. UPP is available at https://github.com/smirarab/sepp.Electronic supplementary materialThe online version of this article (doi:10.1186/s13059-015-0688-z) contains supplementary material, which is available to authorized users.

Sphere Support Vector Machines for large classification tasks

This paper introduces Sphere Support Vector Machines (SVMs) as the new fast classification algorithm based on combining a minimal enclosing ball approach, state of the art nearest point problem solvers and probabilistic techniques. The blending of the three significantly speeds up the training phase of SVMs and also attains practically the same accuracy as the other classification models over several large real datasets within the strict validation frame of a double (nested) cross-validation. The results shown are promoting SphereSVM as outstanding alternatives for handling large and ultra-large datasets in a reasonable time without switching to various parallelization schemes for SVM algorithms recently proposed.