Abstract
Metagenomic datasets contain billions of protein sequences that could greatly enhance large-scale functional annotation and structure prediction. Utilizing this enormous resource would require reducing its redundancy by similarity clustering. However, clustering hundreds of millions of sequences is impractical using current algorithms because their runtimes scale as the input set size N times the number of clusters K, which is typically of similar order as N, resulting in runtimes that increase almost quadratically with N. We developed Linclust, the first clustering algorithm whose runtime scales as N, independent of K. It can also cluster datasets several times larger than the available main memory. We cluster 1.6 billion metagenomic sequence fragments in 10 h on a single server to 50% sequence identity, >1000 times faster than has been possible before. Linclust will help to unlock the great wealth contained in metagenomic and genomic sequence databases.
Highlights
Metagenomic datasets contain billions of protein sequences that could greatly enhance largescale functional annotation and structure prediction
If we denote the average probability by pmatch that this happens by chance between two non-homologous input sequences, the prefilter would speed up the sequence comparison by a factor of up to 1/pmatch at the expense of some loss in sensitivity
CD-HIT and UCLUST employ the following "greedy incremental clustering" approach: each of the N input sequences is compared with the representative sequences of already established clusters
Summary
Metagenomic datasets contain billions of protein sequences that could greatly enhance largescale functional annotation and structure prediction Utilizing this enormous resource would require reducing its redundancy by similarity clustering. Costs and throughput of next-generation sequencing have dropped two-fold each year, twice faster than computational costs This enormous progress has resulted in hundreds of thousands of metagenomes and tens of billions of putative gene and protein sequences[2,3]. The fast sequence prefilters speed up each pairwise comparison by a large factor 1/pmatch but cannot improve the time complexity of O(NK) This almost quadratic scaling results in impractical runtimes for a billion or more sequences
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
