Rediscovery Rates Research Articles

Taking Stock: Estimating Vulnerability Rediscovery

How often do multiple, independent parties discover the same vulnerability? There are ample models of vulnerability discovery, but little academic work on this issue of rediscovery. The immature state of this research and subsequent debate is a problem for the policy community, where the government’s decision to disclose a given vulnerability hinges in part on that vulnerability’s likelihood of being rediscovered and used maliciously by another party. Research into the behavior of malicious software markets and the efficacy of bug bounty programs would similarly benefit from an accurate baseline estimate for how often vulnerabilities are discovered by multiple independent parties. This paper presents a new dataset of more than 2,600 vulnerabilities, and estimates vulnerability rediscovery across different vendors and software types. It concludes that rediscovery happens more often than the 1% to 9% range previously reported. The aggregate rediscovery rate for our dataset is 12.7%, ranging between 10.8% for Chrome between 2009 and 2017, to 21.9% for Android between 2016 and 2017. For Android and Chrome, more than 60% of all rediscovery takes place in the first month after the original vulnerability’s disclosure. These results are largely in line with those of the original version of this paper published in July 2017, and indicate that the information security community should map the impact of rediscovery on the efficacy of bug bounty programs, and policymakers should more rigorously evaluate the costs and requirements for non-disclosure of software vulnerabilities.

Comparative mass spectrometry-based metabolomics strategies for the investigation of microbial secondary metabolites.

Covering: 2000 to 2016The labor-intensive process of microbial natural product discovery is contingent upon identifying discrete secondary metabolites of interest within complex biological extracts, which contain inventories of all extractable small molecules produced by an organism or consortium. Historically, compound isolation prioritization has been driven by observed biological activity and/or relative metabolite abundance and followed by dereplication via accurate mass analysis. Decades of discovery using variants of these methods has generated the natural pharmacopeia but also contributes to recent high rediscovery rates. However, genomic sequencing reveals substantial untapped potential in previously mined organisms, and can provide useful prescience of potentially new secondary metabolites that ultimately enables isolation. Recently, advances in comparative metabolomics analyses have been coupled to secondary metabolic predictions to accelerate bioactivity and abundance-independent discovery work flows. In this review we will discuss the various analytical and computational techniques that enable MS-based metabolomic applications to natural product discovery and discuss the future prospects for comparative metabolomics in natural product discovery.

Mining Bacterial Genomes for Secondary Metabolite Gene Clusters.

With the emergence of bacterial resistance against frequently used antibiotics, novel antibacterial compounds are urgently needed. Traditional bioactivity-guided drug discovery strategies involve laborious screening efforts and display high rediscovery rates. With the progress in next generation sequencing methods and the knowledge that the majority of antibiotics in clinical use are produced as secondary metabolites by bacteria, mining bacterial genomes for secondary metabolites with antimicrobial activity is a promising approach, which can guide a more time and cost-effective identification of novel compounds. However, what sounds easy to accomplish, comes with several challenges. To date, several tools for the prediction of secondary metabolite gene clusters are available, some of which are based on the detection of signature genes, while others are searching for specific patterns in gene content or regulation.Apart from the mere identification of gene clusters, several other factors such as determining cluster boundaries and assessing the novelty of the detected cluster are important. For this purpose, comparison of the predicted secondary metabolite genes with different cluster and compound databases is necessary. Furthermore, it is advisable to classify detected clusters into gene cluster families. So far, there is no standardized procedure for genome mining; however, different approaches to overcome all of these challenges exist and are addressed in this chapter. We give practical guidance on the workflow for secondary metabolite gene cluster identification, which includes the determination of gene cluster boundaries, addresses problems occurring with the use of draft genomes, and gives an outlook on the different methods for gene cluster classification. Based on comprehensible examples a protocol is set, which should enable the readers to mine their own genome data for interesting secondary metabolites.

Targeting Reactive Carbonyls for Identifying Natural Products and Their Biosynthetic Origins.

Natural products (NPs) serve important roles as drug candidates and as tools for chemical biology. However, traditional NP discovery, largely based on bioassay-guided approaches, is biased toward abundant compounds and rediscovery rates are high. Orthogonal methods to facilitate discovery of new NPs are thus needed, and herein we describe an isotope tag-based expansion of reactivity-based NP screening to address these shortcomings. Reactivity-based screening is a directed discovery approach in which a specific reactive handle on the NP is targeted by a chemoselective probe to enable its detection by mass spectrometry. In this study, we have developed an aminooxy-containing probe to guide the discovery of aldehyde- and ketone-containing NPs. To facilitate the detection of labeling events, the probe was dibrominated, imparting a unique isotopic signature to distinguish labeled metabolites from spectral noise. As a proof of concept, the probe was then utilized to screen a collection of bacterial extracts, leading to the identification of a new analogue of antipain, deimino-antipain. The bacterial producer of deimino-antipain was sequenced and the responsible biosynthetic gene cluster was identified by bioinformatic analysis and heterologous expression. These data reveal the previously undetermined genetic basis for a well-known family of aldehyde-containing, peptidic protease inhibitors, including antipain, chymostatin, leupeptin, elastatinal, and microbial alkaline protease inhibitor, which have been widely used for over 40 years.

Culture-independent discovery of natural products from soil metagenomes.

Bacterial natural products have proven to be invaluable starting points in the development of many currently used therapeutic agents. Unfortunately, traditional culture-based methods for natural product discovery have been deemphasized by pharmaceutical companies due in large part to high rediscovery rates. Culture-independent, or "metagenomic," methods, which rely on the heterologous expression of DNA extracted directly from environmental samples (eDNA), have the potential to provide access to metabolites encoded by a large fraction of the earth's microbial biosynthetic diversity. As soil is both ubiquitous and rich in bacterial diversity, it is an appealing starting point for culture-independent natural product discovery efforts. This review provides an overview of the history of soil metagenome-driven natural product discovery studies and elaborates on the recent development of new tools for sequence-based, high-throughput profiling of environmental samples used in discovering novel natural product biosynthetic gene clusters. We conclude with several examples of these new tools being employed to facilitate the recovery of novel secondary metabolite encoding gene clusters from soil metagenomes and the subsequent heterologous expression of these clusters to produce bioactive small molecules.

Nonparametric estimation of the rediscovery rate.

Validation studies have been used to increase the reliability of the statistical conclusions for scientific discoveries; such studies improve the reproducibility of the findings and reduce the possibility of false positives. Here, one of the important roles of statistics is to quantify reproducibility rigorously. Two concepts were recently defined for this purpose: (i) rediscovery rate (RDR), which is the expected proportion of statistically significant findings in a study that can be replicated in the validation study and (ii) false discovery rate in the validation study (vFDR). In this paper, we aim to develop a nonparametric approach to estimate the RDR and vFDR and show an explicit link between the RDR and the FDR. Among other things, the link explains why reproducing statistically significant results even with low FDR level may be difficult. Two metabolomics datasets are considered to illustrate the application of the RDR and vFDR concepts in high-throughput data analysis. Copyright © 2016 John Wiley & Sons, Ltd.

Genomes to natural products PRediction Informatics for Secondary Metabolomes (PRISM)

Microbial natural products are an invaluable source of evolved bioactive small molecules and pharmaceutical agents. Next-generation and metagenomic sequencing indicates untapped genomic potential, yet high rediscovery rates of known metabolites increasingly frustrate conventional natural product screening programs. New methods to connect biosynthetic gene clusters to novel chemical scaffolds are therefore critical to enable the targeted discovery of genetically encoded natural products. Here, we present PRISM, a computational resource for the identification of biosynthetic gene clusters, prediction of genetically encoded nonribosomal peptides and type I and II polyketides, and bio- and cheminformatic dereplication of known natural products. PRISM implements novel algorithms which render it uniquely capable of predicting type II polyketides, deoxygenated sugars, and starter units, making it a comprehensive genome-guided chemical structure prediction engine. A library of 57 tailoring reactions is leveraged for combinatorial scaffold library generation when multiple potential substrates are consistent with biosynthetic logic. We compare the accuracy of PRISM to existing genomic analysis platforms. PRISM is an open-source, user-friendly web application available at http://magarveylab.ca/prism/.

Integration of high-content screening and untargeted metabolomics for comprehensive functional annotation of natural product libraries

Traditional natural products discovery using a combination of live/dead screening followed by iterative bioassay-guided fractionation affords no information about compound structure or mode of action until late in the discovery process. This leads to high rates of rediscovery and low probabilities of finding compounds with unique biological and/or chemical properties. By integrating image-based phenotypic screening in HeLa cells with high-resolution untargeted metabolomics analysis, we have developed a new platform, termed Compound Activity Mapping, that is capable of directly predicting the identities and modes of action of bioactive constituents for any complex natural product extract library. This new tool can be used to rapidly identify novel bioactive constituents and provide predictions of compound modes of action directly from primary screening data. This approach inverts the natural products discovery process from the existing "grind and find" model to a targeted, hypothesis-driven discovery model where the chemical features and biological function of bioactive metabolites are known early in the screening workflow, and lead compounds can be rationally selected based on biological and/or chemical novelty. We demonstrate the utility of the Compound Activity Mapping platform by combining 10,977 mass spectral features and 58,032 biological measurements from a library of 234 natural products extracts and integrating these two datasets to identify 13 clusters of fractions containing 11 known compound families and four new compounds. Using Compound Activity Mapping we discovered the quinocinnolinomycins, a new family of natural products with a unique carbon skeleton that cause endoplasmic reticulum stress.

Structure, Bioactivity, and Resistance Mechanism of Streptomonomicin, an Unusual Lasso Peptide from an Understudied Halophilic Actinomycete

Natural products are the most historically significant source of compounds for drug development. However, unacceptably high rates of compound rediscovery associated with large-scale screening of common microbial producers have resulted in the abandonment of many natural product drug discovery efforts, despite the increasing prevalence of clinically problematic antibiotic resistance. Screening of underexplored taxa represents one strategy to avoid rediscovery. Herein we report the discovery, isolation, and structural elucidation of streptomonomicin (STM), an antibiotic lasso peptide from Streptomonospora alba, and report the genome for its producing organism. STM-resistant clones of Bacillus anthracis harbor mutations to walR, the gene encoding a response regulator for the only known widely distributed and essential two-component signal transduction system in Firmicutes. To the best of our knowledge, Streptomonospora had been hitherto biosynthetically and genetically uncharacterized, with STM being the first reported compound from the genus. Our results demonstrate that understudied microbes remain fruitful reservoirs forthe rapid discovery of novel, bioactive natural products.

Consensus Genotyper for Exome Sequencing (CGES): improving the quality of exome variant genotypes.

The development of cost-effective next-generation sequencing methods has spurred the development of high-throughput bioinformatics tools for detection of sequence variation. With many disparate variant-calling algorithms available, investigators must ask, 'Which method is best for my data?' Machine learning research has shown that so-called ensemble methods that combine the output of multiple models can dramatically improve classifier performance. Here we describe a novel variant-calling approach based on an ensemble of variant-calling algorithms, which we term the Consensus Genotyper for Exome Sequencing (CGES). CGES uses a two-stage voting scheme among four algorithm implementations. While our ensemble method can accept variants generated by any variant-calling algorithm, we used GATK2.8, SAMtools, FreeBayes and Atlas-SNP2 in building CGES because of their performance, widespread adoption and diverse but complementary algorithms. We apply CGES to 132 samples sequenced at the Hudson Alpha Institute for Biotechnology (HAIB, Huntsville, AL) using the Nimblegen Exome Capture and Illumina sequencing technology. Our sample set consisted of 40 complete trios, two families of four, one parent-child duo and two unrelated individuals. CGES yielded the fewest total variant calls (N(CGES) = 139° 897), the highest Ts/Tv ratio (3.02), the lowest Mendelian error rate across all genotypes (0.028%), the highest rediscovery rate from the Exome Variant Server (EVS; 89.3%) and 1000 Genomes (1KG; 84.1%) and the highest positive predictive value (PPV; 96.1%) for a random sample of previously validated de novo variants. We describe these and other quality control (QC) metrics from consensus data and explain how the CGES pipeline can be used to generate call sets of varying quality stringency, including consensus calls present across all four algorithms, calls that are consistent across any three out of four algorithms, calls that are consistent across any two out of four algorithms or a more liberal set of all calls made by any algorithm. To enable accessible, efficient and reproducible analysis, we implement CGES both as a stand-alone command line tool available for download in GitHub and as a set of Galaxy tools and workflows configured to execute on parallel computers. Supplementary data are available at Bioinformatics online.

Rediscovery rate estimation for assessing the validation of significant findings in high-throughput studies.

It is common and advised practice in biomedical research to validate experimental or observational findings in a population different from the one where the findings were initially assessed. This practice increases the generalizability of the results and decreases the likelihood of reporting false-positive findings. Validation becomes critical when dealing with high-throughput experiments, where the large number of tests increases the chance to observe false-positive results. In this article, we review common approaches to determine statistical thresholds for validation and describe the factors influencing the proportion of significant findings from a 'training' sample that are replicated in a 'validation' sample. We refer to this proportion as rediscovery rate (RDR). In high-throughput studies, the RDR is a function of false-positive rate and power in both the training and validation samples. We illustrate the application of the RDR using simulated data and real data examples from metabolomics experiments. We further describe an online tool to calculate the RDR using t-statistics. We foresee two main applications. First, if the validation study has not yet been collected, the RDR can be used to decide the optimal combination between the proportion of findings taken to validation and the size of the validation study. Secondly, if a validation study has already been done, the RDR estimated using the training data can be compared with the observed RDR from the validation data; hence, the success of the validation study can be assessed.

Connecting phenotype and chemotype: Development of an integrated platform for the direct prediction of natural product modes of action

Despite the widespread use of natural products as inspirations for existing drugs, the rate of development of new natural product-based drugs has slowed in recent years. This is due in part to an increase in the rates of rediscovery of existing scaffolds, as well as limited early characterization of biological functions of hits from primary screens. However, recent whole genome sequencing efforts are revealing a wealth of untapped biosynthetic diversity in environmental microorganisms, suggesting that there exists a vast reservoir of potential new drug leads from natural sources.

Nucleophilic 1,4-additions for natural product discovery.

Natural products remain an important source of drug candidates, but the difficulties inherent to traditional isolation, coupled with unacceptably high rates of compound rediscovery, limit the pace of natural product detection. Here we describe a reactivity-based screening method to rapidly identify exported bacterial metabolites that contain dehydrated amino acids (i.e., carbonyl- or imine-activated alkenes), a common motif in several classes of natural products. Our strategy entails the use of a commercially available thiol, dithiothreitol, for the covalent labeling of activated alkenes by nucleophilic 1,4-addition. Modification is easily discerned by comparing mass spectra of reacted and unreacted cell surface extracts. When combined with bioinformatic analysis of putative natural product gene clusters, targeted screening and isolation can be performed on a prioritized list of strains. Moreover, known compounds are easily dereplicated, effectively eliminating superfluous isolation and characterization. As a proof of principle, this labeling method was used to identify known natural products belonging to the thiopeptide, lanthipeptide, and linaridin classes. Further, upon screening a panel of only 23 actinomycetes, we discovered and characterized a novel thiopeptide antibiotic, cyclothiazomycin C.

Discovery of novel enzymes with industrial potential from a cold and alkaline environment by a combination of functional metagenomics and culturing.

BackgroundThe use of cold-active enzymes has many advantages, including reduced energy consumption and easy inactivation. The ikaite columns of SW Greenland are permanently cold (4-6°C) and alkaline (above pH 10), and the microorganisms living there and their enzymes are adapted to these conditions. Since only a small fraction of the total microbial diversity can be cultured in the laboratory, a combined approach involving functional screening of a strain collection and a metagenomic library was undertaken for discovery of novel enzymes from the ikaite columns.ResultsA strain collection with 322 cultured isolates was screened for enzymatic activities identifying a large number of enzyme producers, with a high re-discovery rate to previously characterized strains. A functional expression library established in Escherichia coli identified a number of novel cold-active enzymes. Both α-amylases and β-galactosidases were characterized in more detail with respect to temperature and pH profiles and one of the β-galactosidases, BGalI17E2, was able to hydrolyze lactose at 5°C. A metagenome sequence of the expression library indicated that the majority of enzymatic activities were not detected by functional expression. Phylogenetic analysis showed that different bacterial communities were targeted with the culture dependent and independent approaches and revealed the bias of multiple displacement amplification (MDA) of DNA isolated from complex microbial communities.ConclusionsMany cold- and/or alkaline-active enzymes of industrial relevance were identified in the culture based approach and the majority of the enzyme-producing isolates were closely related to previously characterized strains. The function-based metagenomic approach, on the other hand, identified several enzymes (β-galactosidases, α-amylases and a phosphatase) with low homology to known sequences that were easily expressed in the production host E. coli. The β-galactosidase BGalI17E2 was able to hydrolyze lactose at low temperature, suggesting a possibly use in the dairy industry for this enzyme. The two different approaches complemented each other by targeting different microbial communities, highlighting the usefulness of combining methods for bioprospecting. Finally, we document here that ikaite columns constitute an important source of cold- and/or alkaline-active enzymes with industrial application potential.

Target enrichment using parallel nanoliter quantitative PCR amplification.

BackgroundNext generation targeted resequencing is replacing Sanger sequencing at high pace in routine genetic diagnosis. The need for well validated, high quality enrichment platforms to complement the bench-top next generation sequencing devices is high.ResultsWe used the WaferGen Smartchip platform to perform highly parallelized PCR based target enrichment for a set of known cancer genes in a well characterized set of cancer cell lines from the NCI60 panel. Optimization of PCR assay design and cycling conditions resulted in a high enrichment efficiency. We provide proof of a high mutation rediscovery rate and have included technical replicates to enable SNP calling validation demonstrating the high reproducibility of our enrichment platform.ConclusionsHere we present our custom developed quantitative PCR based target enrichment platform. Using highly parallel nanoliter singleplex PCR reactions makes this a flexible and efficient platform. The high mutation validation rate shows this platform’s promise as a targeted resequencing method for multi-gene routine sequencing diagnostics.

Moved but not gone: an evaluation of real-time methods for discovering replacement web pages

Inaccessible Web pages and 404 “Page Not Found” responses are a common Web phenomenon and a detriment to the user’s browsing experience. The rediscovery of missing Web pages is, therefore, a relevant research topic in the digital preservation as well as in the Information Retrieval realm. In this article, we bring these two areas together by analyzing four content- and link-based methods to rediscover missing Web pages. We investigate the retrieval performance of the methods individually as well as their combinations and give an insight into how effective these methods are over time. As the main result of this work, we are able to recommend not only the best performing methods but also the sequence in which they should be applied, based on their performance, complexity required to generate them, and evolution over time. Our least complex single method results in a rediscovery rate of almost \(70\,\%\) of Web pages of our sample dataset based on URIs sampled from the Open Directory Project (DMOZ). By increasing the complexity level and combining three different methods, our results show an increase of the success rate of up to \(77\,\%\). The results, based on our sample dataset, indicate that Web pages are often not completely lost but have moved to a different location and “just” need to be rediscovered.

Variant Callers for Next-Generation Sequencing Data: A Comparison Study

Next generation sequencing (NGS) has been leading the genetic study of human disease into an era of unprecedented productivity. Many bioinformatics pipelines have been developed to call variants from NGS data. The performance of these pipelines depends crucially on the variant caller used and on the calling strategies implemented. We studied the performance of four prevailing callers, SAMtools, GATK, glftools and Atlas2, using single-sample and multiple-sample variant-calling strategies. Using the same aligner, BWA, we built four single-sample and three multiple-sample calling pipelines and applied the pipelines to whole exome sequencing data taken from 20 individuals. We obtained genotypes generated by Illumina Infinium HumanExome v1.1 Beadchip for validation analysis and then used Sanger sequencing as a “gold-standard” method to resolve discrepancies for selected regions of high discordance. Finally, we compared the sensitivity of three of the single-sample calling pipelines using known simulated whole genome sequence data as a gold standard. Overall, for single-sample calling, the called variants were highly consistent across callers and the pairwise overlapping rate was about 0.9. Compared with other callers, GATK had the highest rediscovery rate (0.9969) and specificity (0.99996), and the Ti/Tv ratio out of GATK was closest to the expected value of 3.02. Multiple-sample calling increased the sensitivity. Results from the simulated data suggested that GATK outperformed SAMtools and glfSingle in sensitivity, especially for low coverage data. Further, for the selected discrepant regions evaluated by Sanger sequencing, variant genotypes called by exome sequencing versus the exome array were more accurate, although the average variant sensitivity and overall genotype consistency rate were as high as 95.87% and 99.82%, respectively. In conclusion, GATK showed several advantages over other variant callers for general purpose NGS analyses. The GATK pipelines we developed perform very well.

Culturable rare Actinomycetes: diversity, isolation and marine natural product discovery

Rare Actinomycetes from underexplored marine environments are targeted in drug discovery studies due to the Actinomycetes' potentially huge resource of structurally diverse natural products with unusual biological activity. Of all marine bacteria, 10 % are Actinomycetes, which have proven an outstanding and fascinating resource for new and potent bioactive molecules. Past and present efforts in the isolation of rare Actinomycetes from underexplored diverse natural habitats have resulted in the isolation of about 220 rare Actinomycete genera of which more than 50 taxa have been reported to be the producers of 2,500 bioactive compounds. That amount represents greater than 25 % of the total Actinomycetes metabolites, demonstrating that selective isolation methods are being developed and extensively applied. Due to the high rediscovery rate of known compounds from Actinomycetes, a renewed interest in the development of new antimicrobial agents from rare and novel Actinomycetes is urgently required to combat the increasing number of multidrug-resistant human pathogens. To facilitate that discovery, this review updates all selective isolation media including pretreatment and enrichment methods for the isolation of marine rare Actinomycetes. In addition, this review demonstrates that discovering new compounds with novel scaffolds can be increased by intensive efforts in isolating and screening rare marine genera of Actinomycetes. Between 2007 and mid-2013, 80 new rare Actinomycete species were reported from marine habitats. They belong to 23 rare families, of which three are novel, and 20 novel genera. Of them, the family Micromonosporaceae is dominant as a producer of promising chemical diversity.

Antibiotics from neglected bacterial sources

The current crop of antibiotics in clinical use are either natural products or their derivatives. However, the rise of a multitude of different antibiotic resistant human pathogens has meant that new antibiotics are urgently needed. Unfortunately, the search for new antibiotics from traditional bacterial sources often results in a high rediscovery rate of known compounds and a low chance of identifying truly novel chemical entities. To overcome this, previously unexplored (or under investigated) bacterial sources are being tapped for their potential to produce novel compounds with new activities. Here, we review a number of antibiotic compounds identified from bacteria of the genera Burkholderia, Clostridium, Lysobacter, Pantoea and Xenorhabdus and describe the potential of organisms and their associated metabolites in future drug discovery efforts.

Bacterial symbionts in agricultural systems provide a strategic source for antibiotic discovery

As increased antibiotic resistance erodes the efficacy of currently used drugs, the need for new candidates with therapeutic potential grows. Although the majority of antibiotics in clinical use originated from natural products, mostly from environmental actinomycetes, high rediscovery rates, among other factors, have diminished the enthusiasm for continued exploration of this historically important source. Several well-studied insect agricultural systems have bacterial symbionts that have evolved to produce small molecules that suppress environmental pathogens. These molecules represent an underexplored reservoir of potentially useful antibiotics. This report describes the multilateral symbioses common to insect agricultural systems, the general strategy used for antibiotic discovery and pertinent examples from three farming systems: fungus-farming ants, southern pine beetles (SPBs) and fungus-growing termites.