Target-decoy Approach Research Articles

When Target–Decoy False Discovery Rate Estimations Are Inaccurate and How to Spot Instances

To address problems with estimating the reliability of proteomic search engine results from mass spectrometry fragmentation data, the use of target-decoy database searching has become the de facto approach for estimating a false discovery rate. Several articles have been written about the effects of different ways of creating the decoy database, effects of the search engine scoring, or effects of search parameters on whether this approach provides an accurate estimate, not all agreeing with each other's conclusions. Hence, there may be some confusion about how effective this approach is and how broadly it can be applied. Although it is generally very effective, in this article I will try to emphasize some of the pitfalls and dangers of using the target-decoy approach and will indicate tell-tale signs that something may be amiss. This information will hopefully help researchers become more astute in their assessment of search results.

False discovery rates in spectral identification

Automated database search engines are one of the fundamental engines of high-throughput proteomics enabling daily identifications of hundreds of thousands of peptides and proteins from tandem mass (MS/MS) spectrometry data. Nevertheless, this automation also makes it humanly impossible to manually validate the vast lists of resulting identifications from such high-throughput searches. This challenge is usually addressed by using a Target-Decoy Approach (TDA) to impose an empirical False Discovery Rate (FDR) at a pre-determined threshold x% with the expectation that at most x% of the returned identifications would be false positives. But despite the fundamental importance of FDR estimates in ensuring the utility of large lists of identifications, there is surprisingly little consensus on exactly how TDA should be applied to minimize the chances of biased FDR estimates. In fact, since less rigorous TDA/FDR estimates tend to result in more identifications (at higher 'true' FDR), there is often little incentive to enforce strict TDA/FDR procedures in studies where the major metric of success is the size of the list of identifications and there are no follow up studies imposing hard cost constraints on the number of reported false positives.Here we address the problem of the accuracy of TDA estimates of empirical FDR. Using MS/MS spectra from samples where we were able to define a factual FDR estimator of 'true' FDR we evaluate several popular variants of the TDA procedure in a variety of database search contexts. We show that the fraction of false identifications can sometimes be over 10× higher than reported and may be unavoidably high for certain types of searches. In addition, we further report that the two-pass search strategy seems the most promising database search strategy.While unavoidably constrained by the particulars of any specific evaluation dataset, our observations support a series of recommendations towards maximizing the number of resulting identifications while controlling database searches with robust and reproducible TDA estimation of empirical FDR.

Two-Dimensional Target Decoy Strategy for Shotgun Proteomics

The target-decoy approach to estimating and controlling false discovery rate (FDR) has become a de facto standard in shotgun proteomics, and it has been applied at both the peptide-to-spectrum match (PSM) and protein levels. Current bioinformatics methods control either the PSM- or the protein-level FDR, but not both. In order to obtain the most reliable information from their data, users must employ one method when the number of tandem mass spectra exceeds the number of proteins in the database and another method when the reverse is true. Here we propose a simple variation of the standard target-decoy strategy that estimates and controls PSM and protein FDRs simultaneously, regardless of the relative numbers of spectra and proteins. We demonstrate that even if the final goal is a list of PSMs with a fixed low FDR and not a list of protein identifications, the proposed two-dimensional strategy offers advantages over a pure PSM-level strategy.

Proteomic Profiling of Adipose Tissue from Zmpste24−/− Mice, a Model of Lipodystrophy and Premature Aging, Reveals Major Changes in Mitochondrial Function and Vimentin Processing

Lipodystrophy is a major disease involving severe alterations of adipose tissue distribution and metabolism. Mutations in genes encoding the nuclear envelope protein lamin A or its processing enzyme, the metalloproteinase Zmpste24, cause diverse human progeroid syndromes that are commonly characterized by a selective loss of adipose tissue. Similarly to humans, mice deficient in Zmpste24 accumulate prelamin A and display phenotypic features of accelerated aging, including lipodystrophy. Herein, we report the proteome and phosphoproteome of adipose tissue as well as serum metabolome in lipodystrophy by using Zmpste24(-/-) mice as experimental model. We show that Zmpste24 deficiency enhanced lipolysis, fatty acid biogenesis and β-oxidation as well as decreased fatty acid re-esterification, thus pointing to an increased partitioning of fatty acid toward β-oxidation and away from storage that likely underlies the observed size reduction of Zmpste24-null adipocytes. Besides the mitochondrial proteins related to lipid metabolism, other protein networks related to mitochondrial function, including those involved in tricarboxylic acid cycle and oxidative phosphorylation, were up-regulated in Zmpste24(-/-) mice. These results, together with the observation of an increased mitochondrial response to oxidative stress, support the relationship between defective prelamin A processing and mitochondrial dysfunction and highlight the relevance of oxidative damage in lipoatrophy and aging. We also show that absence of Zmpste24 profoundly alters the processing of the cytoskeletal protein vimentin and identify a novel protein dysregulated in lipodystrophy, High-Mobility Group Box-1 Protein. Finally, we found several lipid derivates with important roles in energy balance, such as Lysophosphatidylcholine or 2-arachidonoylglycerol, to be dysregulated in Zmpste24(-/-) serum. Together, our findings in Zmpste24(-/-) mice may be useful to unveil the mechanisms underlying adipose tissue dysfunction and its overall contribution to body homeostasis in progeria and other lipodystrophy syndromes as well as to develop novel strategies to prevent or ameliorate these diseases.

Target-decoy approach and false discovery rate: when things may go wrong.

The target-decoy approach (TDA) has done the field of proteomics a great service by filling in the need to estimate the false discovery rates (FDR) of peptide identifications. While TDA is often viewed as a universal solution to the problem of FDR evaluation, we argue that the time has come to critically re-examine TDA and to acknowledge not only its merits but also its demerits. We demonstrate that some popular MS/MS search tools are not TDA-compliant and that it is easy to develop a non-TDA compliant tool that outperforms all TDA-compliant tools. Since the distinction between TDA-compliant and non-TDA compliant tools remains elusive, we are concerned about a possible proliferation of non-TDA-compliant tools in the future (developed with the best intentions). We are also concerned that estimation of the FDR by TDA awkwardly depends on a virtual coin toss and argue that it is important to take the coin toss factor out of our estimation of the FDR. Since computing FDR via TDA suffers from various restrictions, we argue that TDA is not needed when accurate p-values of individual Peptide-Spectrum Matches are available.

Comment on “Unbiased Statistical Analysis for Multi-Stage Proteomic Search Strategies”

Everett et al. recently reported on a statistical bias that arises in the target-decoy approach to false discovery rate estimation in two-pass proteomics search strategies as exemplified by X!Tandem. This bias can cause serious underestimation of the false discovery rate. We argue here that the "unbiased" solution proposed by Everett et al., however, is also biased and under certain circumstances can also result in a serious underestimate of the FDR, especially at the protein level.

Phosphoproteome analysis of human liver tissue by long‐gradient nanoflow LC coupled with multiple stage MS analysis

Reversible protein phosphorylation plays a critical role in liver development and function. Comprehensively cataloging the phosphoproteins and their phosphorylation sites in human liver tissue will facilitate the understanding of physiological and pathological mechanisms of liver. Owing to lacking of efficient approach to fractionate phosphopeptides, nanoflow-RPLC with long-gradient elution was applied to reduce the complexity of the phosphopeptides in this study. Two approaches were performed to further improve the coverage of phosphoproteome analysis of human liver tissue. In one approach, ten-replicated long-gradient LC-MS/MS runs were performed to analyze the enriched phosphopeptides, which resulted in the localization of 1080 phosphorylation sites from 495 proteins. In another approach, proteins from liver tissue were first fractionated by SDS-PAGE and then long-gradient LC-MS/MS analysis was performed to analyze the phosphopeptides derived from each fraction, which resulted in the localization of 1786 phosphorylation sites from 911 proteins. The two approaches showed the complementation in phosphoproteome analysis of human liver tissue. Combining the results of the two approaches, identification of 2225 nonredundant phosphorylation sites from 1023 proteins was obtained. The confidence of phosphopeptide identifications was strictly controlled with false discovery rate (FDR) < or = 1% by a MS(2)/MS(3) target-decoy database search approach. Among the localized 2225 phosphorylated sites, as many as 70.07% (1559 phosphorylated sites) were also reported by others, which confirmed the high confidence of the sites determined in this study. Considering the data acquired from low accuracy mass spectrometer and processed by a conservative MS(2)/MS(3) target-decoy approach, the number of localized phosphorylation sites obtained for human liver tissue in this study is quite impressive.

P90 Ribosomal S6 Kinase and p70 Ribosomal S6 Kinase Link Phosphorylation of the Eukaryotic Chaperonin Containing TCP-1 to Growth Factor, Insulin, and Nutrient Signaling

Chaperonin containing TCP-1 (CCT) is a large multisubunit complex that mediates protein folding in eukaryotic cells. CCT participates in the folding of newly synthesized polypeptides, including actin, tubulin, and several cell cycle regulators; therefore, CCT plays an important role in cytoskeletal organization and cell division. Here we identify the chaperonin CCT as a novel physiological substrate for p90 ribosomal S6 kinase (RSK) and p70 ribosomal S6 kinase (S6K). RSK phosphorylates the beta subunit of CCT in response to tumor promoters or growth factors that activate the Ras-mitogen-activated protein kinase (MAPK) pathway. CCTbeta Ser-260 was identified as the RSK site by mass spectrometry and confirmed by site-directed mutagenesis. RSK-dependent Ser-260 phosphorylation was sensitive to the MEK inhibitor UO126 and the RSK inhibitor BID-1870. Insulin weakly activates RSK but strongly activates the phosphoinositide 3-kinase (PI3K)-mammalian target of rapamycin (mTOR) pathway and utilizes S6K to regulate CCTbeta phosphorylation. Thus, the Ras-MAPK and PI3K-mTOR pathways converge on CCTbeta Ser-260 phosphorylation in response to multiple agonists in various mammalian cells. We also show that RNA interference-mediated knockdown of endogenous CCTbeta causes impaired cell proliferation that can be rescued with ectopically expressed murine CCTbeta wild-type or phosphomimetic mutant S260D, but not the phosphorylation-deficient mutant S260A. Although the molecular mechanism of CCTbeta regulation remains unclear, our findings demonstrate a link between oncogene and growth factor signaling and chaperonin CCT-mediated cellular activities.

Decoy Methods for Assessing False Positives and False Discovery Rates in Shotgun Proteomics

The potential of getting a significant number of false positives (FPs) in peptide-spectrum matches (PSMs) obtained by proteomic database search has been well-recognized. Among the attempts to assess FPs, the concomitant use of target and decoy databases is widely practiced. By adjusting filtering criteria, FPs and false discovery rate (FDR) can be controlled at a desired level. Although the target-decoy approach is gaining in popularity, subtle differences in decoy construction (e.g., reversing vs stochastic methods), rate calculation (e.g., total vs unique PSMs), or searching (separate vs composite) do exist among various implementations. In the present study, we evaluated the effects of these differences on FP and FDR estimations using a rat kidney protein sample and the SEQUEST search engine as an example. On the effects of decoy construction, we found that, when a single scoring filter (XCorr) was used, stochastic methods generated a higher estimation of FPs and FDR than sequence reversing methods, likely due to an increase in unique peptides. This higher estimation could largely be attenuated by creating decoy databases similar in effective size but not by a simple normalization with a unique-peptide coefficient. When multiple filters were applied, the differences seen between reversing and stochastic methods significantly diminished, suggesting multiple filterings reduce the dependency on how a decoy is constructed. For a fixed set of filtering criteria, FDR and FPs estimated by using unique PSMs were almost twice those using total PSMs. The higher estimation seemed to be dependent on data acquisition setup. As to the differences between performing separate or composite searches, in general, FDR estimated from the separate search was about three times that from the composite search. The degree of difference gradually decreased as the filtering criteria became more stringent. Paradoxically, the estimated true positives in separate search were higher when multiple filters were used. By analyzing a standard protein mixture, we demonstrated that the higher estimation of FDR and FPs in the separate search likely reflected an overestimation, which could be corrected with a simple merging procedure. Our study illustrates the relative merits of different implementations of the target-decoy strategy, which should be worth contemplating when large-scale proteomic biomarker discovery is to be attempted.

Comparative Evaluation of Tandem MS Search Algorithms Using a Target-Decoy Search Strategy

Peptide identification of tandem mass spectra by a variety of available search algorithms forms the foundation for much of modern day mass spectrometry-based proteomics. Despite the critical importance of proper evaluation and interpretation of the results generated by these algorithms there is still little consistency in their application or understanding of their similarities and differences. A survey was conducted of four tandem mass spectrometry peptide identification search algorithms, including Mascot, Open Mass Spectrometry Search Algorithm, Sequest, and X! Tandem. The same input data, search parameters, and sequence library were used for the searches. Comparisons were based on commonly used scoring methodologies for each algorithm and on the results of a target-decoy approach to sequence library searching. The results indicated that there is little difference in the output of the algorithms so long as consistent scoring procedures are applied. The results showed that some commonly used scoring procedures may lead to excessive false discovery rates. Finally an alternative method for the determination of an optimal cutoff threshold is proposed.

A probability-based approach for high-throughput protein phosphorylation analysis and site localization

Data analysis and interpretation remain major logistical challenges when attempting to identify large numbers of protein phosphorylation sites by nanoscale reverse-phase liquid chromatography/tandem mass spectrometry (LC-MS/MS) (Supplementary Figure 1 online). In this report we address challenges that are often only addressable by laborious manual validation, including data set error, data set sensitivity and phosphorylation site localization. We provide a large-scale phosphorylation data set with a measured error rate as determined by the target-decoy approach, we demonstrate an approach to maximize data set sensitivity by efficiently distracting incorrect peptide spectral matches (PSMs), and we present a probability-based score, the Ascore, that measures the probability of correct phosphorylation site localization based on the presence and intensity of site-determining ions in MS/MS spectra. We applied our methods in a fully automated fashion to nocodazole-arrested HeLa cell lysate where we identified 1,761 nonredundant phosphorylation sites from 491 proteins with a peptide false-positive rate of 1.3%.