False Discovery Rate Estimation Research Articles

A simple yet efficient method of local false discovery rate estimation designed for genome-wide association data analysis

A simple yet efficient method of local false discovery rate estimation designed for genome-wide association data analysis

MS Annika: A New Cross-Linking Search Engine.

Cross-linking mass spectrometry (XL-MS) has become a powerful technique that enables insights into protein structures and protein interactions. The development of cleavable cross-linkers has further promoted XL-MS through search space reduction, thereby allowing for proteome-wide studies. These new analysis possibilities foster the development of new cross-linkers, which not every search engine can deal with out of the box. In addition, some search engines for XL-MS data also struggle with the validation of identified cross-linked peptides, that is, false discovery rate (FDR) estimation, as FDR calculation is hampered by the fact that not only one but two peptides in a single spectrum have to be correct. We here present our new search engine, MS Annika, which can identify cross-linked peptides in MS2 spectra from a wide variety of cleavable cross-linkers. We show that MS Annika provides realistic estimates of FDRs without the need of arbitrary score cutoffs, being able to provide on average 44% more identifications at a similar or better true FDR than comparable tools. In addition, MS Annika can be used on proteome-wide studies due to fast, parallelized processing and provides a way to visualize the identified cross-links in protein 3D structures.

Most claimed statistical findings in cross-sectional return predictability are likely true

Harvey, Liu, and Zhu (2016) “argue that most claimed research findings in financial economics are likely false.” Surprisingly, their false discovery rate (FDR) estimates suggest most are true. I revisit their results by developing non- and semi-parametric FDR estimators that account for publication bias and empirical correlations. These estimators provide simple closed-form expressions and reliably produce an upper bound on the FDR in simulations that cluster-bootstrap from empirical predictor returns. Applying these estimators to the Chen-Zimmermann dataset of 205 predictors, I find that most claimed statistical findings in the cross-sectional predictability literature are likely true.

New mixture models for decoy-free false discovery rate estimation in mass spectrometry proteomics.

Accurate estimation of false discovery rate (FDR) of spectral identification is a central problem in mass spectrometry-based proteomics. Over the past two decades, target-decoy approaches (TDAs) and decoy-free approaches (DFAs) have been widely used to estimate FDR. TDAs use a database of decoy species to faithfully model score distributions of incorrect peptide-spectrum matches (PSMs). DFAs, on the other hand, fit two-component mixture models to learn the parameters of correct and incorrect PSM score distributions. While conceptually straightforward, both approaches lead to problems in practice, particularly in experiments that push instrumentation to the limit and generate low fragmentation-efficiency and low signal-to-noise-ratio spectra. We introduce a new decoy-free framework for FDR estimation that generalizes present DFAs while exploiting more search data in a manner similar to TDAs. Our approach relies on multi-component mixtures, in which score distributions corresponding to the correct PSMs, best incorrect PSMs and second-best incorrect PSMs are modeled by the skew normal family. We derive EM algorithms to estimate parameters of these distributions from the scores of best and second-best PSMs associated with each experimental spectrum. We evaluate our models on multiple proteomics datasets and a HeLa cell digest case study consisting of more than a million spectra in total. We provide evidence of improved performance over existing DFAs and improved stability and speed over TDAs without any performance degradation. We propose that the new strategy has the potential to extend beyond peptide identification and reduce the need for TDA on all analytical platforms. https://github.com/shawn-peng/FDR-estimation. Supplementary data are available at Bioinformatics online.

Aberrant posterior cingulate connectivity classify first-episode schizophrenia from controls: A machine learning study

BackgroundPosterior cingulate cortex (PCC) is a key aspect of the default mode network (DMN). Aberrant PCC functional connectivity (FC) is implicated in schizophrenia, but the potential for PCC related changes as biological classifier of schizophrenia has not yet been evaluated. MethodsWe conducted a data-driven approach using resting-state functional MRI data to explore differences in PCC-based region- and voxel-wise FC patterns, to distinguish between patients with first-episode schizophrenia (FES) and demographically matched healthy controls (HC). Discriminative PCC FCs were selected via false discovery rate estimation. A gradient boosting classifier was trained and validated based on 100 FES vs. 93 HC. Subsequently, classification models were tested in an independent dataset of 87 FES patients and 80 HC using resting-state data acquired on a different MRI scanner. ResultsPatients with FES had reduced connectivity between PCC and frontal areas, left parahippocampal regions, left anterior cingulate cortex, and right inferior parietal lobule, but hyperconnectivity with left lateral temporal regions. Predictive voxel-wise clusters were similar to region-wise selected brain areas functionally connected with PCC in relation to discriminating FES from HC subject categories. Region-wise analysis of FCs yielded a relatively high predictive level for schizophrenia, with an average accuracy of 72.28% in the independent samples, while selected voxel-wise connectivity yielded an accuracy of 68.72%. ConclusionFES exhibited a pattern of both increased and decreased PCC-based connectivity, but was related to predominant hypoconnectivity between PCC and brain areas associated with DMN, that may be a useful differential feature revealing underpinnings of neuropathophysiology for schizophrenia.

Enhancing reproducibility of gene expression analysis with known protein functional relationships: The concept of well-associated protein.

Identification of differentially expressed genes (DEGs) is well recognized to be variable across independent replications of genome-wide transcriptional studies. These are often employed to characterize disease state early in the process of discovery and prioritize novel targets aimed at addressing unmet medical need. Increasing reproducibility of biological findings from these studies could potentially positively impact the success rate of new clinical interventions. This work demonstrates that statistically sound combination of gene expression data with prior knowledge about biology in the form of large protein interaction networks can yield quantitatively more reproducible observations from studies characterizing human disease. The novel concept of Well-Associated Proteins (WAPs) introduced herein—gene products significantly associated on protein interaction networks with the differences in transcript levels between control and disease—does not require choosing a differential expression threshold and can be computed efficiently enough to enable false discovery rate estimation via permutation. Reproducibility of WAPs is shown to be on average superior to that of DEGs under easily-quantifiable conditions suggesting that they can yield a significantly more robust description of disease. Enhanced reproducibility of WAPs versus DEGs is first demonstrated with four independent data sets focused on systemic sclerosis. This finding is then validated over thousands of pairs of data sets obtained by random partitions of large studies in several other diseases. Conditions that individual data sets must satisfy to yield robust WAP scores are examined. Reproducible identification of WAPs can potentially benefit drug target selection and precision medicine studies.

Repeat-Preserving Decoy Database for False Discovery Rate Estimation in Peptide Identification.

The sequence database searching method is widely used in proteomics for peptide identification. To control the false discovery rate (FDR) of the searching results, the target-decoy method generates and searches a decoy database together with the target database. A known problem is that the target protein sequence database may contain numerous repeated peptides. The structures of these repeats are not preserved by most existing decoy generation algorithms. Previous studies suggest that such discrepancy between the target and decoy databases may lead to an inaccurate FDR estimation. Based on the de Bruijn graph model, we propose a new repeat-preserving algorithm to generate decoy databases. We prove that this algorithm preserves the structures of the repeats in the target database to a great extent. The de Bruijn method has been compared with a few other commonly used methods and demonstrated superior FDR estimation accuracy and an improved number of peptide identification.

Evaluation of the associations between circulating microRNAs and kidney function in coronary angiography patients.

Circulating microRNAs (miRNAs) have been linked to chronic kidney disease. Little is known about the association between circulating miRNAs and kidney function in patients at high cardiovascular risk. We therefore investigated the association between a panel of candidate miRNAs and kidney function, based on estimated glomerular filtration rate (eGFR), in two independent cohorts of patients undergoing coronary angiography. The present study totally included 438 patients undergoing coronary angiography, who were divided into a discovery cohort (n = 120) and a validation cohort (n = 318). A candidate miRNA panel comprising 50 renal miRNAs was selected from the literature, and expression levels of circulating miRNAs were determined by real-time PCR. Out of the initially tested candidate miRNAs, 38 miRNAs were sufficiently detectable in plasma. Their association with kidney function was evaluated in the discovery cohort. Associations of seven of these miRNAs with eGFR were significant after multiple testing correction via false discovery rate estimation. To verify obtained results, miRNAs with significant false discovery rates were further analyzed in the validation cohort. miR-106b-5p, miR-16-5p, miR-19b-3p, miR-20a-5p, miR-25-3p, and miR-451a proved to be significantly associated with eGFR also in the validation cohort (all P < 0.001). Association between the identified renal miRNAs and kidney function was confirmed by analysis of covariance adjusting for age, sex, type 2 diabetes, hypertension, and albumin-to-creatinine ratio. In conclusion, our study showed that miR-16-5p, miR-19b-3p, miR-20a-5p, miR-25-3p, miR-106b-5p, and miR-451a are significantly linked to kidney function in patients undergoing coronary angiography.

Estimating and Controlling the False Discovery Rate of the PC Algorithm Using Edge-specific P-Values

Many causal discovery algorithms infer graphical structure from observational data. The PC algorithm in particular estimates a completed partially directed acyclic graph (CPDAG), or an acyclic graph containing directed edges identifiable with conditional independence testing. However, few groups have investigated strategies for estimating and controlling the false discovery rate (FDR) of the edges in the CPDAG. In this article, we introduce PC with p-values (PC-p), a fast algorithm that robustly computes edge-specific p-values and then estimates and controls the FDR across the edges. PC-p specifically uses the p-values returned by many conditional independence (CI) tests to upper bound the p-values of more complex edge-specific hypothesis tests. The algorithm then estimates and controls the FDR using the bounded p-values and the Benjamini-Yekutieli FDR procedure. Modifications to the original PC algorithm also help PC-p accurately compute the upper bounds despite non-zero Type II error rates. Experiments show that PC-p yields more accurate FDR estimation and control across the edges in a variety of CPDAGs compared to alternative methods.

Omega-3 Fatty Acids Responsive Proteins and Reduction in Breast Density in Obese Postmenopausal Women.

We reported that breast density (BD) was inversely correlated with the plasma level of DHA in postmenopausal obese, but not in nonobese, women given Lovaza (n-3FA). To identify protein biomarkers for the possible differential effect of n-3FA on BD between obese and nonobese women, an iTRAQ method was performed to analyze plasma from obese and lean women at each time point (baseline, 12 and 24-months, n = 10 per group); 173 proteins with >95% confidence (Unuses Score >1.3 and local false discovery rate estimation <5%) wereidentified. Comparative analysis between various groups identified several differentially expressed proteins (hemopexin precursor, vitamin D binding protein isoform 1 precursor [VDBP], fibronectin isoform 10 precursor [FN], and α-2 macroglobulin precursor [A2M]). Western blot analysis was performed to verify the differential expression of proteins in the iTRAQ study, and those found to be altered in a tumor protective fashion by an n-3FA rich diet in our previous preclinical study; gelsolin, VDBP, and FN were altered by n-3FA in a manner consistent with reduction in inflammation in obese women. To test the impact of our findings on breast cancer risk reduction by n-3FA, a posthoc analysis revealed that n-3FA administration reduced BD selectively in obese postmenopausal women.

Bias in False Discovery Rate Estimation in Mass-Spectrometry-Based Peptide Identification.

Accurate target-decoy-based false discovery rate (FDR) control of peptide identification from tandem mass-spectrometry data relies on an important but often neglected assumption that incorrect spectrum annotations are equally likely to receive either target or decoy peptides. Here we argue that this assumption is often violated in practice, even by popular methods. Preference can be given to target peptides by biased scoring functions, which result in liberal FDR estimations, or to decoy peptides by correlated spectra, which result in conservative estimations.

Accurate Estimation of Context-Dependent False Discovery Rates in Top-Down Proteomics

Within the last several years, top-down proteomics has emerged as a high throughput technique for protein and proteoform identification. This technique has the potential to identify and characterize thousands of proteoforms within a single study, but the absence of accurate false discovery rate (FDR) estimation could hinder the adoption and consistency of top-down proteomics in the future. In automated identification and characterization of proteoforms, FDR calculation strongly depends on the context of the search. The context includes MS data quality, the database being interrogated, the search engine, and the parameters of the search. Particular to top-down proteomics-there are four molecular levels of study: proteoform spectral match (PrSM), protein, isoform, and proteoform. Here, a context-dependent framework for calculating an accurate FDR at each level was designed, implemented, and validated against a manually curated training set with 546 confirmed proteoforms. We examined several search contexts and found that an FDR calculated at the PrSM level under-reported the true FDR at the protein level by an average of 24-fold. We present a new open-source tool, the TDCD_FDR_Calculator, which provides a scalable, context-dependent FDR calculation that can be applied post-search to enhance the quality of results in top-down proteomics from any search engine.

RECAP reveals the true statistical significance of ChIP-seq peak calls.

MotivationChromatin Immunopreciptation (ChIP)-seq is used extensively to identify sites of transcription factor binding or regions of epigenetic modifications to the genome. A key step in ChIP-seq analysis is peak calling, where genomic regions enriched for ChIP versus control reads are identified. Many programs have been designed to solve this task, but nearly all fall into the statistical trap of using the data twice—once to determine candidate enriched regions, and again to assess enrichment by classical statistical hypothesis testing. This double use of the data invalidates the statistical significance assigned to enriched regions, thus the true significance or reliability of peak calls remains unknown.ResultsUsing simulated and real ChIP-seq data, we show that three well-known peak callers, MACS, SICER and diffReps, output biased P-values and false discovery rate estimates that can be many orders of magnitude too optimistic. We propose a wrapper algorithm, RECAP, that uses resampling of ChIP-seq and control data to estimate a monotone transform correcting for biases built into peak calling algorithms. When applied to null hypothesis data, where there is no enrichment between ChIP-seq and control, P-values recalibrated by RECAP are approximately uniformly distributed. On data where there is genuine enrichment, RECAP P-values give a better estimate of the true statistical significance of candidate peaks and better false discovery rate estimates, which correlate better with empirical reproducibility. RECAP is a powerful new tool for assessing the true statistical significance of ChIP-seq peak calls.Availability and implementationThe RECAP software is available through www.perkinslab.ca or on github at https://github.com/theodorejperkins/RECAP.Supplementary information Supplementary data are available at Bioinformatics online.

Pytheas: A Software to Map RNA Modifications via Tandem Mass Spectrometry

Ribonucleic acids (RNA) such as ribosomal RNA (rRNA) and messenger RNA (mRNA) are decorated at the post-transcriptional level by more than 150 characterized chemical modifications. These modifications range from positional isomerization to the addition of complex chemical groups to nucleosides. In the last decades, RNA modifications have been shown to enhance RNAs structural diversity and their intramolecular interactions, as well as promoting RNA diverse functions within complex regulatory networks involved in cellular metabolism. To characterize and quantify RNA modifications, tandem Mass Spectrometry (MS/MS) has been proven to be a powerful and flexible analytical tool. We present here a novel software pipeline, Pytheas, to assist in the interpretation of MS/MS data with the goal to characterize modified RNAs. Through an interactive multi-step process, Pytheas generates a theoretical database from arbitrary RNA sequences and the user-defined parameters such as digestion enzymes, nucleotide chemical compositions and isotopic labeling. Then it uses an empirical scoring function derived from the SEQUEST proteomics platform to score measured MS/MS spectra vs the theoretical library, adopting the target-decoy approach for the false discovery rate (FDR) estimation. The final steps allow visualization of the scored spectra and map the modifications on the RNA sequence. Pytheas is freely available, easy to use and highly customizable; at any step the user can set parameters relevant for their experiments, such as type and number of modifications or instrument-based uncertainty windows on m/z measurements. The applicability of Pytheas has been tested using the Liquid Chromatography MS/MS datasets acquired on different instruments, and was successful in mapping known and discover novel modifications present in tRNAs and B. subtilis 16S and 23S RNA.

Reassessing False Discoveries in Mutual Fund Performance: Skill, Luck, or Lack of Power? A Reply

Andrikogiannopoulou and Papakonstantinou (AP; 2019) conduct an inquiry into the bias of the False Discovery Rate (FDR) estimators of Barras, Scaillet, and Wermers (BSW; 2010). In this Reply, we replicate their results, then further explore the bias issue by (i) using different parameter values, and (ii) updating the sample period. Over the original period (1975-2006), we show how reasonable adjustments to the parameter choices made by BSW and AP results in a sizeable reduction in the bias relative to AP. Over the updated period (1975-2018), we further show that the performance of the FDR improves dramatically across a large range of parameter values. Specifically, we find that the probability of misclassifying a fund with a true alpha of 2% per year is 32% (versus 65% in AP). Our results, in combination with those of AP, indicate that the use of the FDR in finance should be accompanied by a careful evaluation of the underlying data generating process, especially when the sample size is small.

Averaging Strategy To Reduce Variability in Target-Decoy Estimates of False Discovery Rate.

Decoy database search with target-decoy competition (TDC) provides an intuitive, easy-to-implement method for estimating the false discovery rate (FDR) associated with spectrum identifications from shotgun proteomics data. However, the procedure can yield different results for a fixed data set analyzed with different decoy databases, and this decoy-induced variability is particularly problematic for smaller FDR thresholds, data sets, or databases. The average TDC (aTDC) protocol combats this problem by exploiting multiple independently shuffled decoy databases to provide an FDR estimate with reduced variability. We provide a tutorial introduction to aTDC, describe an improved variant of the protocol that offers increased statistical power, and discuss how to deploy aTDC in practice using the Crux software toolkit.

A direct approach to estimating false discovery rates conditional on covariates.

Modern scientific studies from many diverse areas of research abound with multiple hypothesis testing concerns. The false discovery rate (FDR) is one of the most commonly used approaches for measuring and controlling error rates when performing multiple tests. Adaptive FDRs rely on an estimate of the proportion of null hypotheses among all the hypotheses being tested. This proportion is typically estimated once for each collection of hypotheses. Here, we propose a regression framework to estimate the proportion of null hypotheses conditional on observed covariates. This may then be used as a multiplication factor with the Benjamini–Hochberg adjusted p-values, leading to a plug-in FDR estimator. We apply our method to a genome-wise association meta-analysis for body mass index. In our framework, we are able to use the sample sizes for the individual genomic loci and the minor allele frequencies as covariates. We further evaluate our approach via a number of simulation scenarios. We provide an implementation of this novel method for estimating the proportion of null hypotheses in a regression framework as part of the Bioconductor package swfdr.

A new estimation of protein-level false discovery rate

BackgroundIn mass spectrometry-based proteomics, protein identification is an essential task. Evaluating the statistical significance of the protein identification result is critical to the success of proteomics studies. Controlling the false discovery rate (FDR) is the most common method for assuring the overall quality of the set of identifications. Existing FDR estimation methods either rely on specific assumptions or rely on the two-stage calculation process of first estimating the error rates at the peptide-level, and then combining them somehow at the protein-level. We propose to estimate the FDR in a non-parametric way with less assumptions and to avoid the two-stage calculation process.ResultsWe propose a new protein-level FDR estimation framework. The framework contains two major components: the Permutation+BH (Benjamini–Hochberg) FDR estimation method and the logistic regression-based null inference method. In Permutation+BH, the null distribution of a sample is generated by searching data against a large number of permuted random protein database and therefore does not rely on specific assumptions. Then, p-values of proteins are calculated from the null distribution and the BH procedure is applied to the p-values to achieve the relationship of the FDR and the number of protein identifications. The Permutation+BH method generates the null distribution by the permutation method, which is inefficient for online identification. The logistic regression model is proposed to infer the null distribution of a new sample based on existing null distributions obtained from the Permutation+BH method.ConclusionsIn our experiment based on three public available datasets, our Permutation+BH method achieves consistently better performance than MAYU, which is chosen as the benchmark FDR calculation method for this study. The null distribution inference result shows that the logistic regression model achieves a reasonable result both in the shape of the null distribution and the corresponding FDR estimation result.

Empirical Bayes shrinkage and false discovery rate estimation, allowing for unwanted variation.

We combine two important ideas in the analysis of large-scale genomics experiments (e.g. experiments that aim to identify genes that are differentially expressed between two conditions). The first is use of Empirical Bayes (EB) methods to handle the large number of potentially-sparse effects, and estimate false discovery rates and related quantities. The second is use of factor analysis methods to deal with sources of unwanted variation such as batch effects and unmeasured confounders. We describe a simple modular fitting procedure that combines key ideas from both these lines of research. This yields new, powerful EB methods for analyzing genomics experiments that account for both sparse effects and unwanted variation. In realistic simulations, these new methods provide significant gains in power and calibration over competing methods. In real data analysis, we find that different methods, while often conceptually similar, can vary widely in their assessments of statistical significance. This highlights the need for care in both choice of methods and interpretation of results.

ABRF Proteome Informatics Research Group (iPRG) 2016 Study: Inferring Proteoforms from Bottom-up Proteomics Data.

This report presents the results from the 2016 Association of Biomolecular Resource Facilities Proteome Informatics Research Group (iPRG) study on proteoform inference and false discovery rate (FDR) estimation from bottom-up proteomics data. For this study, 3 replicate Q Exactive Orbitrap liquid chromatography-tandom mass spectrometry datasets were generated from each of 4 Escherichia coli samples spiked with different equimolar mixtures of small recombinant proteins selected to mimic pairs of homologous proteins. Participants were given raw data and a sequence file and asked to identify the proteins and provide estimates on the FDR at the proteoform level. As part of this study, we tested a new submission system with a format validator running on a virtual private server (VPS) and allowed methods to be provided as executable R Markdown or IPython Notebooks. The task was perceived as difficult, and only eight unique submissions were received, although those who participated did well with no one method performing best on all samples. However, none of the submissions included a complete Markdown or Notebook, even though examples were provided. Future iPRG studies need to be more successful in promoting and encouraging participation. The VPS and submission validator easily scale to much larger numbers of participants in these types of studies. The unique "ground-truth" dataset for proteoform identification generated for this study is now available to the research community, as are the server-side scripts for validating and managing submissions.