Optimal Discovery Procedure Research Articles

Empirical Bayes factors for common hypothesis tests.

Bayes factors for composite hypotheses have difficulty in encoding vague prior knowledge, as improper priors cannot be used and objective priors may be subjectively unreasonable. To address these issues I revisit the posterior Bayes factor, in which the posterior distribution from the data at hand is re-used in the Bayes factor for the same data. I argue that this is biased when calibrated against proper Bayes factors, but propose adjustments to allow interpretation on the same scale. In the important case of a regular normal model, the bias in log scale is half the number of parameters. The resulting empirical Bayes factor is closely related to the widely applicable information criterion. I develop test-based empirical Bayes factors for several standard tests and propose an extension to multiple testing closely related to the optimal discovery procedure. When only a P-value is available, an approximate empirical Bayes factor is 10p. I propose interpreting the strength of Bayes factors on a logarithmic scale with base 3.73, reflecting the sharpest distinction between weaker and stronger belief. This provides an objective framework for interpreting statistical evidence, and realises a Bayesian/frequentist compromise.

Efficient testing and effect size estimation for set-based genetic association inference via semiparametric multilevel mixture modeling.

In genetic association studies, rare variants with extremely low allele frequencies play a crucial role in complex traits. Therefore, set-based testing methods that jointly assess the effects of groups of single nucleotide polymorphisms (SNPs) were developed to increase the powers of the association tests. However, these powers are still insufficient, and precise estimations of the effect sizes of individual SNPs are largely impossible. In this article, we provide an efficient set-based statistical inference framework that addresses both of these important issues simultaneously using an empirical Bayes method with semiparametric multilevel mixture modeling. We propose to utilize the hierarchical model that incorporates variations in set-specific effects and to apply the optimal discovery procedure (ODP) that achieves the largest overall power in multiple significance testing. In addition, we provide an optimal "set-based" estimator of the empirical distribution of effect sizes. The efficiency of the proposed methods is demonstrated through application to a genome-wide association study of coronary artery disease and through simulation studies. The results demonstrated numerous rare variants with large effect sizes for coronary artery disease, and the number of significant sets detected by the ODP was much greater than those identified by existingmethods.

The optimal discovery procedure for significance analysis of general gene expression studies.

MotivationAnalysis of biological data often involves the simultaneous testing of thousands of genes. This requires two key steps: the ranking of genes and the selection of important genes based on a significance threshold. One such testing procedure, called the optimal discovery procedure (ODP), leverages information across different tests to provide an optimal ranking of genes. This approach can lead to substantial improvements in statistical power compared to other methods. However, current applications of the ODP have only been established for simple study designs using microarray technology. Here, we extend this work to the analysis of complex study designs and RNA-sequencing studies.ResultsWe apply our extended framework to a static RNA-sequencing study, a longitudinal study, an independent sampling time-series study,and an independent sampling dose–response study. Our method shows improved performance compared to other testing procedures, finding more differentially expressed genes and increasing power for enrichment analysis. Thus, the extended ODP enables a favorable significance analysis of genome-wide gene expression studies.Availability and implementationThe algorithm is implemented in our freely available R package called edge and can be downloaded at https://www.bioconductor.org/packages/release/bioc/html/edge.html.Supplementary information Supplementary data are available at Bioinformatics online.

Efficient screening of predictive biomarkers for individual treatment selection.

The development of molecular diagnostic tools to achieve individualized medicine requires identifying predictive biomarkers associated with subgroups of individuals who might receive beneficial or harmful effects from different available treatments. However, due to the large number of candidate biomarkers in the large-scale genetic and molecular studies, and complex relationships among clinical outcome, biomarkers, and treatments, the ordinary statistical tests for the interactions between treatments and covariates have difficulties from their limited statistical powers. In this paper, we propose an efficient method for detecting predictive biomarkers. We employ weighted loss functions of Chen etal. to directly estimate individual treatment scores and propose synthetic posterior inference for effect sizes of biomarkers. We develop an empirical Bayes approach, namely, we estimate unknown hyperparameters in the prior distribution based on data. We then provide efficient screening methods for the candidate biomarkers via optimal discovery procedure with adequate control of false discovery rate. The proposed method is demonstrated in simulation studies and an application to a breast cancer clinical study in which the proposed method was shown to detect the much larger numbers of significant biomarkers than existing standardmethods.

Multi-subgroup gene screening using semi-parametric hierarchical mixture models and the optimal discovery procedure: Application to a randomized clinical trial in multiple myeloma.

This article proposes an efficient approach to screening genes associated with a phenotypic variable of interest in genomic studies with subgroups. In order to capture and detect various association profiles across subgroups, we flexibly estimate the underlying effect size distribution across subgroups using a semi-parametric hierarchical mixture model for subgroup-specific summary statistics from independent subgroups. We then perform gene ranking and selection using an optimal discovery procedure based on the fitted model with control of false discovery rate. Efficiency of the proposed approach, compared with that based on standard regression models with covariates representing subgroups, is demonstrated through application to a randomized clinical trial with microarray gene expression data in multiple myeloma, and through a simulation experiment.

Evaluations of the Optimal Discovery Procedure for Multiple Testing.

The Optimal Discovery Procedure (ODP) is a method for simultaneous hypothesis testing that attempts to gain power relative to more standard techniques by exploiting multivariate structure [1]. Specializing to the example of testing whether components of a Gaussian mean vector are zero, we compare the power of the ODP to a Bonferroni-style method and to the Benjamini-Hochberg method when the testing procedures aim to respectively control certain Type I error rate measures, such as the expected number of false positives or the false discovery rate. We show through theoretical results, numerical comparisons, and two microarray examples that when the rejection regions for the ODP test statistics are chosen such that the procedure is guaranteed to uniformly control a Type I error rate measure, the technique is generally less powerful than competing methods. We contrast and explain these results in light of previously proven optimality theory for the ODP. We also compare the ordering given by the ODP test statistics to the standard rankings based on sorting univariate p-values from smallest to largest. In the cases we considered the standard ordering was superior, and ODP rankings were adversely impacted by correlation.

Consistent variable selection via the optimal discovery procedure in multiple testing

ABSTRACTIn this paper, we translate variable selection for linear regression into multiple testing, and select significant variables according to testing result. New variable selection procedures are proposed based on the optimal discovery procedure (ODP) in multiple testing. Due to ODP’s optimality, if we guarantee the number of significant variables included, it will include less non significant variables than marginal p-value based methods. Consistency of our procedures is obtained in theory and simulation. Simulation results suggest that procedures based on multiple testing have improvement over procedures based on selection criteria, and our new procedures have better performance than marginal p-value based procedures.

Microarray evaluation of bovine hepatic gene response to fescue toxicosis

‘Fescue toxicosis’ is a disease in livestock caused by ingestion of ergot alkaloids produced by the fungal endophyteNeotyphodium coenophialum in tall fescue; it is estimated to cost 1 billion USD in damages per year to the beef industry alone. Clinical signs include decreased reproductive fitness, necrosis of extremities, and reduced average daily gain and milk production. Little is known about the cellular mechanisms that mediate these toxic sequelae. We evaluated the effects of ergovaline-based fescue toxicosis on gene expression via oligonucleotide microarray. Liver biopsies were obtained from steers (n=4) pre- and post-exposure (0 and 29 days) to feed containing 579 ng/g ergovaline. Analyses were performed using both ANOVA with false discovery rate correction and Storey's optimal discovery procedure. Overall, down-regulation of gene expression was observed; heart contraction and cardiac development, apoptosis, cell cycle control, and RNA processing genes represented the bulk of differentially expressed transcripts. 2 CYPs (CYP2E1 and CYP4F6) were amongst the significantly upregulated results. Thus, exposure of cattle to toxic levels of ergovaline caused widespread changes in hepatic gene expression, which can both help explain macroscopic clinical signs observed in ruminant animals, and reinforce previous findings in monogastric models.

On Multiple Hypothesis Testing Maximizing the Average Power

A general theory is described for making decisions as to which of two modelled hypotheses (that can depend on unknown parameters) best fits each of a set of data sets, such that the average power is maximized. Statistical independence between the large number of data sets of the same type is assumed. Therefore error rates can be expressed as proportions and the continuous approach to the data model is used. The framework of decision theory is used and the equivalence between different criteria for optimization is demonstrated. General procedures are shown to satisfy this criterion in the cases when each hypothesis has a finite number of unknown parameters, and when the alternative hypothesis is vacuous. If the null hypothesis is determined by a known distribution of a test statistic, this reduces to using the density of $p$-values of this test statistic as the final test statistic to rank the data into significance order. For two scenarios, one of three density estimation methods based on the kernel density estimate gave a result almost equivalent in power to the likelihood ratio test that uses full knowledge of the null and alternative models, and compared favourably with the optimal discovery procedure (ODP) and its iterated variant. For genetic expression data from microarrays and more recently RNA-Seq experiments where the data for different genes are not generally independent, it is suggested to use this technique with the $p$-values from methods such as Surrogate Variable Analysis that removes much of the effects of dependence.

An Empirical Bayes Optimal Discovery Procedure Based on Semiparametric Hierarchical Mixture Models

Multiple testing has been widely adopted for genome-wide studies such as microarray experiments. For effective gene selection in these genome-wide studies, the optimal discovery procedure (ODP), which maximizes the number of expected true positives for each fixed number of expected false positives, was developed as a multiple testing extension of the most powerful test for a single hypothesis by Storey (Journal of the Royal Statistical Society, Series B, vol. 69, no. 3, pp. 347–368, 2007). In this paper, we develop an empirical Bayes method for implementing the ODP based on a semiparametric hierarchical mixture model using the “smoothing-by-roughening" approach. Under the semiparametric hierarchical mixture model, (i) the prior distribution can be modeled flexibly, (ii) the ODP test statistic and the posterior distribution are analytically tractable, and (iii) computations are easy to implement. In addition, we provide a significance rule based on the false discovery rate (FDR) in the empirical Bayes framework. Applications to two clinical studies are presented.

Statistical Analysis of Biomarkers for Personalized Medicine

Recently there has been enhanced and advanced biomedical technology such as high-throughput microarrays and molecular imaging to monitor SNPs, gene and protein expressions, and so forth, to provide exhaustive situations for individuals. In principle we could get much more information to know the biological and medical status from such data sets, which are viewed as biomarkers in a wide sense to help to do identification, association, and prediction studies for phenotypes such as cancer subtypes, prognosis, treatment responsiveness, and adverse reactions for personalized medicine. However, it is frequently difficult to extract only an informative part in the hull of data sets which include a garbage heap of uninformative observations. In particular, we cannot confirm statistical evidence because of the small sample size relative to the dimension of the data when we achieve the practical use of such statistical applications. For example, if we implement machine learning methods for the prediction of the treatment effects, then we typically face difficulties to confirm the reproducibility and robustness for the performance. In the meanwhile, effective study design is crucial for developing and validating biomarkers. In effect we present eight papers challenging this difficult situation of statistical analysis for biomarkers including statistical estimation, prediction, and testing hypothesis. The paper, “The number of candidate variants in exome sequencing for Mendelian Disease under no genetic heterogeneity,” by J. Nishino and S. Mano studies the expected number of candidate single nucleotide variants (SNVs) in exome data for autosomal dominant or recessive Mendelian disorders so that the filtering method is useful with an assumption of “no genetic heterogeneity.” The paper, “Radial basis function-sparse partial least squares for application to brain imaging data,” by H. Yoshida et al. proposes a novel statistical method, called radial basis function sparse partial least squares (RBF-sPLS), for investigating the relationship between clinical characteristics and brain morphology based on three-dimensional MRI data. The paper, “Power analysis of C-TDT for small sample size genome-wide association studies by the joint use of case-parent trios and pairs,” by F. Rajabli et al. addresses family-based genetic association studies with missing genotype information for one of the parents. The paper provides a simulation study for the disequilibrium test statistic across different sample sizes for a family-based genome-wide association study. The paper, “Genomic biomarkers for personalized medicine: development and validation in clinical studies” by S. Matsui addresses the two critical statistical analyses of high-dimensional genomic data, gene screening and prediction, in the framework of development and validation of genomic biomarkers or signatures with biomarker-based clinical trial designs to assess clinical utility of a biomarker or a new treatment with a companion biomarker. The paper, “Multiple suboptimal solutions for prediction rules in gene expression data,” by O. Komori et al. reanalyzes the gene expression data for pattern recognition of phenotypes to extract informative features embedded in the data. The paper concludes that it is not possible to extract only informative genes with high performance in the all observed genes and points out that the mutual coherence among genes is too high to find the solution of informative genes. The paper, “An empirical Bayes optimal discovery procedure based on semiparametric hierarchical mixture models,” by H. Noma and S. Matsui proposes an empirical Bayes optimal discovery procedure with the smoothing-by-roughening approach in semiparametric hierarchical mixture models. This approach provides analytically a tractable posterior distribution with the flexible prior section in genome-wide studies. The paper, “Cancer outlier analysis based on mixture modeling of gene expression data,” by K. Mori et al. proposes a gene-based statistic for gene selection on the basis of a posterior probability of cancer outlier for each cancer sample with an application to real data from hematologic malignancies. The paper, “A robust rerank approach for feature selection and its application to pooling-based GWA studies,” by J. Liu et al. proposes a method based on the concept of rank-over-variable which is shown to be insensitive to the selection of tuning parameters compared with t-statistics, AUC statistics, and SAM with simulation studies and real data analysis of pooling-based genome-wide association (GWA) study to demonstrate the usefulness of their method. Shinto Eguchi Shigeyuki Matsui Su-Yun Huang Chuhsing Kate Hsiao

Investigations into refinements of Storey’s method of multiple hypothesis testing minimising the FDR, and its application to test binomial data

Storey’s method for multiple hypothesis testing “the Optimal Discovery Procedure” (ODP) minimising the false discovery rate (FDR) and giving p-values and q-values (estimates of FDR) for each test, was extended by iteration to enforce consistency between the p-values of the tests and the binary parameters defining which data points contribute to the fitted null hypothesis. These parameters arise when the null hypothesis has to be estimated from the data. The ODP as previously described, is only optimal for fixed values of these parameters. The extension proposed here requires the introduction of a cut-off parameter for the p-values. Motivated by using this method to analyse a set of pairs of frequencies representing gene expression for a set of genes in two libraries, from which it was desired to select those that are most likely to be not following the null hypothesis that the frequency ratio is a fixed unknown number, this method was tested by analysing many similar simulated datasets. The results showed that the ODP modified by iteration could be improved sometimes greatly by a suitable choice of the cut-off parameter, but varying this parameter alone may not lead to the globally optimal solution because statistical testing based on the binomial distribution is more efficient than using a form of the ODP when the number of non-null hypotheses in the data is small, but the reverse is true when it is large. This may be an effect of using discrete data. Efficiency here is defined in terms of the expected proportion of errors that occur (q-value) when a given proportion of the data is declared “significant” (i.e. the null hypothesis is believed not to hold for them). An improved version of the ODP along these lines is likely to have numerous applications such as in the optimised search for candidate genes that show unusual expression patterns for example when more than two experimental conditions are simultaneously compared and to cases when additional categorical variables or a time series is present in the experimental design.

Comparison of methods for identifying differentially expressed genes across multiple conditions from microarray data.

Identification of genes differentially expressed across multiple conditions has become an important statistical problem in analyzing large-scale microarray data. Many statistical methods have been developed to address the challenging problem. Therefore, an extensive comparison among these statistical methods is extremely important for experimental scientists to choose a valid method for their data analysis. In this study, we conducted simulation studies to compare six statistical methods: the Bonferroni (B-) procedure, the Benjamini and Hochberg (BH-) procedure, the Local false discovery rate (Localfdr) method, the Optimal Discovery Procedure (ODP), the Ranking Analysis of F-statistics (RAF), and the Significant Analysis of Microarray data (SAM) in identifying differentially expressed genes. We demonstrated that the strength of treatment effect, the sample size, proportion of differentially expressed genes and variance of gene expression will significantly affect the performance of different methods. The simulated results show that ODP exhibits an extremely high power in indentifying differentially expressed genes, but significantly underestimates the False Discovery Rate (FDR) in all different data scenarios. The SAM has poor performance when the sample size is small, but is among the best-performing methods when the sample size is large. The B-procedure is stringent and thus has a low power in all data scenarios. Localfdr and RAF show comparable statistical behaviors with the BH-procedure with favorable power and conservativeness of FDR estimation. RAF performs the best when proportion of differentially expressed genes is small and treatment effect is weak, but Localfdr is better than RAF when proportion of differentially expressed genes is large.

The optimal discovery procedure in multiple significance testing: an empirical Bayes approach

Multiple testing has been widely adopted for genome-wide studies such as microarray experiments. To improve the power of multiple testing, Storey (J. Royal Statist. Soc. B 2007; 69: 347-368) recently developed the optimal discovery procedure (ODP) which maximizes the number of expected true positives for each fixed number of expected false positives. However, in applying the ODP, we must estimate the true status of each significance test (null or alternative) and the true probability distribution corresponding to each test. In this article, we derive the ODP under hierarchical, random effects models and develop an empirical Bayes estimation method for the derived ODP. Our methods can effectively circumvent the estimation problems in applying the ODP presented by Storey. Simulations and applications to clinical studies of leukemia and breast cancer demonstrated that our empirical Bayes method achieved theoretical optimality and performed well in comparison with existing multiple testing procedures.

Optimal detection of functional connectivity from high-dimensional EEG synchrony data

Computing phase-locking values between EEG signals is a popular method for quantifying functional connectivity. However, this method involves large-scale, high-resolution datasets, which impose a serious multiple testing problem. Standard multiple testing methods fail to exploit the information from the complex dependence structure that varies across hypotheses in spectral, temporal, and spatial dimensions and result in a severe loss of power. They tend to control the false positives at the cost of hiding true positives. We introduce a new approach, called optimal discovery procedure (ODP) for identifying synchrony that is statistically significant. ODP maximizes the number of true positives for a given number of false positives, and thus offers a theoretical optimum for detecting significant synchrony in a multiple testing situation. We demonstrate the utility of this method with PLV data obtained from a visual search study. We also present simulation analysis to confirm the validity and relevance of using ODP in comparison with the standard FDR method for given configurations of true synchrony. We also compare the effectiveness of ODP with our previously published investigation of hierarchical FDR method (Singh and Phillips, 2010).

Application of the Optimal Discovery Procedure to Genetic Case-Control Studies: Comparison with p Values and Asymptotic Bayes Factors

Objectives: The power of genetic association studies is limited by stringent levels of statistical significance. To improve power, Bayes factors (BFs) have been suggested as an alternative measure to the p value, and Storey recently introduced an optimal discovery procedure (ODP) for multiple testing. We aimed to adapt the ODP to genetic case-control studies and to compare its power to p values and asymptotic BFs (ABFs). Methods: We propose estimators of the ODP based on prospective and retrospective likelihoods. We performed simulations based on independent common SNPs and on sequence data including rare variants. Effects of causal SNPs were simulated under various distributions of effect size. Results: The true ODP is never outperformed, but the estimated ODP has similar power to p values and ABFs. For common SNPs the ODP offers power advantages only in extreme scenarios. However, for rare variants the ODP and ABF detect more associations at low false-positive rates than do p values. Conclusions: The ODP can provide higher power than p values for genetic case-control studies of common variants. However, as the ABF has similar power to the ODP and is computed more rapidly, it is our currently preferred method.

A computationally efficient modular optimal discovery procedure

It is well known that patterns of differential gene expression across biological conditions are often shared by many genes, particularly those within functional groups. Taking advantage of these patterns can lead to increased statistical power and biological clarity when testing for differential expression in a microarray experiment. The optimal discovery procedure (ODP), which maximizes the expected number of true positives for each fixed number of expected false positives, is a framework aimed at this goal. Storey et al. introduced an estimator of the ODP for identifying differentially expressed genes. However, their ODP estimator grows quadratically in computational time with respect to the number of genes. Reducing this computational burden is a key step in making the ODP practical for usage in a variety of high-throughput problems. Here, we propose a new estimate of the ODP called the modular ODP (mODP). The existing 'full ODP' requires that the likelihood function for each gene be evaluated according to the parameter estimates for all genes. The mODP assigns genes to modules according to a Kullback-Leibler distance, and then evaluates the statistic only at the module-averaged parameter estimates. We show that the mODP is relatively insensitive to the choice of the number of modules, but dramatically reduces the computational complexity from quadratic to linear in the number of genes. We compare the full ODP algorithm and mODP on simulated data and gene expression data from a recent study of Morrocan Amazighs. The mODP and full ODP algorithm perform very similarly across a range of comparisons. The mODP methodology has been implemented into EDGE, a comprehensive gene expression analysis software package in R, available at http://genomine.org/edge/.

Optimal Tests Shrinking Both Means and Variances Applicable to Microarray Data Analysis

As a consequence of the "large p small n" characteristic for microarray data, hypothesis tests based on individual genes often result in low average power. There are several proposed tests that attempt to improve power. Among these, the FS test that was developed using the concept of James-Stein shrinkage to estimate the variances showed a striking average power improvement. In this paper, we establish a framework in which we model the key parameters with a distribution to find an optimal Bayes test which we call the MAP test (where MAP stands for Maximum Average Power). Under this framework, the FS test can be derived as an empirical Bayes test approximating the MAP test corresponding to modeling the variances. By modeling both the means and the variances with a distribution, a MAP statistic is derived which is optimal in terms of average power but is computationally intensive. An empirical Bayes test called the FSS test is derived as an approximation to the MAP tests and can be computed instantaneously. The FSS statistic shrinks both the means and the variances and has numerically identical average power to the MAP tests. Much numerical evidence is presented in this paper that shows that the proposed test performs uniformly better in average power than the other tests in the literature, including the classical F test, the FS test, the test of Wright and Simon, the moderated t-test, SAM, Efron's t test, the B-statistic and Storey's optimal discovery procedure. A theory is established which indicates that the proposed test is optimal in power when controlling the false discovery rate (FDR).

Differential gene detection incorporating common expression patterns

In detection of differentially expressed (DE) genes between different groups of samples based on a high-throughput expression measurement system, we often use a classical statistical testing based on a simple assumption that the expression of a certain DE gene in one group is higher or lower in average than that in the other group. Based on this simple assumption, the theory of optimal discovery procedure (ODP) (Storey, 2005) provided an optimal thresholding function for DE gene detection. However, expression patterns of DE genes over samples may have such a structure that is not exactly consistent with group labels assigned to the samples. Appropriate treatment of such a structure can increase the detection ability. Namely, genes showing similar expression patterns to other biologically meaningful genes can be regarded as statistically more significant than those showing expression patterns independent of other genes, even if differences in mean expression levels are comparable. In this study, we propose a new statistical thresholding function based on a latent variable model incorporating expression patterns together with the ODP theory. The latent variable model assumes hidden common signals behind expression patterns over samples and the ODP theory is extended to involve the latent variables. When applied to several gene expression data matrices which include cluster structures or 'cancer outlier' structures, the newly-proposed thresholding functions showed prominently better detection performance of DE genes than the original ODP thresholding function did. We also demonstrate how the proposed methods behave through analyses of real breast cancer and lymphoma datasets.

A Bayesian Discovery Procedure

We discuss a Bayesian discovery procedure for multiple comparison problems. We show that under a coherent decision theoretic framework, a loss function combining true positive and false positive counts leads to a decision rule based on a threshold of the posterior probability of the alternative. Under a semi-parametric model for the data, we show that the Bayes rule can be approximated by the optimal discovery procedure (ODP), recently introduced by Storey (2007a). Improving the approximation leads us to a Bayesian discovery procedure (BDP), which exploits the multiple shrinkage in clusters implied by the assumed nonparametric model. We compare the BDP and the ODP estimates in a simple simulation study and in an assessment of differential gene expression based on microarray data from tumor samples. We extend the setting of the ODP by discussing modifications of the loss function that lead to different single thresholding statistics. Finally, we provide an application of the previous arguments to dependent (spatial) data.