Multiple Testing Problem Research Articles

Transfer posterior error probability estimation for peptide identification

BackgroundIn shotgun proteomics, database searching of tandem mass spectra results in a great number of peptide-spectrum matches (PSMs), many of which are false positives. Quality control of PSMs is a multiple hypothesis testing problem, and the false discovery rate (FDR) or the posterior error probability (PEP) is the commonly used statistical confidence measure. PEP, also called local FDR, can evaluate the confidence of individual PSMs and thus is more desirable than FDR, which evaluates the global confidence of a collection of PSMs. Estimation of PEP can be achieved by decomposing the null and alternative distributions of PSM scores as long as the given data is sufficient. However, in many proteomic studies, only a group (subset) of PSMs, e.g. those with specific post-translational modifications, are of interest. The group can be very small, making the direct PEP estimation by the group data inaccurate, especially for the high-score area where the score threshold is taken. Using the whole set of PSMs to estimate the group PEP is inappropriate either, because the null and/or alternative distributions of the group can be very different from those of combined scores.ResultsThe transfer PEP algorithm is proposed to more accurately estimate the PEPs of peptide identifications in small groups. Transfer PEP derives the group null distribution through its empirical relationship with the combined null distribution, and estimates the group alternative distribution, as well as the null proportion, using an iterative semi-parametric method. Validated on both simulated data and real proteomic data, transfer PEP showed remarkably higher accuracy than the direct combined and separate PEP estimation methods.ConclusionsWe presented a novel approach to group PEP estimation for small groups and implemented it for the peptide identification problem in proteomics. The methodology of the approach is in principle applicable to the small-group PEP estimation problems in other fields.

Covariate-adjusted Gaussian graphical model estimation with false discovery rate control

Recent genetic/genomic studies have shown that genetic markers can have potential effects on the dependence structure of genes. Motivated by such findings, we are interested in the estimation of covariate-adjusted Gaussian graphical model (CGGM). Most existing approaches depend on regularization techniques, which makes the precise relationship between the regularized parameter and the number of false discovered edges in CGGM estimation ambiguous. In this paper, we formulate CGGM estimation as a multiple testing problem. A new test statistic is introduced and shown to be asymptotic normal null distribution. We then propose a multiple testing procedure for CGGM estimation. The procedure is shown to control the false discovery rate (FDR) at any pre-specified significance level asymptotically. Finally, we provide numerical results to show the performance of our method in both simulation studies and real data analysis.

On the power of some sequential multiple testing procedures

We study an online multiple testing problem where the hypotheses arrive sequentially in a stream. The test statistics are independent and assumed to have the same distribution under their respective null hypotheses. We investigate two recently proposed procedures LORD and LOND, which are proved to control the FDR in an online manner. In some (static) model, we show that LORD is optimal in some asymptotic sense, in particular as powerful as the (static) Benjamini–Hochberg procedure to first asymptotic order. We also quantify the performance of LOND. Some numerical experiments complement our theory.

Biostatistics for Multiple Testing

Multiple testings are instances that contain simultaneous tests for more than one hypothesis. When multiple testings are conducted at the same time, it is more likely that the null hypothesis is rejected, even if the null hypothesis is correct. If individual hypothesis decisions are based on unadjusted <i>p</i>-values, it is usually more likely that some of the true null hypotheses will be rejected. In order to solve the multiple testing problems, various studies have attempted to increase the power by taking into account the family-wise error rate or false discovery rate and statistics required for testing hypotheses. This article discuss methods that account for the multiplicity issue and introduces various statistical techniques.

Frequentist Properties of Bayesian Multiplicity Control for Multiple Testing of Normal Means

We consider the standard problem of multiple testing of normal means, obtaining Bayesian multiplicity control by assuming that the prior inclusion probability (the assumed equal prior probability that each mean is nonzero) is unknown and assigned a prior distribution. The asymptotic frequentist behavior of the Bayesian procedure is studied, as the number of tests grows. Studied quantities include the false positive probability, which is shown to go to zero asymptotically. The asymptotics of a Bayesian decision-theoretic approach are also presented.

Multiple testing correction over contrasts for brain imaging

The multiple testing problem arises not only when there are many voxels or vertices in an image representation of the brain, but also when multiple contrasts of parameter estimates (that represent hypotheses) are tested in the same general linear model. We argue that a correction for this multiplicity must be performed to avoid excess of false positives. Various methods for correction have been proposed in the literature, but few have been applied to brain imaging. Here we discuss and compare different methods to make such correction in different scenarios, showing that one classical and well known method is invalid, and argue that permutation is the best option to perform such correction due to its exactness and flexibility to handle a variety of common imaging situations.

Influence of multiple hypothesis testing on reproducibility in neuroimaging research: A simulation study and Python-based software

BackgroundReproducibility of research findings has been recently questioned in many fields of science, including psychology and neurosciences. One factor influencing reproducibility is the simultaneous testing of multiple hypotheses, which entails false positive findings unless the analyzed p-values are carefully corrected. While this multiple testing problem is well known and studied, it continues to be both a theoretical and practical problem. New methodHere we assess reproducibility in simulated experiments in the context of multiple testing. We consider methods that control either the family-wise error rate (FWER) or false discovery rate (FDR), including techniques based on random field theory (RFT), cluster-mass based permutation testing, and adaptive FDR. Several classical methods are also considered. The performance of these methods is investigated under two different models. ResultsWe found that permutation testing is the most powerful method among the considered approaches to multiple testing, and that grouping hypotheses based on prior knowledge can improve power. We also found that emphasizing primary and follow-up studies equally produced most reproducible outcomes. Comparison with existing method(s)We have extended the use of two-group and separate-classes models for analyzing reproducibility and provide a new open-source software “MultiPy” for multiple hypothesis testing. ConclusionsOur simulations suggest that performing strict corrections for multiple testing is not sufficient to improve reproducibility of neuroimaging experiments. The methods are freely available as a Python toolkit “MultiPy” and we aim this study to help in improving statistical data analysis practices and to assist in conducting power and reproducibility analyses for new experiments.

Comparison of methods for the proportion of true null hypotheses in microarray studie

We consider estimating the proportion of true null hypotheses in multiple testing problems. A traditional multiple testing rate, family-wise error rate is too conservative and old to control type I error in multiple testing setups; however, false discovery rate (FDR) has received significant attention in many research areas such as GWAS data, FMRI data, and signal processing. Identify differentially expressed genes in microarray studies involves estimating the proportion of true null hypotheses in FDR procedures. However, we need to account for unknown dependence structures among genes in microarray data in order to estimate the proportion of true null hypothesis since the genuine dependence structure of microarray data is unknown. We compare various procedures in simulation data and real microarray data. We consider a hidden Markov model for simulated data with dependency. Cai procedure (2007) and a sliding linear model procedure (2011) have a relatively smaller bias and standard errors, being more proper for estimating the proportion of true null hypotheses in simulated data under various setups. Real data analysis shows that 5 estimation procedures among 9 procedures have almost similar values of the estimated proportion of true null hypotheses in microarray data.

A flexible procedure for mixture proportion estimation in positive‐unlabeled learning

AbstractPositive‐unlabeled (PU) learning considers two samples, a positive set P with observations from only one class and an unlabeled set U with observations from two classes. The goal is to classify observations in U. Class mixture proportion estimation (MPE) in U is a key step in PU learning. Blanchard et al. showed that MPE in PU learning is a generalization of the problem of estimating the proportion of true null hypotheses in multiple testing problems. Motivated by this idea, we propose reducing the problem to one‐dimension via construction of a probabilistic classifier trained on the P and U data sets followed by application of a one‐dimensional mixture proportion method from the multiple testing literature to the observation class probabilities. The flexibility of this framework lies in the freedom to choose the classifier and the one‐dimensional MPE method. We prove consistency of two mixture proportion estimators using bounds from empirical process theory, develop tuning parameter free implementations, and demonstrate that they have competitive performance on simulated waveform data and a protein signaling problem.

Controlling the false discovery exceedance for heterogeneous tests

Several classical methods exist for controlling the false discovery exceedance (FDX) for large-scale multiple testing problems, among them the Lehmann-Romano procedure (Lehmann and Romano 2005) ($[\mathrm{LR}]$ below) and the Guo-Romano procedure (Guo and Romano 2007) ($[\mathrm{GR}]$ below). While these two procedures are the most prominent, they were originally designed for homogeneous test statistics, that is, when the null distribution functions of the $p$-values $F_{i}$, $1\leq i\leq m$, are all equal. In many applications, however, the data are heterogeneous which leads to heterogeneous null distribution functions. Ignoring this heterogeneity induces a lack of power. In this paper, we develop three new procedures that incorporate the $F_{i}$’s, while maintaining rigorous FDX control. The heterogeneous version of $[\mathrm{LR}]$, denoted $[\mathrm{HLR}]$, is based on the arithmetic average of the $F_{i}$’s, while the heterogeneous version of $[\mathrm{GR}]$, denoted $[\mathrm{HGR}]$, is based on the geometric average of the $F_{i}$’s. We also introduce a procedure $[\mathrm{PB}]$, that is based on the Poisson-binomial distribution and that uniformly improves $[\mathrm{HLR}]$ and $[\mathrm{HGR}]$, at the price of a higher computational complexity. Perhaps surprisingly, this shows that, contrary to the known theory of false discovery rate (FDR) control under heterogeneity, the way to incorporate the $F_{i}$’s can be particularly simple in the case of FDX control, and does not require any further correction term. The performances of the new proposed procedures are illustrated by real and simulated data in two important heterogeneous settings: first, when the test statistics are continuous but the $p$-values are weighted by some known independent weight vector, e.g., coming from co-data sets; second, when the test statistics are discretely distributed, as is the case for data representing frequencies or counts. Our new procedures are implemented in the R package $\mathtt{FDX}$, see Junge and Döhler (2020).

1167. Dialysis-Related Bloodstream Infections: A Pre- and Post-ClearGuard HD Cap Conception Study

Background25% of patients on hemodialysis receive therapy via a tunneled dialysis catheter (TDC). Bacteremia is one of the most feared complications with an all-cause mortality rate of 12–25%, Medicare spending costs amounting nearly $600 million annually and average hospital length of stay of 12 days per episode. The ClearGuard HD chlorhexidine impregnated catheter (ClearGuard cap) has recently shown potential to significantly reduce rates of bacteremia.MethodsRetrospective review of 150 patients receiving hemodialysis at a single outpatient center in Brooklyn, NY from January 1, 2015 to January 31, 2019. As of February 1, 2019, the ClearGuard cap was implemented for all patients. Poisson regression was used to estimate crude event rate for both pre- and post-intervention periods, with 95% Wald-based confidence intervals (CI). Follow-up for pre-intervention cases terminated at the date the intervention was introduced. A likelihood ratio test of difference in incidence pre vs. post-intervention was conducted. A Kaplan–Meier (K-M) plot of time from the first dialysis since January 2015 to first infection was constructed (pre-intervention cases only). Subsequent subgroup analyses distinguished TDC vs. other patients. The log-rank test was used to test differences between subgroups; Bonferroni corrections to P-values were applied to address the multiple testing problem.ResultsMedian total tracking period (including post-infection follow-up) was 1.75 years (range 0.02–4.26) for pre-intervention cases; 0.19 years (range 0.08–0.21) for post-intervention cases. Event rate was estimated as 9.7 events per 100 person-years (95% CI 6.7, 14.1) for pre-intervention cases; zero (95% CI 0.0, infinity) for post-intervention cases (P = 0.318 for pre- vs. post-comparison) with a clear limitation being lack of power given recent implementation date. We found a statistically significant risk for infection in patients with TDC (P < 0.001).ConclusionIn an underserved, poorly health literate, largely foreign-born, socioeconomically challenged population such as ours, we not only established a significant risk of bacteremia with TDC’s but preliminary post-ClearGuard cap conception data currently being followed is promising for a significant reduction in catheter-related bacteremia. Disclosures All authors: No reported disclosures.

A Sequential Bayesian Partitioning Approach for Online Steady-State Detection of Multivariate Systems

The steady-state detection is critically important in many engineering fields, such as fault detection and diagnosis and process monitoring and control. However, most of the existing methods were designed for univariate signals and, thus, are not effective in handling multivariate signals. In this paper, we propose an efficient online steady-state detection method for multivariate systems through a sequential Bayesian partitioning approach. The signal is modeled by a Bayesian piecewise constant mean and covariance model, and a recursive updating method is developed to calculate the posterior distributions analytically. The duration of the current segment is utilized for steady-state testing. Insightful guidance is also provided for hyperparameter selection. The effectiveness of the proposed method is demonstrated through thorough numerical and real case studies. Note to Practitioners —This paper addresses the problem of online steady-state detection of systems captured by multivariate signals. Existing approaches often monitor each signal independently, and the system is claimed steady when all signals reach steady state. These methods have many shortcomings, such as failing to consider the correlations among signals and suffering the multiple testing problems. In this paper, we propose a novel joint monitoring approach, where the multivariate signal is sequentially partitioned into segments of constant mean and covariance through an online Bayesian inference scheme, and once the current segment duration is sufficiently large, the signal is considered steady. We also provide several insightful guidelines to select appropriate hyperparameters under different scenarios. The proposed approach is much more accurate and robust than existing ones. However, this method may face prohibitive computational cost and ill-posed covariance inversion problem when there are hundreds or even thousands of variables in the system. In future research, we will develop efficient distributed monitoring and data fusion techniques to overcome these challenges.

Reusing Natural Experiments

After a natural experiment is first used, other researchers often reuse the setting, examining different outcome variables. We use simulations based on real data to illustrate the multiple hypothesis testing problem that arises when researchers reuse natural experiments. We then provide guidance for future inference based on popular empirical settings including difference-in-differences regressions, instrumental variables regressions, and regression discontinuity designs. When we apply our guidance to two extensively studied natural experiments, business combination laws and the Regulation SHO pilot, we find that many results that were statistically significant using single hypothesis testing do not survive corrections for multiple hypothesis testing.

Family-Wise Error Rate Controlling Procedures for Discrete Data

In applications such as clinical safety analysis, the data of the experiments usually consist of frequency counts. In the analysis of such data, researchers often face the problem of multiple testing based on discrete test statistics, aimed at controlling family-wise error rate (FWER). Most existing FWER controlling procedures are developed for continuous data, which are often conservative when analyzing discrete data. By using minimal attainable p-values, several FWER controlling procedures have been specifically developed for discrete data in the literature. In this article, by using known marginal distributions of true null p-values, three more powerful stepwise procedures are developed, which are modified versions of the conventional Bonferroni, Holm and Hochberg procedures, respectively. It is shown that the first two procedures strongly control the FWER under arbitrary dependence and are more powerful than the existing Tarone-type procedures, while the last one only ensures control of the FWER in special settings. Through extensive simulation studies, we provide numerical evidence of superior performance of the proposed procedures in terms of the FWER control and minimal power. A real clinical safety data are used to demonstrate applications of our proposed procedures. An R package “MHTdiscrete” and a web application are developed for implementing the proposed procedures.

Should Pearson's correlation coefficient be avoided?

To survey the use of Pearson's correlation coefficient (r) and related statistical methods in the ophthalmic literature, to consider the limitations of r, and to suggest suitable alternative methods of analysis. Searching Ophthalmic and Physiological Optics (OPO), Optometry and Vision Science (OVS), and Clinical and Experimental Optometry (CXO) online archives using correlation and Pearson's r as search terms resulted in 4057 and 281 hits respectively. Coefficient of determination, r square, or r squared received fewer hits (65, 8, and 22 hits respectively). The assumption that r follows a bivariate normal distribution was rarely encountered (3 hits) although several studies applied Spearman's rank correlation (70 hits). The intra-class correlation coefficient (ICC) was widely used (178 hits), but fewer hits were recorded for partial correlation (43 hits) and multiple correlation (13) hits. There was little evidence that the problem of sample size was addressed in correlation studies. Investigators should be alert to whether: (1) the relationship between two variables could be non-linear, (2) the data are bivariate normal, (3) r accounts for a significant proportion of the variance in Y, (4) outliers are present, the data are clustered, or have a restricted range, (5) the sample size is appropriate, and (6) a significant correlation indicates causality. In addition, the number of significant digits used to express r and the problems of multiple testing should be addressed. The problems and limitations of r suggest a more cautious approach regarding its use and the application of alternative methods where appropriate.

Associations Between Immigration-Related User Factors and eHealth Activities for Self-Care: Case of First-Generation Immigrants From Pakistan in the Oslo Area, Norway.

BackgroundImmigrant populations are often disproportionally affected by chronic diseases, such as type 2 diabetes mellitus (T2DM). Use of information and communication technology (ICT) is one promising approach for better self-care of T2DM to mitigate the social health inequalities, if designed for a wider population. However, knowledge is scarce about immigrant populations’ diverse electronic health (eHealth) activities for self-care, especially in European countries.ObjectiveWith a target group of first-generation immigrants from Pakistan in the Oslo area, Norway, we aimed to understand their diverse eHealth activities for T2DM self-care in relation to immigration-related user factors specific to this target group: proficiency in relevant languages (Urdu, Norwegian, English), length of residence in Norway, and diagnosis of T2DM compared with general user factors (age, gender, education and digital skills, and self-rated health status).MethodsData were from a survey among the target population (N=176) conducted in 2015-2016. Using logistic regression, we analyzed associations between user factors and experiences of each of the following eHealth activities for T2DM self-care in the last 12 months: first, information seeking by (1) search engines and (2) Web portals or email subscriptions; second, communication and consultation (1) by closed conversation with a few acquaintances using ICT and (2) on social network services; and third, active decision making by using apps for (1) tracking health information and (2) self-assessment of health status. Using Poisson regression, we also assessed the relationship between user factors and variety of eHealth activities experienced. The Bonferroni correction was used to address the multiple testing problem.ResultsRegression analyses yielded the following significantly positive associations: between Urdu literacy and (1) information seeking by Web portals or email subscriptions (odds ratio [OR] 2.155, 95% CI 1.388-3.344), (2) communication and consultation on social network services (OR 5.697, 95% CI 2.487-13.053), and (3) variety (estimate=0.350, 95% CI 0.148-0.552); between length of residence in Norway and (1) communication and consultation by closed conversation with a few acquaintances using ICT (OR 1.728, 95% CI 1.193-2.503), (2) communication and consultation on social network services (OR 2.098, 95% CI 1.265-3.480), and (3) variety (estimate=0.270, 95% CI 0.117-0.424); between Norwegian language proficiency and active decision making by using apps for self-assessment of health status (OR 2.285, 95% CI 1.294-4.036); between education and digital skills and active decision making by using apps for tracking health information (OR 3.930, 95% CI 1.627-9.492); and between being a female and communication and consultation by closed conversation with a few acquaintances using ICT (OR 2.883, 95% CI 1.335-6.227).ConclusionsThis study implies immigration-related factors may confound associations between general user factors and eHealth activities. Further studies are needed to explore the influence of immigration-related user factors for eHealth activities in other immigrant groups and countries.International Registered ReportRR2-DOI 10.2196/resprot.5468

TEST FOR HIGH DIMENSIONAL CORRELATION MATRICES.

Testing correlation structures has attracted extensive attention in the literature due to both its importance in real applications and several major theoretical challenges. The aim of this paper is to develop a general framework of testing correlation structures for the one-, two-, and multiple sample testing problems under a high-dimensional setting when both the sample size and data dimension go to infinity. Our test statistics are designed to deal with both the dense and sparse alternatives. We systematically investigate the asymptotic null distribution, power function, and unbiasedness of each test statistic. Theoretically, we make great efforts to deal with the non-independency of all random matrices of the sample correlation matrices. We use simulation studies and real data analysis to illustrate the versatility and practicability of our test statistics.

Finding the intended use population for a new treatment

ABSTRACTGenomic tools are demonstrating that many human diseases are molecularly heterogeneous and likely to respond differently to molecularly targeted therapeutics. For many widely used treatments, the number of patients needed to treat (NNT) for each patient who benefits is large indicating that many patients are being exposed to the risks of serious adverse effects although they do not benefit from the drug. Consequently, more accurately determining the intended use population for new therapeutics is of increased importance. In this paper, we describe a new paradigm for identifying and internally validating an estimate of the intended use population in randomized phase III clinical trials. The approach preserves the type I error of the trial and approaches determination of the intended use population as a classification problem, not a multiple hypothesis testing problems.

Meta-Qtest: meta-analysis of quadratic test for rare variants

BackgroundIn genome-wide association studies (GWASs), meta-analysis has been widely used to improve statistical power by combining the results of different studies. Meta-analysis can detect phenotype associated variants that are failed to be detected in single studies. Especially, in biomedical sciences, meta-analysis is often necessary not only for improving statistical power, but also for reducing unavoidable limitation in data collection. As next-generation sequencing (NGS) technology has been developed, meta-analysis of rare variants is proceeding briskly along with meta-analysis of common variants in GWASs. However, meta-analysis on a single variant that is commonly used in common variant association test is improper for rare variants. A sparse signal of rare variant undermines the association signal and its large number causes multiple testing problem. To over-come these problems, we propose a meta-analysis method at the gene-level rather than variant level.ResultsAmong many methods that have been developed, we used the unified quadratic tests (Q-tests); Q-test is more powerful than or as powerful as other tests such as Sequence Kernel Association Tests (SKAT). Since there are three different versions of Q-test (QTest1, QTest2, QTest3), each assumes different relationships among multiple rare variants, we extended them into meta-study accordingly. For meta-analysis, we consider two types of approaches, the one is to combine regression coefficients and the other is to combine test statistics from each single study. We extend the Q-test for meta-analysis, proposing Meta Quadratic Test (Meta-Qtest). Meta Q-test avoids the limitations of MetaSKAT. It does not only consider genetic heterogeneity among studies as MetaSKAT but also reflects diverse real situations; since we extend three different Q-tests into meta-analysis respectively, flexible Meta Q-test suggests way to deal with gene-level rare variant meta-analysis efficiently From the results of real data analysis of blood pressure trait, our meta-analysis could successfully discovered genes, KCNA5 and CABIN1 that are already well known for relevance with hypertension disease and they are not detected in MetaSKAT.ConclusionAs exemplified by an application to T2D Genes projects data set, Meta-Qtest more effectively identified genes associated with hypertension disease than MetaSKAT did.

The correct estimate of the probability of false detection of the matched filter in weak-signal detection problems

The matched filter (MF) represents one of the main tools to detect signals from known sources embedded in the noise. In the Gaussian isotropic case, the noise can be assumed to be the realization of a Gaussian random field (GRF). The most important property of the MF, the maximization of the probability of detection subject to a constant probability of false detection or false alarm (PFA), makes it one of the most popular techniques. However, the MF technique relies upon the a priori knowledge of the number and the position of the searched signals in the GRF (e.g. an emission line in a spectrum or a point-source on a map), which usually are not available. A typical way out is to assume that, if present, the position of a signal coincides with one of the peaks in the matched filtered data. A detection is claimed when the probability that a given peak is due only to the noise (i.e. the PFA) is smaller than a prefixed threshold. This last step represents a critical point in the detection procedure. Since a signal is searched for amongst the peaks, the probability density function (PDF) of the amplitudes of the latter has to be used for the computation of the PFA. Such a PDF, however, is different from the Gaussian. Moreover, the probability that a detection is false depends on the number of peaks present in the filtered GRF. This is because the greater the number of peaks in a GRF, the higher the probability of peaks due to the noise that exceed the detection threshold. If this fact is not taken into account, the PFA can be severely underestimated. In statistics this is a well-known problem named the multiple comparisons, multiple testing, or multiple hypotheses problem, whereas in other fields it is known as the look-elsewhere effect. Many solutions have been proposed to this problem. However, most of them are of a non-parametric type hence not able to exploit all the available information. Recently, this limitation has been overcome by means of two efficient parametric approaches. One is explicitly based on the PDF of the peak amplitudes of a smooth and isotropic GRF whereas the other makes use of the Gumbel distribution, which represents the asymptotic PDF of the corresponding extreme. On the basis of numerical experiments as well of an application to an interferometric map obtained with the Atacama Large Millimeter/submillimeter Array (ALMA), we show that, although the two methods produce almost identical results, the first is more flexible and at the same time allows us to check the reliability of the detection procedure.