Step-up Multiple Test Procedure Research Articles

Outlier Detection Algorithms Over Fuzzy Data with Weighted Least Squares

In the classical leave-one-out procedure for outlier detection in regression analysis, we exclude an observation and then construct a model on the remaining data. If the difference between predicted and observed value is high we declare this value an outlier. As a rule, those procedures utilize single comparison testing. The problem becomes much harder when the observations can be associated with a given degree of membership to an underlying population, and the outlier detection should be generalized to operate over fuzzy data. We present a new approach for outlier detection that operates over fuzzy data using two inter-related algorithms. Due to the way outliers enter the observation sample, they may be of various order of magnitude. To account for this, we divided the outlier detection procedure into cycles. Furthermore, each cycle consists of two phases. In Phase 1, we apply a leave-one-out procedure for each non-outlier in the dataset. In Phase 2, all previously declared outliers are subjected to Benjamini–Hochberg step-up multiple testing procedure controlling the false-discovery rate, and the non-confirmed outliers can return to the dataset. Finally, we construct a regression model over the resulting set of non-outliers. In that way, we ensure that a reliable and high-quality regression model is obtained in Phase 1 because the leave-one-out procedure comparatively easily purges the dubious observations due to the single comparison testing. In the same time, the confirmation of the outlier status in relation to the newly obtained high-quality regression model is much harder due to the multiple testing procedure applied hence only the true outliers remain outside the data sample. The two phases in each cycle are a good trade-off between the desire to construct a high-quality model (i.e., over informative data points) and the desire to use as much data points as possible (thus leaving as much observations as possible in the data sample). The number of cycles is user defined, but the procedures can finalize the analysis in case a cycle with no new outliers is detected. We offer one illustrative example and two other practical case studies (from real-life thrombosis studies) that demonstrate the application and strengths of our algorithms. In the concluding section, we discuss several limitations of our approach and also offer directions for future research.

Fast inference in convolutional neural networks based on sequential three-way decisions

A novel image recognition algorithm based on sequential three-way decisions is introduced to speed up the inference in a convolutional neural network. In contrast to the majority of existing studies, our approach does not require a special procedure to train a neural network, and thus it can be used with arbitrary architectures including pre-trained convolutional nets. Each image is associated with a sequence of features extracted at different layers of the neural network. Features from earlier layers stand for coarse-grained image representation. Fine-grained representations include embeddings from one of later layers. Confidence scores of classifiers representing the input image at each granularity level are computed in order to populate a set of unlikely classes with low confidence scores. The thresholds for these scores are chosen by using the step-up multiple testing procedure. The categories from this set are not considered at the next levels with finer granularity. The algorithm selecting the granularity levels and thresholds for each level is trained on a small sample. An experimental study for several datasets and neural architectures demonstrated that the proposed approach reduces the running time by up to 40% with a controllable decrease in accuracy.

A Flexible Choice of Critical Constants for the Improved Hybrid Hochberg-Hommel Procedure

In Gou et al. (Biometrika 101(4), 899–911 2014), we proposed an improved hybrid Hochberg-Hommel (HH0) step-up multiple test procedure. The procedure used a fixed, specified set of critical constants. In this follow-up note, we show that an entire class of critical constants can be derived by relaxing one of the assumptions in that paper. This allows a choice of more powerful critical constants. These critical constants do not control the type I error under positive dependence among the test statistics as shown by simulations. Therefore, they should be used only under independence and negative dependence when not only they control the type I error but also yield higher power. If positive dependence is expected, as is common in practice, then the HH0 procedure should be used since it is robust with respect to type I error control under dependence.

A class of improved hybrid Hochberg-Hommel type step-up multiple test procedures

In this paper we derive a new p-value based multiple testing procedure that improves upon the Hommel procedure by gaining power as well as having a simpler step-up structure similar to the Hochberg procedure. The key to this improvement is that the Hommel procedure can be improved by a consonant procedure. Exact critical constants of this new procedure can be numerically determined. The zeroth-order approximations to the exact critical constants, albeit slightly conservative, are simple to use and need no tabling, and hence are recommended in practice. The proposed procedure is shown to control the familywise error rate under independence among the p-values. Simulations empirically demonstrate familywise error rate control under positive and negative dependence. Power superiority of the proposed procedure over competing ones is also empirically demonstrated. Illustrative examples are given.

A Step-Up Test Procedure to Find the Minimum Effective Dose

It is of great interest to find the minimum effective dose (MED) in dose-response studies. A sequence of decreasing null hypotheses to find the MED is formulated under the assumption of nondecreasing dose response means. A step-up multiple test procedure that controls the familywise error rate (FWER) is constructed based on the maximum likelihood estimators for the monotone normal means. When the MED is equal to one, the proposed test is uniformly more powerful than Hsu and Berger’s test (1999). Also, a simulation study shows a substantial power improvement for the proposed test over four competitors. Three R-codes are provided in Supplemental Materials for this article. Go to the publishers online edition of Journal of Biopharmaceutical Statistics to view the files.

Adaptive linear step-up multiple testing procedure with the bias-reduced estimator

Abstract This paper suggests two novel adaptive linear step-up procedures to reduce the bias of the estimator of π 0 , the proportion of true null hypotheses. Estimators of π 0 are based on the number of p -values less than a threshold which converges to one.

The Interval Property in Multiple Testing of Pairwise Differences

The usual step-down and step-up multiple testing procedures most often lack an important intuitive, practical, and theoretical property called the interval property. In short, the interval property is simply that for an individual hypothesis, among the several to be tested, the acceptance sections of relevant statistics are intervals. Lack of the interval property is a serious shortcoming. This shortcoming is demonstrated for testing various pairwise comparisons in multinomial models, multivariate normal models and in nonparametric models. Residual based stepwise multiple testing procedures that do have the interval property are offered in all these cases.

Simultaneous confidence regions for closed tests, includingHolm‐,Hochberg‐, andHommel‐related procedures

The derivation of simultaneous confidence regions for some multiple-testing procedures (MTPs) of practical interest has remained an unsolved problem. This is the case, for example, for Hochberg's step-up MTP and Hommel's more powerful MTP that is neither a step-up nor a step-down procedure. It is shown in this article how the direct approach used previously by the author to construct confidence regions for certain closed-testing procedures (CTPs) can be extended to a rather general setup. The general results are then applied to a situation with one-sided inferences and CTPs belonging to a class studied by Wei Liu. This class consists of CTPs based on ordered marginal p-values. It includes Holm's, Hochberg's, and Hommel's MTPs. A property of the confidence regions derived for these three MTPs is that no confidence assertions sharper than rejection assertions can be made unless all null hypotheses are rejected. Briefly, this is related to the fact that these MTPs are quite powerful. The class of CTPs considered includes, however, also MTPs related to Holm's, Hochberg's, and Hommel's MTPs that are less powerful but are such that confidence assertions sharper than rejection assertions are possible even if not all null hypotheses are rejected. One may thus choose and prespecify such an MTP, though this is at the cost of less rejection power.

On the evaluation of the joint distribution of order statistics

Dunnett and Tamhane [Dunnett, C.W., Tamhane, A.C., 1992. A step-up multiple test procedure. J. Amer. Statist. Assoc. 87, 162–170.] proposed a step-up procedure for comparing k treatments with a control and showed that the step-up procedure is more powerful than its counterpart single step and step-down procedures. Since then, several modified step-up procedures have been suggested to deal with different testing environments. In order to establish those step-up procedures, it is necessary to derive approaches for evaluating the joint distribution of the order statistics. In some cases, experimenters may have difficulty in applying those step-up procedures in multiple hypothesis testing because of the computational limitation of existing algorithms in evaluating the critical values for a large number of multiple comparisons. As a result, most procedures are only workable when the design of the experiment is balanced with k ≤ 20 or unbalanced with k ≤ 8 . In this paper, new algorithms are proposed in order to effectively compute the joint distribution of order statistics in various situations. An extensive numerical study shows that the proposed algorithms can easily handle the testing situations with a much larger k . Examples of applying the proposed algorithms to evaluate the critical values of two existing step-up procedures are also presented.

On weighted Hochberg procedures

SUMMARY We consider different ways of constructing weighted Hochberg-type step-up multiple test procedures including closed procedures based on weighted Simes tests and their conservative step-up short-cuts, and step-up counterparts of two weighted Holm procedures. It is shown that the step-up counterparts have some serious pitfalls such as lack of familywise error rate control and lack of monotonicity in rejection decisions in terms of p-values. Therefore an exact closed procedure appears to be the best alternative, its only drawback being lack of simple stepwise structure. A conservative step-up short-cut to the closed procedure may be used instead, but with accompanying loss of power. Simulations are used to study the familywise error rate and power properties of the competing procedures for independent and correlated p-values. Although many of the results of this paper are negative, they are useful in highlighting the need for caution when procedures with similar pitfalls may be used.

A smooth version of the step-up procedure for multiple tests of hypotheses

Cohen and Sackrowitz [2005. Characterization of Bayes procedures for multiple endpoint problems and inadmissibility of the step-up procedure. Ann. Statist. 33, 145–158; 2007. More on the inadmissibility of step-up. J. Multivariate Anal. 97, 481–492] have demonstrated that the popular step-up (SU) multiple testing procedure is inadmissible under a wide variety of conditions. All conditions, however, did assume a permutation invariant (symmetric) model. In this paper we find a necessary condition for admissibility of multiple testing procedures in the asymmetric case. Once again SU does not satisfy the condition and is inadmissible. Since SU has a somewhat less favorable practical property and a less favorable theoretical property, we offer a smooth version of SU which retains the favorable practical properties and avoids some of the less favorable ones. In terms of performance the smooth version and nonsmooth version seem to be comparable at least in low dimensions.

Adaptive linear step-up procedures that control the false discovery rate

The linear step-up multiple testing procedure controls the false discovery rate at the desired level q for independent and positively dependent test statistics. When all null hypotheses are true, and the test statistics are independent and continuous, the bound is sharp. When some of the null hypotheses are not true, the procedure is conservative by a factor which is the proportion m0/m of the true null hypotheses among the hypotheses. We provide a new two-stage procedure in which the linear step-up procedure is used in stage one to estimate m0, providing a new level q′ which is used in the linear step-up procedure in the second stage. We prove that a general form of the two-stage procedure controls the false discovery rate at the desired level q. This framework enables us to study analytically the properties of other procedures that exist in the literature. A simulation study is presented that shows that two-stage adaptive procedures improve in power over the original procedure, mainly because they provide tighter control of the false discovery rate. We further study the performance of the current suggestions, some variations of the procedures, and previous suggestions, in the case where the test statistics are positively dependent, a case for which the original procedure controls the false discovery rate. In the setting studied here the newly proposed two-stage procedure is the only one that controls the false discovery rate. The procedures are illustrated with two examples of biological importance.

Unimprovability of the Bonferroni procedure in the class of general step-up multiple testing procedures

We introduce the class of general step-up multiple testing procedures (step-up MTPs), which contains the usually considered Benjamini–Hochberg type procedures (we call them threshold step-up MTPs) as a parametric subclass. We show that, under the natural condition of monotonicity, the Bonferroni procedure cannot be improved on, while controlling the family-wise error rate (FWER) at the same level, in the class of step-up procedures. This is in contrast to the class of step-down MTPs, where the Holm procedure is a classic example of such an improvement.

More on the inadmissibility of step-up

Cohen and Sackrowitz [Characterization of Bayes procedures for multiple endpoint problems and inadmissibility of the step-up procedure, Ann. Statist. 33 (2005) 145–158] proved that the step-up multiple testing procedure is inadmissible for a multivariate normal model with unknown mean vector and known intraclass covariance matrix. The hypotheses tested are each mean is zero vs. each mean is positive. The risk function is a 2 × 1 vector where one component is average size and the other component is one minus average power. In this paper, we extend the inadmissibility result to several different models, to two-sided alternatives, and to other risk functions. The models include one-parameter exponential families, independent t-variables, independent χ 2 -variables, t-tests arising from the analysis of variance, and t-tests arising from testing treatments against a control. The additional risk functions are linear combinations where one component is the false discovery rate (FDR).

Modified Simes’ critical values under positive dependence

A modification of the critical values of Simes’ test is suggested in this article when the underlying test statistics are multivariate normal with a common non-negative correlation, yielding a more powerful test than the original Simes’ test. A step-up multiple testing procedure with these modified critical values, which is shown to control false discovery rate (FDR), is presented as a modification of the traditional Benjamini–Hochberg (BH) procedure. Simulations were carried out to compare this modified BH procedure with the BH and other modified BH procedures in terms of false non-discovery rate (FNR), 1–FDR–FNR and average power. The present modified BH procedure is observed to perform well compared to others when the test statistics are highly correlated and most of the hypotheses are true.

On optimality of stepdown and stepup multiple test procedures

Consider the multiple testing problem of testing k null hypotheses, where the unknown family of distributions is assumed to satisfy a certain monotonicity assumption. Attention is restricted to procedures that control the familywise error rate in the strong sense and which satisfy a monotonicity condition. Under these assumptions, we prove certain maximin optimality results for some well-known stepdown and stepup procedures.

Solution to Dalal and Mallows conjecture on monotone property of the joint distribution of order statistics

The joint distributions of order statistics play a critical role in constructing some testing procedures e.g. establishing a stopping rule of testing software faults and determining the critical values of step-up multiple test procedures. Despite the well development of those procedures, whether the set of critical values derived by the joint distribution of order statistics has monotone property remains an open question in most procedures. Dalal and Mallows (Ann. Appl. Probab. 2 (1992) 752–765) provided a theorem to show the monotone property in one particular testing procedure and conjectured the property in the other procedure based on an extensive numerical study. In this paper, we would state and prove a more general theorem such that their theorem and the conjecture are just two special cases.

An algorithm for construction of multiple hypothesis testing

Recently, the Simes method for constructing multiple hypothesis tests involving multivariate distributions of the test statistics with a particular form of positive dependence has been proved to strongly control the Type I familywise error rate. In this paper, an algorithm is provided so that distributions of ordered test statistics with certain correlation structures can be exactly and efficiently evaluated. Therefore, in some multiple hypothesis testing we can apply the algorithm to obtain tests which are more powerful than the conservative tests based on the Simes method. An example of how to apply the algorithm to the step-up multiple test procedure with a control treatment is presented.

A note on the monotonicity of the critical values of a step-up test

Abstract The monotonicity of the critical values is essential for the implementation of a step-up multiple test procedure. For a step-up test controlling the familywise error rate exactly at α, while a theoretical justification of the existence of monotone critical values is available in the literature for the case of i.i.d. test statistics, it is still an open problem for the general exchangeable case. Recently, Finner and Roters (1998 Ann. Statist. 26 505–524) gave an example of three exchangeable random variables casting doubt about this existence in general. We provide a positive result in this paper by proving that the critical values of a step-up test involving an equicorrelated trivariate standard normal distribution is indeed monotone when the common correlation is positive and at most (zα/22−z1/42)/(zα/22+z1/42), where zα is the upper 100α percent point of N(0,1). Our example raises the hope of establishing the desired monotonicity property in the case of test statistics with totally positive multivariate distributions in the sense of Karlin and Rinott.

A general approach to stepup multiple test procedures for free-combinations families

Abstract Dunnett and Tamhane (1992, J. Amer. Statist. Assoc. 87, 162–170) proposed a stepup multiple test procedure for simultaneous testing of k hypotheses satisfying the free-combinations condition under the normal setup in the equicorrelated case and later extended it for the non-equicorrelated case (1995, Biometrics, 51, 217–227). However, our counterexample shows that the last one does not control the familywise error rate. We suggest a new approach to constructing stepup procedures which yields as particular cases the procedures by Hochberg (1988, Biometrika, 75, 800–802), Rom (1990, Biometrika, 77, 663–665) and Dunnett and Tamhane (1992) though not by Dunnett and Tamhane (1995). Furthermore, using this approach we develop a stepup test procedure for the non-equicorrelated case that generalizes the Dunnett and Tamhane (1992) procedure but differs from the Dunnett and Tamhane (1995) one. A new test algorithm for the generalized procedure has no break rules and in general makes k tests associated with k nodes on the graph of subset intersection hypotheses whose positions are dictated by the observations. Under the normality assumption, we suggest approximations for sharp critical values and for p -values. The performance of the procedure is illustrated by examples.