Step-down Procedure Research Articles

P-Value combiners for graphical modelling of EEG data in the frequency domain

BackgroundIn the graphical modelling of brain data, we are interested in estimating connectivity between various regions of interest, and evaluating statistical significance in order to derive a network model. This process involves aggregating results across frequency ranges and several patients, in order to obtain an overall result that can serve to construct a graph. New methodIn this paper, we propose a method based on p-value combiners, which have never been used in applications to EEG data analysis. This new method is split into two aspects: frequency-wide tests and group-wide tests. The first step can be effectively adjusted to control for false detection rate. ResultsThis two-step protocol is applied to EEG data collected from distinct groups of mental health patients, in order to draw graphical models for each group and highlight structural connectivity differences. Using the method proposed, we show that it is possible to reliably achieve this while effectively controlling for false connections detection. Comparison with existing method(s)Conventionally, the Holm's Stepdown procedure is used for this type of problem, as it is robust to type I errors. However, it is known to be conservative and prone to false negatives. Furthermore, unlike the proposed methods, it does not directly output a decision rule on whether to accept or reject a statement. ConclusionsThe proposed methodology offers significant improvements over the stepdown procedure in terms of error rate and false negative rate across the network models, as well as in term of applicability.

QuickMMCTest: quick multiple Monte Carlo testing

Multiple hypothesis testing is widely used to evaluate scientific studies involving statistical tests. However, for many of these tests, p-values are not available and are thus often approximated using Monte Carlo tests such as permutation tests or bootstrap tests. This article presents a simple algorithm based on Thompson Sampling to test multiple hypotheses. It works with arbitrary multiple testing procedures, in particular with step-up and step-down procedures. Its main feature is to sequentially allocate Monte Carlo effort, generating more Monte Carlo samples for tests whose decisions are so far less certain. A simulation study demonstrates that for a low computational effort, the new approach yields a higher power and a higher degree of reproducibility of its results than previously suggested methods.

Time scale and fractionality in financial time series

Purpose– Turvey (2007, Physica A) introduced a scaled variance ratio procedure for testing the random walk hypothesis (RWH) for financial time series by estimating Hurst coefficients for a fractional Brownian motion model of asset prices. The purpose of this paper is to extend his work by making the estimation procedure robust to heteroskedasticity and by addressing the multiple hypothesis testing problem.Design/methodology/approach– Unbiased, heteroskedasticity consistent, variance ratio estimates are calculated for end of day price data for eight time lags over 12 agricultural commodity futures (front month) and 40 US equities from 2000-2014. A bootstrapped stepdown procedure is used to obtain appropriate statistical confidence for the multiplicity of hypothesis tests. The variance ratio approach is compared against regression-based testing for fractionality.Findings– Failing to account for bias, heteroskedasticity, and multiplicity of testing can lead to large numbers of erroneous rejections of the null hypothesis of efficient markets following an independent random walk. Even with these adjustments, a few futures contracts significantly violate independence for short lags at the 99 percent level, and a number of equities/lags violate independence at the 95 percent level. When testing at the asset level, futures prices are found not to contain fractional properties, while some equities do.Research limitations/implications– Only a subsample of futures and equities, and only a limited number of lags, are evaluated. It is possible that multiplicity adjustments for larger numbers of tests would result in fewer rejections of independence.Originality/value– This paper provides empirical evidence that violations of the RWH for financial time series are likely to exist, but are perhaps less common than previously thought.

A nonparametric coherent confidence procedure

ABSTRACTHolm's step-down testing procedure starts with the smallest p-value and sequentially screens larger p-values without any information on confidence intervals. This article changes the conventional step-down testing framework by presenting a nonparametric procedure that starts with the largest p-value and sequentially screens smaller p-values in a step-by-step manner to construct a set of simultaneous confidence sets. We use a partitioning approach to prove that the new procedure controls the simultaneous confidence level (thus strongly controlling the familywise error rate). Discernible features of the new stepwise procedure include consistency with individual inference, coherence, and confidence estimations for follow-up investigations. In a simple simulation study, the proposed procedure (treated as a testing procedure), is more powerful than Holm's procedure when the correlation coefficient is large, and vice versa when it is small. In the data analysis of a medical study, the new procedure is able to detect the efficacy of Aspirin as a cardiovascular prophylaxis in a nonparametric setting.

Efficient computation of adjusted [formula omitted]-values for resampling-based stepdown multiple testing

There has been a recent interest in reporting p-values adjusted for the resampling-based stepdown multiple testing procedures proposed in Romano and Wolf (2005a,b). The original papers only describe how to carry out multiple testing at a fixed significance level. Computing adjusted p-values instead in an efficient manner is not entirely trivial. Therefore, this paper fills an apparent gap by detailing such an algorithm.

Improving Holm's procedure using pairwise dependencies

SUMMARY Seneta & Chen (2005) tightened the familywise error rate control of Holm’s procedure by sharpening its critical values using pairwise dependencies of the p-values. In this paper we further sharpen these critical values in the case where the distribution functions of the pairwise maxima of null p-values are convex, a property shown to hold in some applications of Holm’s procedure. The newer critical values are uniformly larger, providing tighter familywise error rate control than the approach of Seneta & Chen (2005), significantly so under high pairwise positive dependencies. The critical values can be further improved under exchangeable null p-values. Control of the familywise error rate, the probability of falsely rejecting at least one true null hypothesis, is commonly undertaken when testing multiple hypotheses. Among procedures for controlling this error rate, that of Holm (1979) is one of the most popular. Seneta & Chen (2005) attempted to improve upon Holm’s procedure in situations where pairwise dependencies among the p-values can be quantified. They applied an inequality of Kounias (1968) to obtain an upper bound for the distribution function of the minimum of a set of null p-values which is tighter than that provided by Bonferroni’s inequality, while modifying Holm’s critical values. The modification tightens the familywise error rate control of Holm’s procedure, and can be more powerful than the original step-up procedure of Hochberg (1988), as Seneta & Chen (2005) showed for some multiple testing problems associated with normally distributed test statistics with known correlations. We propose two improved versions of Holm’s step-down procedure for p-values such that the pairwise maxima of the null p-values have known convex distribution functions. The different versions depend on whether the null p-values are exchangeable, and each provides uniformly larger critical values than the method of Seneta & Chen (2005). The convexity of p-values is shown for some commonly used multivariate distributions. Numerical and simulation studies reveal that each of our proposed procedures is a better choice than the method of Seneta & Chen (2005), especially for small-scale multiple testing with high pairwise dependencies among the test statistics.

Variable selection using stepdown procedures in high-dimensional linear models

Variable selection using stepdown procedures in high-dimensional linear models

Efficient Computation of Adjusted P-Values for Resampling-Based Stepdown Multiple Testing

There has been a recent interest in reporting p-values adjusted for resampling-based stepdown multiple testing procedures proposed in Romano and Wolf (2005a,b). The original papers only describe how to carry out multiple testing at a fixed significance level. Computing adjusted p-values instead in an efficient manner is not entirely trivial. Therefore, this paper fills an apparent gap by detailing such an algorithm.

Early intervention and child physical health: Evidence from a Dublin-based randomized controlled trial

This article investigates the impact of an early intervention program, which experimentally modifies the parenting and home environment of disadvantaged families, on child physical health in the first 3 years of life. We recruited and randomized 233 (115 intervention, 118 control) pregnant women from a socioeconomically disadvantaged community in Dublin, Ireland into an intervention or control group. The treatment includes regular home visits commencing antenatally and an additional parenting course commencing at 2 years. Maternal reports of child health are assessed at 6, 12, 18, 24, and 36 months. Treatment effects are estimated using permutation testing to account for small sample size, inverse probability weighting to account for differential attrition, and both the stepdown procedure and an indices approach to account for multiple hypothesis testing. Following adjustment for multiple testing and attrition, we observe a positive and statistically significant main treatment effect for wheezing/asthma. The intervention group are 15.5 percentage points (pp) less likely to require medical attention for wheezing/asthma compared to the control group. Subgroup analysis reveals more statistically significant adjusted treatment effects for boys than girls regarding fewer health problems (d=0.63), accidents (23.9pp), and chest infections (22.8–37.9pp). Our results suggest that a community-based home visiting program may have favorable impacts on early health conditions.

Convexity issues in multivariate multiple testing of treatments vs. control

The problem of multiple testing of each of several treatment mean vectors versus a control mean vector is considered. Both one-sided and two-sided alternatives are treated. It is shown that typical choices for marginal test procedures will lead to step-down procedures that do not have convex acceptance regions. This lack of convexity has both intuitive and theoretical disadvantages. The only exception being linear tests in the one-sided problem. Although such a procedure is atypical, it not only has convex acceptance regions but is such that critical values are obtainable so that the overall procedure can control FDR or FWER. For both one-sided and two-sided alternatives, two other stepwise multiple testing methods are presented that do have convex acceptance regions.

Effects of the step-up/step-down and step-down milk feeding procedures on the performance, structural growth, and blood metabolites of Holstein dairy calves

The objective of this study was to compare the efficacy of the step-up/step-down (SUSD) and step-down (STP) methods of milk feeding on the performance, growth parameters, blood metabolites, and health of dairy calves. For this purpose, 45 Holstein male calves (3d of age and 41±3kg of body weight) were randomly assigned to one of the following milk feeding groups: (1) conventional (CONV; 4L/d milk from d 1 to 52, and 2L/d milk from d 53 to 56 of the study), (2) STP (6L/d milk from d 1 to 29, and 4L/d milk from d 30 to 45 of the study followed by feeding 2L/d milk from d 46 to 56 of the study), and (3) SUSD (6L/d milk from d 1 to 5, 8L/d milk from d 6 to 15, 10L/d milk from d 16 to 35, 8L/d milk from d 36 to 42, 6L/d milk from d 43 to 47, 4L/d milk from d 48 to 52 of the study followed by feeding 2L/d milk from d 53 to 56 of the study). Calves were weaned on d 56 and followed until d 70 of the study period. Animals had ad libitum access to the same starter ration during the entire study period (d 3 to 70). Starter intake, total dry matter intake, and average daily gain were greater in the SUSD calves than those in the CONV and STP treatments during the preweaning period. The SUSD procedure was found to increase body weight during the entire study and improved body length, wither height, heart girth, hip height, and hip width on d 56 and 70 of the study compared with the STP and CONV calves. The SUSD treatment increased blood glucose concentration on d 35 compared with CONV and STP procedures. The STP group had a higher starter intake than the CONV and SUSD animals. The STP animals exhibited a higher plasma concentration of total protein and BHBA than did the SUSD animals during the preweaning period. Neither the SUSD nor the STP procedure negatively affected the fecal score. In conclusion, the SUSD milk feeding procedure was found to improve the performance of dairy calves compared with the STP and CONV procedures. However, it appears that the STP procedure induces earlier reticulo-rumen developement in dairy calves than does the SUSD procedure.

An adaptive step-down procedure for fault variable identification

In a process with a large number of process variables (high-dimensional process), identifying which variables cause an out-of-control signal is a challenging issue for quality engineers. In this paper, we propose an adaptive step-down procedure using conditional T2 statistic for fault variable identification. While existing procedures focus on selecting variables that have strong evidence of a change, the proposed step-down procedure selects a variable having the weakest evidence of a change at each step based on the variables that are selected in previous steps. The information of selected unchanged variables is effectively utilised in obtaining a powerful conditional T2 test statistic for identifying the changed elements of the mean vector. The proposed procedure is designed to utilise the correlation information between fault and non-fault variables for the efficient fault variables identification. Further, the simulation results show that the proposed procedure has the better diagnostic performance compared with existing methods in terms of fault variable identification and computational complexity, especially when the number of the variables is high and the number of fault variables is small.

Multi-Objective Model Selection via Racing.

Model selection is a core aspect in machine learning and is, occasionally, multi-objective in nature. For instance, hyper-parameter selection in a multi-task learning context is of multi-objective nature, since all the tasks' objectives must be optimized simultaneously. In this paper, a novel multi-objective racing algorithm (RA), namely S-Race, is put forward. Given an ensemble of models, our task is to reliably identify Pareto optimal models evaluated against multiple objectives, while minimizing the total computational cost. As a RA, S-Race attempts to eliminate non-promising models with confidence as early as possible, so as to concentrate computational resources on promising ones. Simultaneously, it addresses the problem of multi-objective model selection (MOMS) in the sense of Pareto optimality. In S-Race, the nonparametric sign test is utilized for pair-wise dominance relationship identification. Moreover, a discrete Holm's step-down procedure is adopted to control the family-wise error rate of the set of hypotheses made simultaneously. The significance level assigned to each family is adjusted adaptively during the race. In order to illustrate its merits, S-Race is applied on three MOMS problems: 1) selecting support vector machines for classification; 2) tuning the parameters of artificial bee colony algorithms for numerical optimization; and 3) constructing optimal hybrid recommendation systems for movie recommendation. The experimental results confirm that S-Race is an efficient and effective MOMS algorithm compared to a brute-force approach.

Factors Associated with Quality of Life in Adult Epilepsy Patients: a Hospital Based Study from South India

Background: Epilepsy is a chronic condition which affects quality of life (QOL). QOL is considered as an important outcome measure and component of management in studies of adult epilepsy. Objective: To evaluate factors associated with quality of life in adult epilepsy patients. Materials and Methods: A cross-sectional study was performed to examine the quality of life in 560 adult patients with epilepsy. The data collected included detailed clinical and socio-demographic data, epilepsy details, psychiatric diagnosis (ICD-10 for Mental and behavioral disorders), Liverpool seizure severity scale (LSSS) and Quality of life in epilepsy (QOLIE-31) are assessed. Descriptive statistics-Percentages, ANOVA, Univariate odds and Multiple Logistic Regression analysis with step-down procedure were done. Results: Study population comprised 337 males and 223 females with mean age of 29.26 + 10.83 (range 18-50) years. On Univariate Odds ratio (OR) (95% CI) for single anti-epileptic drug (AED) 2.75(1.85 to 4.09) and absence of psychiatric diagnosis 1.60(1.02 to 2.53) are predictors for good QOL. On one-way ANOVA with QOLIE subscales, seizure frequency and psychiatric diagnosis were found to be statistically significant (p<0.01) but no significant interaction between them. On Logistic Regression step down procedure, psychiatric diagnosis (OR 95% C.I) 7.29(1.65 to 32.10) and multiple AED 1.86(1.24 to 3.51) was found to be predicting factors for poor QOLIE. Conclusion: The presence of psychiatric diagnosis (Depression and Anxiety) was the strongest predictor of poor QOLIE patients. Early psychiatric evaluation and intervention would improve quality of life in epilepsy patients.

Family-Wise Separation Rates for multiple testing

Starting from a parallel between some minimax adaptive tests of a single null hypothesis, based on aggregation approaches, and some tests of multiple hypotheses, we propose a new second kind error-related evaluation criterion, as the core of an emergent minimax theory for multiple tests. Aggregation-based tests are justified through their first kind error rate, which is controlled by the prescribed level on the one hand, and through their separation rates over various classes of alternatives, rates that are minimax on the other hand. We show that these tests can be viewed as the first steps of classical step-down multiple testing procedures, and ac-cordingly be evaluated from the multiple testing point of view also, through a control of their Family-Wise Error Rate (FWER). Conversely, many multiple testing procedures, from the historical ones of Bonferroni and Holm, to more recent ones like min-p procedures or randomized procedures, can be investigated from the minimax adaptive testing point of view. To this end, we extend the notion of separation rate to the multiple testing field, by defining the weak Family-Wise Separation Rate and its stronger counterpart, the Family-Wise Separation Rate (FWSR). As for non-parametric tests of a single null hypothesis, we prove that these new concepts allow an accurate analysis of the second kind error of a multiple testing procedure, leading to clear definitions of minimax and minimax adaptive multiple tests. Some illustrations in a classical Gaussian framework corroborate several expected results under particular conditions on the tested hypotheses, but also lead to more surprising results.

Expression, purification, and characterization of scar tissue neovasculature endothelial cell-targeted rhIL10 in Escherichia coli.

Interleukin 10 (IL10) plays a pivotal role in the anti-inflammatory response and immunosuppressive reactions. It has also been identified as a new promising therapy for scar formation. Treatment of scars with IL10 has significant effects, but there are some shortcomings, including poor tissue-binding specificity and low effectiveness. RGD peptide has been demonstrated to bind specifically to αvβ3 integrin on neovasculature endothelial cells, and the excess production of neovasculature is crucial to scar formation. To increase efficacy against scar formation and to decrease the side effects on normal tissues, a novel hybrid protein combining human IL10 with RGD was designed. The DNA sequence encoding the recombinant fusion protein IL10-RGD (rhIL10-RGD) was subcloned into a pET22b (+) vector for protein expression in E. coli strain BL21 (DE3). SDS-PAGE analysis displayed an induced expression product band at a molecular weight of 19.3 kDa, which constituted 30 % of the total bacterial protein. We developed a procedure to purify rhIL10-RGD from inclusion bodies and then renatured the protein using dialysis against urea with a step-down concentration procedure. Hypertrophic scar fibroblasts (HSFs) were treated with rhIL10-RGD, and the fibrosis-related protein levels were assessed by Western blotting. The results indicated that rhIL10-RGD can downregulate the expression levels of Col1 and α-SMA in HSFs and suppress tube formation of HUVECs. These results indicate that rhIL10-RGD has anti-fibrosis effects and can potentially be used to treat the neovasculature in scar formation and improve the abnormal deposition of the extracellular matrix (ECM). Thus, rhIL10-RGD may be a more effective candidate for scar-improvement and anti-fibrosis therapy.

Multiple Comparisons for a Psychophysical Test in Bootstrap Logistic Regression

We propose an algorithm of multiple comparisons with a control for a psychophysical test. Our algorithm is based on the step-down procedure and is applicable to the bootstrap test in logistic regression. We use logistic regression combined with guessing rate and log-likelihood ratio test statistics of multiple samples in order to test hypotheses by using non-parametric bootstrap resampling. We apply our algorithm to visual acuity measurement, and show that bootstrap resampling can be used to resolve problems with multiple comparisons especially when the numbers of observations among samples are not identical.

Further results on controlling the false discovery proportion

The probability of false discovery proportion (FDP) exceeding $\gamma\in[0,1)$, defined as $\gamma$-FDP, has received much attention as a measure of false discoveries in multiple testing. Although this measure has received acceptance due to its relevance under dependency, not much progress has been made yet advancing its theory under such dependency in a nonasymptotic setting, which motivates our research in this article. We provide a larger class of procedures containing the stepup analog of, and hence more powerful than, the stepdown procedure in Lehmann and Romano [Ann. Statist. 33 (2005) 1138-1154] controlling the $\gamma$-FDP under similar positive dependence condition assumed in that paper. We offer better alternatives of the stepdown and stepup procedures in Romano and Shaikh [IMS Lecture Notes Monogr. Ser. 49 (2006a) 33-50, Ann. Statist. 34 (2006b) 1850-1873] using pairwise joint distributions of the null $p$-values. We generalize the notion of $\gamma$-FDP making it appropriate in situations where one is willing to tolerate a few false rejections or, due to high dependency, some false rejections are inevitable, and provide methods that control this generalized $\gamma$-FDP in two different scenarios: (i) only the marginal $p$-values are available and (ii) the marginal $p$-values as well as the common pairwise joint distributions of the null $p$-values are available, and assuming both positive dependence and arbitrary dependence conditions on the $p$-values in each scenario. Our theoretical findings are being supported through numerical studies.

Selection in parametric models via some stepdown procedures

Selection in parametric models via some stepdown procedures

On some stepdown procedures with application to consistent variable selection in linear regression

The paper considers the problem of consistent variable selection with the use of stepdown procedures in the classical linear regression model and for the model with dependent errors. The stated results complete the results obtained by Bunea et al. [Consistent variable selection in high dimensional regression via multiple testing. J Stat Plann Inference. 2006;136(12):4349–4364].