True Null Hypotheses Research Articles

Adaptive $p$-value weighting with power optimality

Weighting the p-values is a well-established strategy that improves the power of multiple testing procedures while dealing with heterogeneous data. However, how to achieve this task in an optimal way is rarely considered in the literature. This paper contributes to fill the gap in the case of group-structured null hypotheses, by introducing a new class of procedures named ADDOW (for Adaptive Data Driven Optimal Weighting) that adapts both to the alternative distribution and to the proportion of true null hypotheses. We prove the asymptotical FDR control and power optimality among all weighted procedures of ADDOW, which shows that it dominates all existing procedures in that framework. Some numerical experiments show that the proposed method preserves its optimal properties in the finite sample setting when the number of tests is moderately large.

Variability and stability of the false discovery proportion

Much effort has been done to control the “false discovery rate” (FDR) when $m$ hypotheses are tested simultaneously. The FDR is the expectation of the “false discovery proportion” $\text{FDP}=V/R$ given by the ratio of the number of false rejections $V$ and all rejections $R$. In this paper, we have a closer look at the FDP for adaptive linear step-up multiple tests. These tests extend the well known Benjamini and Hochberg test by estimating the unknown amount $m_{0}$ of the true null hypotheses. We give exact finite sample formulas for higher moments of the FDP and, in particular, for its variance. Using these allows us a precise discussion about the stability of the FDP, i.e., when the FDP is asymptotically close to its mean. We present sufficient and necessary conditions for this stability. They include the presence of a stable estimator for the proportion $m_{0}/m$. We apply our results to convex combinations of generalized Storey type estimators with various tuning parameters and (possibly) data-driven weights. The corresponding step-up tests allow a flexible adaptation. Moreover, these tests control the FDR at finite sample size. We compare these tests to the classical Benjamini and Hochberg test and discuss the advantages of them.

Dynamic adaptive procedures that control the false discovery rate

In the multiple testing problem with independent tests, the classical linear step-up procedure controls the false discovery rate (FDR) at level $\pi_{0}\alpha$, where $\pi_{0}$ is the proportion of true null hypotheses and $\alpha$ is the target FDR level. Adaptive procedures can improve power by incorporating estimates of $\pi_{0}$, which typically rely on a tuning parameter. Fixed adaptive procedures set their tuning parameters before seeing the data and can be shown to control the FDR in finite samples. We develop theoretical results for dynamic adaptive procedures whose tuning parameters are determined by the data. We show that, if the tuning parameter is chosen according to a stopping time rule, the corresponding dynamic adaptive procedure controls the FDR in finite samples. Examples include the recently proposed right-boundary procedure and the widely used lowest-slope procedure, among others. Simulation results show that the right-boundary procedure is more powerful than other dynamic adaptive procedures under independence and mild dependence conditions. The right-boundary procedure is implemented in the Bioconductor R package calm.

Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence

From a survey of the papers published in leading accounting journals in 2014, we find that accounting researchers conduct significance testing almost exclusively at a conventional level of significance, without considering key factors such as the sample size or power of a test. We present evidence that a vast majority of the accounting studies favour large or massive sample sizes and conduct significance tests with the power extremely close to or equal to one. As a result, statistical inference is severely biased towards Type I error, frequently rejecting the true null hypotheses. Under the ‘p‐value less than 0.05’ criterion for statistical significance, more than 90% of the surveyed papers report statistical significance. However, under alternative criteria, only 40% of the results are statistically significant. We propose that substantial changes be made to the current practice of significance testing for more credible empirical research in accounting.

Hyper Nonlocal Priors for Variable Selection in Generalized Linear Models

We propose two novel hyper nonlocal priors for variable selection in generalized linear models. To obtain these priors, we first derive two new priors for generalized linear models that combine the Fisher information matrix with the Johnson-Rossell moment and inverse moment priors. We then obtain our hyper nonlocal priors from our nonlocal Fisher information priors by assigning hyperpriors to their scale parameters. As a consequence, the hyper nonlocal priors bring less information on the effect sizes than the Fisher information priors, and thus are very useful in practice whenever the prior knowledge of effect size is lacking. We develop a Laplace integration procedure to compute posterior model probabilities, and we show that under certain regularity conditions the proposed methods are variable selection consistent. We also show that, when compared to local priors, our hyper nonlocal priors lead to faster accumulation of evidence in favor of a true null hypothesis. Simulation studies that consider binomial, Poisson, and negative binomial regression models indicate that our methods select true models with higher success rates than other existing Bayesian methods. Furthermore, the simulation studies show that our methods lead to mean posterior probabilities for the true models that are closer to their empirical success rates. Finally, we illustrate the application of our methods with an analysis of the Pima Indians diabetes dataset.

Spurious weather effects

AbstractRainfall is exogenous to human actions and hence popular as an exogenous source of variation. But it is also spatially correlated. This can generate spurious relationships between rainfall and other spatially correlated outcomes. As an illustration, rainfall on almost any day of the year has seemingly high predictive power of electoral turnout in Norwegian municipalities. In Monte Carlo analyses, I find that standard tests reject true null hypotheses in as much as 99% of cases. Standard approaches to estimating consistent standard errors do not solve the problem. Instead, I suggest controlling for spatial and spatiotemporal trends using multidimensional polynomials.

Did assessors select attributes by chance alone in your TDS study, and how relevant is it to know?

Dominance rates arising from Temporal Dominance of Sensations (TDS) data are almost always plotted and understood with reference to chance and significance lines, which are based on the assumption of random attribute selection. These lines are auxiliary to the chart, and used to interpret the dominance rates; when the dominance rate for some attribute exceeds the significance line, the conventional interpretation is that this attribute is being noticed significantly. Thus, these auxiliary lines are used to provide a convenient, qualitative interpretation of TDS data. However, the concept of the significance lines has some deficiencies that we explore in this paper.We were interested in what real TDS data look like under a true null hypothesis of random attribute selection. To obtain such data, we sampled from real TDS data (keeping products and assessors' selection patterns intact) to create dominance rates based on random attribute selections. Dominance rates exceed the significance line about 5% of the time (across attributes and time points), as expected. These curves differed from any real TDS curves that we have ever observed previously insofar as that they were unusually flattened.Next, we sampled again from real TDS data (keeping attributes and assessors' selection patterns intact) to create dominance rates based on attribute selections for random (hybrid) products. Derived TDS curves much more closely resemble curves from real TDS studies and show a lot of “significances”, even though products are artificially created by the permutations. This is due to properties of the products that are inherent to the product category: e.g., hard will occur early during the evaluation of a chocolate (if at all), but not towards the end.From these simulations, we reach two conclusions. First, we recognize that the hypothesis that assessors select the dominant attribute completely at random is almost always false. Second, the reason that it is false is that, regardless of the product, there is a time-dependent category signature present, as well as competition between attributes, such that significances for one attribute depend on which other attributes are on the list. These properties limit the informative value of the conventional TDS significance lines that are widely used.As an alternative, we explore the possibility of context-dependent significance thresholds which are customized for and aligned with research objectives. These reference lines are recommended for improved interpretation of data arising from TDS studies.

Weighted multiple testing procedure for grouped hypotheses with k-FWER control

In this paper, k-FWER (generalized familywise error rate) control for grouped hypotheses testing is considered. We offer the weights for the p-values in each group, by maximizing an objective function, which is the expectation of the proportion of rejected hypotheses. This objective function utilizes not only the information of the proportion of true null hypotheses, but also the null and non-null distributions of the p-values in each group. When this information is known prior, our weighted testing procedure controls k-FWER for arbitrarily dependent p-values. When this information is unknown, and is estimated from the data, our procedure asymptotically controls k-FWER under the weak dependence assumption of the p-values in each group. The new procedure is shown to be more powerful than some existing procedures both in theory and simulations. For illustration, the proposed procedure is applied to analyse the adequate yearly progress data.

Non-inferiority studies: is 'better' the enemy of 'good enough'?

Non-inferiority studies: is 'better' the enemy of 'good enough'?

Spatially Dependent Multiple Testing Under Model Misspecification, With Application to Detection of Anthropogenic Influence on Extreme Climate Events

ABSTRACTThe Weather Risk Attribution Forecast (WRAF) is a forecasting tool that uses output from global climate models to make simultaneous attribution statements about whether and how greenhouse gas emissions have contributed to extreme weather across the globe. However, in conducting a large number of simultaneous hypothesis tests, the WRAF is prone to identifying false “discoveries.” A common technique for addressing this multiple testing problem is to adjust the procedure in a way that controls the proportion of true null hypotheses that are incorrectly rejected, or the false discovery rate (FDR). Unfortunately, generic FDR procedures suffer from low power when the hypotheses are dependent, and techniques designed to account for dependence are sensitive to misspecification of the underlying statistical model. In this article, we develop a Bayesian decision-theoretical approach for dependent multiple testing and a nonparametric hierarchical statistical model that flexibly controls false discovery and is robust to model misspecification. We illustrate the robustness of our procedure to model error with a simulation study, using a framework that accounts for generic spatial dependence and allows the practitioner to flexibly specify the decision criteria. Finally, we apply our procedure to several seasonal forecasts and discuss implementation for the WRAF workflow. Supplementary materials for this article, including a standardized description of the materials available for reproducing the work, are available as an online supplement.

Multiple testing with discrete data: Proportion of true null hypotheses and two adaptive FDR procedures.

We consider multiple testing with false discovery rate (FDR) control when p values have discrete and heterogeneous null distributions. We propose a new estimator of the proportion of true null hypotheses and demonstrate that it is less upwardly biased than Storey's estimator and two other estimators. The new estimator induces two adaptive procedures, that is, an adaptive Benjamini-Hochberg (BH) procedure and an adaptive Benjamini-Hochberg-Heyse (BHH) procedure. We prove that the adaptive BH (aBH) procedure is conservative nonasymptotically. Through simulation studies, we show that these procedures are usually more powerful than their nonadaptive counterparts and that the adaptive BHH procedure is usually more powerful than the aBH procedure and a procedure based on randomized p-value. The adaptive procedures are applied to a study of HIV vaccine efficacy, where they identify more differentially polymorphic positions than the BH procedure at the same FDR level.

Nonparametric inference for functional‐on‐scalar linear models applied to knee kinematic hop data after injury of the anterior cruciate ligament

AbstractMotivated by the analysis of the dependence of knee movement patterns during functional tasks on subject‐specific covariates, we introduce a distribution‐free procedure for testing a functional‐on‐scalar linear model with fixed effects. The procedure does not only test the global hypothesis on the entire domain but also selects the intervals where statistically significant effects are detected. We prove that the proposed tests are provided with an asymptotic control of the intervalwise error rate, that is, the probability of falsely rejecting any interval of true null hypotheses. The procedure is applied to one‐leg hop data from a study on anterior cruciate ligament injury. We compare knee kinematics of three groups of individuals (two injured groups with different treatments and one group of healthy controls), taking individual‐specific covariates into account.

Control procedures and estimators of the false discovery rate and their application in low-dimensional settings: an empirical investigation

BackgroundWhen many (up to millions) of statistical tests are conducted in discovery set analyses such as genome-wide association studies (GWAS), approaches controlling family-wise error rate (FWER) or false discovery rate (FDR) are required to reduce the number of false positive decisions. Some methods were specifically developed in the context of high-dimensional settings and partially rely on the estimation of the proportion of true null hypotheses. However, these approaches are also applied in low-dimensional settings such as replication set analyses that might be restricted to a small number of specific hypotheses. The aim of this study was to compare different approaches in low-dimensional settings using (a) real data from the CKDGen Consortium and (b) a simulation study.ResultsIn both application and simulation FWER approaches were less powerful compared to FDR control methods, whether a larger number of hypotheses were tested or not. Most powerful was the q-value method. However, the specificity of this method to maintain true null hypotheses was especially decreased when the number of tested hypotheses was small. In this low-dimensional situation, estimation of the proportion of true null hypotheses was biased.ConclusionsThe results highlight the importance of a sizeable data set for a reliable estimation of the proportion of true null hypotheses. Consequently, methods relying on this estimation should only be applied in high-dimensional settings. Furthermore, if the focus lies on testing of a small number of hypotheses such as in replication settings, FWER methods rather than FDR methods should be preferred to maintain high specificity.

Adaptive False Discovery Rate Control for Heterogeneous Data

Efforts to develop more efficient multiple hypothesis testing procedures for false discovery rate (FDR) control have focused on incorporating an estimate of the proportion of true null hypotheses (such procedures are called adaptive) or exploiting heterogeneity across tests via some optimal weighting scheme. This paper combines these approaches using a weighted adaptive multiple decision function (WAMDF) framework. Optimal weights for a flexible random effects model are derived and a WAMDF that controls the FDR for arbitrary weighting schemes when test statistics are independent under the null hypotheses is given. Asymptotic and numerical assessment reveals that, under weak dependence, the proposed WAMDFs provide more efficient FDR control even if optimal weights are misspecified. The robustness and flexibility of the proposed methodology facilitates the development of more efficient, yet practical, FDR procedures for heterogeneous data. To illustrate, two different weighted adaptive FDR methods for heterogeneous sample sizes are developed and applied to data.

Bridging observational studies and randomized experiments by embedding the former in the latter.

Consider a statistical analysis that draws causal inferences from an observational dataset, inferences that are presented as being valid in the standard frequentist senses; i.e. the analysis produces: (1) consistent point estimates, (2) valid p-values, valid in the sense of rejecting true null hypotheses at the nominal level or less often, and/or (3) confidence intervals, which are presented as having at least their nominal coverage for their estimands. For the hypothetical validity of these statements, the analysis must embed the observational study in a hypothetical randomized experiment that created the observed data, or a subset of that hypothetical randomized data set. This multistage effort with thought-provoking tasks involves: (1) a purely conceptual stage that precisely formulate the causal question in terms of a hypothetical randomized experiment where the exposure is assigned to units; (2) a design stage that approximates a randomized experiment before any outcome data are observed, (3) a statistical analysis stage comparing the outcomes of interest in the exposed and non-exposed units of the hypothetical randomized experiment, and (4) a summary stage providing conclusions about statistical evidence for the sizes of possible causal effects. Stages 2 and 3 may rely on modern computing to implement the effort, whereas Stage 1 demands careful scientific argumentation to make the embedding plausible to scientific readers of the proffered statistical analysis. Otherwise, the resulting analysis is vulnerable to criticism for being simply a presentation of scientifically meaningless arithmetic calculations. The conceptually most demanding tasks are often the most scientifically interesting to the dedicated researcher and readers of the resulting statistical analyses. This perspective is rarely implemented with any rigor, for example, completely eschewing the first stage. We illustrate our approach using an example examining the effect of parental smoking on children's lung function collected in families living in East Boston in the 1970s.

Significance Testing in Accounting Research: A Critical Evaluation Based on Evidence

From a survey of the papers published in leading accounting journals in 2014, we find that accounting researchers conduct significance testing almost exclusively at a conventional level of significance, without considering key factors such as the sample size or power of a test. We present evidence that a vast majority of the accounting studies favour large or massive sample sizes and conduct significance tests with the power extremely close to or equal to one. As a result, statistical inference is severely biased towards Type I error, frequently rejecting the true null hypotheses. Under the ‘p‐value less than 0.05’ criterion for statistical significance, more than 90% of the surveyed papers report statistical significance. However, under alternative criteria, only 40% of the results are statistically significant. We propose that substantial changes be made to the current practice of significance testing for more credible empirical research in accounting.

Adaptive procedure for generalized familywise error rate control

ABSTRACTThis article considers multiple hypotheses testing with the generalized familywise error rate k-FWER control, which is the probability of at least k false rejections. We first assume the p-values corresponding to the true null hypotheses are independent, and propose adaptive generalized Bonferroni procedure with k-FWER control based on the estimation of the number of true null hypotheses. Then, we assume the p-values are dependent, satisfying block dependence, and propose adaptive procedure with k-FWER control. Extensive simulations compare the performance of the adaptive procedures with different estimators.

Misspecification in event studies

We examine the statistical error and efficiency associated with two commonly used event-study techniques when applied to samples of various sizes. Previous research has established that the frequently used Patell (1976) test is not well specified when the event itself creates additional return variance. We find that even under ideal conditions when the event creates no additional variance, the Patell test rejects a true null hypothesis substantially more often than the stated significance level. In contrast, the alternate test of Boehmer et al. (1991) performs well in samples of all sizes and under all conditions we consider.

Penalised empirical likelihood for the additive hazards model with high-dimensional data

ABSTRACTIn this article, we apply the empirical likelihood (EL) method to the additive hazards model with high-dimensional data and propose the penalised empirical likelihood (PEL) method for parameter estimation and variable selection. It is shown that the estimator based on the EL method has the efficient property, and the limit distribution of the EL ratio statistic for the parameters is a asymptotic normal distribution under the true null hypothesis. In a high-dimensional setting, we prove that the PEL method in the additive hazards model has the oracle property, that is, with probability tending to 1, and the estimator based on the PEL method for the nonzero parameters is estimation and selection consistent if the hypothesised model is true. Moreover, the PEL ratio statistic for the parameters is a distribution under the true null hypothesis. The performance of the proposed methods is illustrated via a real data application and numerical simulations.

Untangling the Tangled Bank: A Novel Method for Partitioning the Effects of Phylogenies and Traits on Ecological Networks

Understanding how evolutionary and ecological processes shape species interaction networks remains as one of the main challenges in eco-evolutionary studies. Here, we present an integrative analytical framework to partition the effects of phylogenies and functional traits on the structure of ecological networks. The method combines fuzzy set theory and matrix correlation, implemented under a Monte Carlo framework. We designed a simulation study in order to estimate the accuracy of the methods proposed here, measuring Type I Error rates. The simulation study shows that the method is accurate, i.e., incorrectly rejecting a true null hypothesis in ~5% of the cases and falling within the confidence interval. We illustrate our framework using data from a seed dispersal network from southern Brazil. Our analyses suggest that birds must have specific traits in order to consume their plant resources, and that phylogenetic resemblance has no explanatory power for species traits and species interactions in this seed-dispersal network.