Multiple Testing Problem Research Articles

The ghost of selective inference in spatiotemporal trend analysis

In spatiotemporal trend analysis, selective inference occurs when researchers are only interested in significant trends based on a fixed threshold (α, often 0.05), without considering the total number of statistical tests performed. Using simultaneous inference in gridded data involves thousands of trend tests, one for each pixel, leading to multiple testing or multiplicity problems. Multiplicity increases the chance of false discoveries in an unknown way unless the p-values of all tests performed are appropriately considered and adjusted.This discussion paper provides a selective and non-exhaustive review of the problems of multiplicity and selective inference. We discuss some appropriate methods to cope with the inflation of spurious results and comment on some examples based on gridded data in the context of research on spatiotemporal trend analysis. In addition, we suggest some good practices in transparency to facilitate the replicability of studies.The effects of uncorrected multiplicity and selective interference can be likened to a ghostly layer over the data, projecting illusions of significance that vanish with rigorous correction methods, revealing the true statistical skeleton of the results. The basis for addressing these problems is to assume that, although it may sometimes seem counterintuitive, the reality of what we perceive as statistically significant (i.e., p-values <0.05) also depends on the number (and value) of what we perceive as non-significant (i.e., p-values ≥0.05). Indeed, in a multiplicity context, one cannot correctly decide what is statistically significant until the whole story is known. Uncorrected selective inference precisely involves ignoring part of the story.

ONDSA: a testing framework based on Gaussian graphical models for differential and similarity analysis of multiple omics networks.

The Gaussian graphical model (GGM) is a statistical network approach that represents conditional dependencies among components, enabling a comprehensive exploration of disease mechanisms using high-throughput multi-omics data. Analyzing differential and similar structures in biological networks across multiple clinical conditions can reveal significant biological pathways and interactions associated with disease onset and progression. However, most existing methods for estimating group differences in sparse GGMs only apply to comparisons between two groups, and the challenging problem of multiple testing across multiple GGMs persists. This limitation hinders the ability to uncover complex biological insights that arise from comparing multiple conditions simultaneously. To address these challenges, we propose the Omics Networks Differential and Similarity Analysis (ONDSA) framework, specifically designed for continuous omics data. ONDSA tests for structural differences and similarities across multiple groups, effectively controlling the false discovery rate (FDR) at a desired level. Our approach focuses on entry-wise comparisons of precision matrices across groups, introducing two test statistics to sequentially estimate structural differences and similarities while adjusting for correlated effects in FDR control procedures. We show via comprehensive simulations that ONDSA outperforms existing methods under a range of graph structures and is a valuable tool for joint comparisons of multiple GGMs. We also illustrate our method through the detection of neuroinflammatory pathways in a multi-omics dataset from the Framingham Heart Study Offspring cohort, involving three apolipoprotein E genotype groups. It highlights ONDSA's ability to provide a more holistic view of biological interactions and disease mechanisms through multi-omics data integration.

Two-sided Estimates for the Sum of Probabilities of Errors in the Multiple Hypothesis Testing Problem with Finite Number of Hypotheses on a Nonhomogeneous Sample

Two-sided Estimates for the Sum of Probabilities of Errors in the Multiple Hypothesis Testing Problem with Finite Number of Hypotheses on a Nonhomogeneous Sample

Addressing researcher degrees of freedom through minP adjustment

When different researchers study the same research question using the same dataset they may obtain different and potentially even conflicting results. This is because there is often substantial flexibility in researchers’ analytical choices, an issue also referred to as “researcher degrees of freedom”. Combined with selective reporting of the smallest p-value or largest effect, researcher degrees of freedom may lead to an increased rate of false positive and overoptimistic results. In this paper, we address this issue by formalizing the multiplicity of analysis strategies as a multiple testing problem. As the test statistics of different analysis strategies are usually highly dependent, a naive approach such as the Bonferroni correction is inappropriate because it leads to an unacceptable loss of power. Instead, we propose using the “minP” adjustment method, which takes potential test dependencies into account and approximates the underlying null distribution of the minimal p-value through a permutation-based procedure. This procedure is known to achieve more power than simpler approaches while ensuring a weak control of the family-wise error rate. We illustrate our approach for addressing researcher degrees of freedom by applying it to a study on the impact of perioperative paO2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$paO_2$$\\end{document} on post-operative complications after neurosurgery. A total of 48 analysis strategies are considered and adjusted using the minP procedure. This approach allows to selectively report the result of the analysis strategy yielding the most convincing evidence, while controlling the type 1 error—and thus the risk of publishing false positive results that may not be replicable.

Correction to: Optimal and Maximin Procedures for Multiple Testing Problems

Correction to: Optimal and Maximin Procedures for Multiple Testing Problems

Survey expectations and adjustments for multiple testing

Testing hypotheses regarding how individual survey respondents form their expectations is susceptible to the multiple testing problem. The probability of falsely rejecting the null hypothesis for one or more respondents will exceed the nominal single-hypothesis significance level. The Bonferroni correction and related approaches control the family-wise error rate, but are conservative and result in low power when the null hypotheses are false.We compare controlling the family-wise error rate with the effect of controlling the false discovery rate and the false discovery proportion, in terms of the conclusions we draw about forecaster behaviour.The effects of adjustments for multiple testing are investigated for tests of weak efficiency and the over-reaction hypothesis, for beliefs about the persistence of shocks to output growth, and for the accuracy of survey respondents’ perceptions of the uncertainty they face.

Missing data in amortized simulation-based neural posterior estimation.

Amortized simulation-based neural posterior estimation provides a novel machine learning based approach for solving parameter estimation problems. It has been shown to be computationally efficient and able to handle complex models and data sets. Yet, the available approach cannot handle the in experimental studies ubiquitous case of missing data, and might provide incorrect posterior estimates. In this work, we discuss various ways of encoding missing data and integrate them into the training and inference process. We implement the approaches in the BayesFlow methodology, an amortized estimation framework based on invertible neural networks, and evaluate their performance on multiple test problems. We find that an approach in which the data vector is augmented with binary indicators of presence or absence of values performs the most robustly. Indeed, it improved the performance also for the simpler problem of data sets with variable length. Accordingly, we demonstrate that amortized simulation-based inference approaches are applicable even with missing data, and we provide a guideline for their handling, which is relevant for a broad spectrum of applications.

Statistical inference for diagnostic test accuracy studies with multiple comparisons.

Diagnostic accuracy studies assess the sensitivity and specificity of a new index test in relation to an established comparator or the reference standard. The development and selection of the index test are usually assumed to be conducted prior to the accuracy study. In practice, this is often violated, for instance, if the choice of the (apparently) best biomarker, model or cutpoint is based on the same data that is used later for validation purposes. In this work, we investigate several multiple comparison procedures which provide family-wise error rate control for the emerging multiple testing problem. Due to the nature of the co-primary hypothesis problem, conventional approaches for multiplicity adjustment are too conservative for the specific problem and thus need to be adapted. In an extensive simulation study, five multiple comparison procedures are compared with regard to statistical error rates in least-favourable and realistic scenarios. This covers parametric and non-parametric methods and one Bayesian approach. All methods have been implemented in the new open-source R package cases which allows us to reproduce all simulation results. Based on our numerical results, we conclude that the parametric approaches (maxT and Bonferroni) are easy to apply but can have inflated type I error rates for small sample sizes. The two investigated Bootstrap procedures, in particular the so-called pairs Bootstrap, allow for a family-wise error rate control in finite samples and in addition have a competitive statistical power.

Unraveling the gut-brain connection: The association of microbiota-linked structural brain biomarkers with behavior and mental health.

The gut microbiota can influence human behavior. However, due to the massive multiple-testing problem, research into the relationship between microbiome ecosystems and the human brain faces drawbacks. This problem arises when attempting to correlate thousands of gut bacteria with thousands of brain voxels. We performed brain magnetic resonance imaging (MRI) scans on 133 participants and applied machine-learning algorithms (Ridge regressions) combined with permutation tests. Using this approach, we were able to correlate specific gut bacterial families with brain MRI signals, circumventing the difficulties of massive multiple testing while considering sex, age, and body mass index as confounding factors. The relative abundance (RA) of the Selenomonadaceae, Clostridiaceae, and Veillonellaceae families in the gut was associated with altered cerebellar, visual, and frontal T2-mapping and diffusion tensor imaging measures. Conversely, decreased relative abundance of the Eubacteriaceae family was also linked to T2-mapping values in the cerebellum. Significantly, the brain regions associated with the gut microbiome were also correlated with depressive symptoms and attentional deficits. Our analytical strategy offers a promising approach for identifying potential brain biomarkers influenced by gut microbiota. By gathering a deeper understanding of the microbiota-brain connection, we can gain insights into the underlying mechanisms and potentially develop targeted interventions to mitigate the detrimental effects of dysbiosis on brain function and mental health.

Transfer Learning in Multiple Hypothesis Testing.

In this investigation, a synthesis of Convolutional Neural Networks (CNNs) and Bayesian inference is presented, leading to a novel approach to the problem of Multiple Hypothesis Testing (MHT). Diverging from traditional paradigms, this study introduces a sequence-based uncalibrated Bayes factor approach to test many hypotheses using the same family of sampling parametric models. A two-step methodology is employed: initially, a learning phase is conducted utilizing simulated datasets encompassing a wide spectrum of null and alternative hypotheses, followed by a transfer phase applying this fitted model to real-world experimental sequences. The outcome is a CNN model capable of navigating the complex domain of MHT with improved precision over traditional methods, also demonstrating robustness under varying conditions, including the number of true nulls and dependencies between tests. Although indications of empirical evaluations are presented and show that the methodology will prove useful, more work is required to provide a full evaluation from a theoretical perspective. The potential of this innovative approach is further illustrated within the critical domain of genomics. Although formal proof of the consistency of the model remains elusive due to the inherent complexity of the algorithms, this paper also provides some theoretical insights and advocates for continued exploration of this methodology.

Cost-Aware Generalized α-Investing for Multiple Hypothesis Testing

We consider the problem of sequential multiple hypothesis testing with nontrivial data collection costs. This problem appears, for example, when conducting biological experiments to identify differentially expressed genes of a disease process. This work builds on the generalized α-investing framework which enables control of the marginal false discovery rate in a sequential testing setting. We make a theoretical analysis of the long term asymptotic behavior of α-wealth which motivates a consideration of sample size in the α-investing decision rule. Posing the testing process as a game with nature, we construct a decision rule that optimizes the expected α-wealth reward (ERO) and provides an optimal sample size for each test. Empirical results show that a cost-aware ERO decision rule correctly rejects more false null hypotheses than other methods for $n=1$ where n is the sample size. When the sample size is not fixed cost-aware ERO uses a prior on the null hypothesis to adaptively allocate of the sample budget to each test. We extend cost-aware ERO investing to finite-horizon testing which enables the decision rule to allocate samples in a non-myopic manner. Finally, empirical tests on real data sets from biological experiments show that cost-aware ERO balances the allocation of samples to an individual test against the allocation of samples across multiple tests.

An Empirical Bayes Approach to Controlling the False Discovery Exceedance

In large-scale multiple hypothesis testing problems, the false discovery exceedance (FDX) provides a desirable alternative to the widely used false discovery rate (FDR) when the false discovery proportion (FDP) is highly variable. We develop an empirical Bayes approach to control the FDX. We show that, for independent hypotheses from a two-group model and dependent hypotheses from a Gaussian model fulfilling the exchangeability condition, an oracle decision rule based on ranking and thresholding the local false discovery rate (lfdr) is optimal in the sense that the power is maximized subject to the FDX constraint. We propose a data-driven FDX procedure that uses carefully designed computational shortcuts to emulate the oracle rule. We investigate the empirical performance of the proposed method using both simulated and real data and study the merits of FDX control through an application for identifying abnormal stock trading strategies.

Open-source complex-geometry 3D fluid dynamics for applications with unpredictable heterogeneous dynamic high-performance-computing loads

We discuss the software implementation of a finite element method, intended for the simulation of complex-geometry 3D flows, using hardware resources effectively, including problems with a priori unknown, heterogeneous, and dynamic parallel computational load. Our fundamental choices of data structures, algorithm, and software design specifically target engineering resolution and accuracy requirements. Some of these choices are: unstructured grids (to explicitly resolve complex 3D geometries), tetrahedra-only computational elements (to enable automatic mesh generation), edge-based finite element scheme (to reduce indirect addressing for increased performance), distributed-memory parallel computing paradigm (to enable large problems), and Charm++ (https://charmplusplus.org) as the runtime system (to effectively use computing resources even in the presence of hardware heterogeneities and dynamic application requirements). We discuss aspects of the implementation that enable exercising unique features of the runtime system, e.g., the single Charm++ programming abstraction, overdecomposition, asynchronous execution, latency-hiding parallel communication and computation, task-parallelism, and dynamic load balancing via object migratability. Multiple test problems are used to verify and validate the numerical solutions and computational performance and scalability to high-performance computing environments are discussed. For maximum transparency and reproducibility, and to encourage future research, development, and use, the full source code, together with regression tests and documentation, is publicly available at https://xyst.cc.

Cerebrospinal Fluid Metabolomics Identified Ongoing Analgesic Medication in Neuropathic Pain Patients.

Cerebrospinal fluid (CSF) can reasonably be hypothesized to mirror central nervous system pathophysiology in chronic pain conditions. Metabolites are small organic molecules with a low molecular weight. They are the downstream products of genes, transcripts and enzyme functions, and their levels can mirror diseased metabolic pathways. The aim of this metabolomic study was to compare the CSF of patients with chronic neuropathic pain (n = 16) to healthy controls (n = 12). Nuclear magnetic resonance spectroscopy was used for analysis of the CSF metabolome. Multivariate data analysis by projection discriminant analysis (OPLS-DA) was used to separate information from noise and minimize the multiple testing problem. The significant OPLS-DA model identified 26 features out of 215 as important for group separation (R2 = 0.70, Q2 = 0.42, p = 0.017 by CV-ANOVA; 2 components). Twenty-one out of twenty-six features were statistically significant when comparing the two groups by univariate statistics and remained significant at a false discovery rate of 10%. For six out of the top ten metabolite features, the features were absent in all healthy controls. However, these features were related to medication, mainly acetaminophen (=paracetamol), and not to pathophysiological processes. CSF metabolomics was a sensitive method to detect ongoing analgesic medication, especially acetaminophen.

Detection of outlying patterns from sparse and irregularly sampled electronic health records data

Within the intensive care unit (ICU), vital signs such as arterial blood pressure (ABP) collected from electronic health records (EHRs) are typically recorded at different and uneven sampling frequencies and are often infrequently measured due to the nature of the medical treatment. Furthermore, from a temporal trajectory perspective, EHR data are likely to be corrupted by outlying patterns that deviate from normal samples in terms of the curves’ magnitude and shape. In this work, we propose a two-stage outlier detection approach for sparse and irregularly sampled (SiS) temporal data using functional data analysis (FDA) tools. In the first stage, an outlier identification measure is defined by a max–min statistic and a clean subset that contains nonoutliers. In the second stage, a multiple hypothesis testing problem is formulated based on the asymptotic distribution of the proposed measure. The simulation-based framework shows that the proposed method is robust to different types of shape and magnitude outliers. The detection results are more accurate than the widely used functional depth methods, especially in extremely sparse settings where the proportion of the observed data points over the entire time series is approximately 10%. Extensive experiments are also conducted on the real-world MIMIC-II dataset, which demonstrate that the method effectively detects clinically meaningful outlying patterns.

Comparison of regmed and BayesNetty for exploring causal models with many variables.

Here we compare a recently proposed method and software package, regmed, with our own previously developed package, BayesNetty, designed to allow exploratory analysis of complex causal relationships between biological variables. We find that regmed generally has poorer recall but much better precision than BayesNetty. This is perhaps not too surprising as regmed is specifically designed for use with high-dimensional data. BayesNetty is found to be more sensitive to the resulting multiple testing problem encountered in these circumstances. However, as regmed is not designed to handle missing data, its performance is severely affected when missing data is present, whereas the performance of BayesNetty is only slightly affected. The performance of regmed can be rescued in this situation by first using BayesNetty to impute the missing data, and then applying regmed to the resulting "filled-in" data set.

Stock return anomalies identification during the Covid-19 with the application of a grouped multiple comparison procedure

This study investigates the impact of COVID-19 pandemic on the Chinese stock market in 2020. Using daily data of three industries, this study addresses the identification of abnormal stock returns as a multiple hypothesis testing problem and proposes to apply a grouped comparison procedure for better detection. By comparing the numbers of daily signals and numbers of stocks with abnormal positive and negative returns, the empirical result shows that the three industries perform differently under the pandemic. Compared to the non-grouped testing procedure, the signals found by the grouped procedure are more prominent, which is advantageous for some situations when there tends to be abnormal performance clustering at the occurrence of major event. This paper on stock return anomalies gives a new perspective on the impact of major events to the stock market, like the global outbreak disease.

Reusing Natural Experiments

ABSTRACTAfter a natural experiment is first used, other researchers often reuse the setting, examining different outcome variables. We use simulations based on real data to illustrate the multiple hypothesis testing problem that arises when researchers reuse natural experiments. We then provide guidance for future inference based on popular empirical settings including difference‐in‐differences, instrumental variables, and regression discontinuity designs. When we apply our guidance to two extensively studied natural experiments, business combination laws and the Regulation SHO pilot, we find that many results that were statistically significant using single hypothesis testing do not survive corrections for multiple hypothesis testing.

NetworkGWAS: a network-based approach to discover genetic associations.

While the search for associations between genetic markers and complex traits has led to the discovery of tens of thousands of trait-related genetic variants, the vast majority of these only explain a small fraction of the observed phenotypic variation. One possible strategy to overcome this while leveraging biological prior is to aggregate the effects of several genetic markers and to test entire genes, pathways or (sub)networks of genes for association to a phenotype. The latter, network-based genome-wide association studies, in particular suffer from a vast search space and an inherent multiple testing problem. As a consequence, current approaches are either based on greedy feature selection, thereby risking that they miss relevant associations, or neglect doing a multiple testing correction, which can lead to an abundance of false positive findings. To address the shortcomings of current approaches of network-based genome-wide association studies, we propose networkGWAS, a computationally efficient and statistically sound approach to network-based genome-wide association studies using mixed models and neighborhood aggregation. It allows for population structure correction and for well-calibrated P-values, which are obtained through circular and degree-preserving network permutations. networkGWAS successfully detects known associations on diverse synthetic phenotypes, as well as known and novel genes in phenotypes from Saccharomycescerevisiae and Homo sapiens. It thereby enables the systematic combination of gene-based genome-wide association studies with biological network information. https://github.com/BorgwardtLab/networkGWAS.git.

Numerical solution of delay Volterra functional integral equations with variable bounds

This work presents a new numerical method for solving Volterra-type integro functional equations with variable bounds and mixed delay. This paper applies discrete orthogonal Hahn polynomials and their properties numerically. A discrete scalar product is associated with discrete orthogonal polynomials. Several numerical experiments (including linear and nonlinear) for multiple test problems are provided to validate the accuracy of this method.