Bayesian Hypothesis Testing Research Articles

Towards improving the predictive capability of computer simulations by integrating inverse Uncertainty Quantification and quantitative validation with Bayesian hypothesis testing

The Best Estimate plus Uncertainty (BEPU) approach for nuclear systems modeling and simulation requires that the prediction uncertainty must be quantified in order to prove that the investigated design stays within acceptance criteria. A rigorous Uncertainty Quantification (UQ) process should simultaneously consider multiple sources of quantifiable uncertainties: (1) parameter uncertainty due to randomness or lack of knowledge; (2) experimental uncertainty due to measurement noise; (3) model uncertainty caused by missing/incomplete physics and numerical approximation errors, and (4) code uncertainty when surrogate models are used. In this paper, we propose a comprehensive framework to integrate results from inverse UQ and quantitative validation to provide robust predictions so that all these sources of uncertainties can be taken into consideration.Inverse UQ quantifies the parameter uncertainties based on experimental data while taking into account uncertainties from model, code and measurement. In the validation step, we use a quantitative validation metric based on Bayesian hypothesis testing. The resulting metric, called the Bayes factor, is then used to form weighting factors to combine the prior and posterior knowledge of the parameter uncertainties in a Bayesian model averaging process. In this way, model predictions will be able to integrate the results from inverse UQ and validation to account for all available sources of uncertainties. This framework is a step towards addressing the ANS Nuclear Grand Challenge on “Simulation/Experimentation” by bridging the gap between models and data.

How to Choose between Different Bayesian Posterior Indices for Hypothesis Testing in Practice

Hypothesis testing is an essential statistical method in experimental psychology and the cognitive sciences. The problems of traditional null hypothesis significance testing (NHST) have been discussed widely, and among the proposed solutions to the replication problems caused by the inappropriate use of significance tests and p-values is a shift toward Bayesian data analysis. However, Bayesian hypothesis testing is concerned with various posterior indices for significance and the size of an effect. This complicates Bayesian hypothesis testing in practice, as the availability of multiple Bayesian alternatives to the traditional p-value causes confusion which one to select and why. In this paper, various Bayesian posterior indices which have been proposed in the literature are compared and their benefits and limitations are discussed. The comparison shows that conceptually not all proposed Bayesian alternatives to NHST and p-values are beneficial, and the usefulness of some indices strongly depends on the study design and research goal. However, the comparison also reveals that there exist at least two candidates among the available Bayesian posterior indices which have appealing theoretical properties and are widely underused in the cognitive sciences.

Kenneth (Ken) Ronald Walter Brewer (1931–2021)

Kenneth (Ken) Ronald Walter Brewer (1931–2021)

Power analysis and type I and type II error rates of Bayesian nonparametric two-sample tests for location-shifts based on the Bayes factor under Cauchy priors

Hypothesis testing is a central statistical method in the biomedical sciences. The ongoing debate about the concept of statistical significance and the reliability of null hypothesis significance tests (NHST) and p-values has brought the advent of various Bayesian hypothesis tests as possible alternatives, which often employ the Bayes factor. However, careful calibration of the prior parameters is necessary for the type I error rates or power of these alternatives to be any better. Also, the availability of various Bayesian tests for the same statistical problem leads to the question which test to choose based on which criteria. Recently proposed Bayesian nonparametric two-sample tests are analyzed with regard to their type I error rates and power to detect an effect. Results show that approaches vary substantially in their ability to control the type I and II errors, and it is shown how to select the prior parameters to balance power and type I error control. This allows for prior elicitation and power analyses based on objective criteria like type I and II error rates even when conducting a Bayesian nonparametric two-sample test. Also, it is shown that existing nonparametric Bayesian two-sample tests are adequate only to test for location-shifts. Together, the results provide guidance how to perform a nonparametric Bayesian two-sample test while simultaneously improving the reliability of research.

Rethinking remdesivir for COVID-19: A Bayesian reanalysis of trial findings.

Following testing in clinical trials, the use of remdesivir for treatment of COVID-19 has been authorized for use in parts of the world, including the USA and Europe. Early authorizations were largely based on results from two clinical trials. A third study published by Wang et al. was underpowered and deemed inconclusive. Although regulators have shown an interest in interpreting the Wang et al. study, under a frequentist framework it is difficult to determine if the non-significant finding was caused by a lack of power or by the absence of an effect. Bayesian hypothesis testing does allow for quantification of evidence in favor of the absence of an effect. Results of our Bayesian reanalysis of the three trials show ambiguous evidence for the primary outcome of clinical improvement and moderate evidence against the secondary outcome of decreased mortality rate. Additional analyses of three studies published after initial marketing approval support these findings. We recommend that regulatory bodies take all available evidence into account for endorsement decisions. A Bayesian approach can be beneficial, in particular in case of statistically non-significant results. This is especially pressing when limited clinical efficacy data is available.

Worked-out examples of the adequacy of Bayesian optional stopping.

The practice of sequentially testing a null hypothesis as data are collected until the null hypothesis is rejected is known as optional stopping. It is well known that optional stopping is problematic in the context of p value-based null hypothesis significance testing: The false-positive rates quickly overcome the single test's significance level. However, the state of affairs under null hypothesis Bayesian testing, where p values are replaced by Bayes factors, has perhaps surprisingly been much less consensual. Rouder (2014) used simulations to defend the use of optional stopping under null hypothesis Bayesian testing. The idea behind these simulations is closely related to the idea of sampling from prior predictive distributions. Deng et al. (2016) and Hendriksen et al. (2020) have provided mathematical evidence to the effect that optional stopping under null hypothesis Bayesian testing does hold under some conditions. These papers are, however, exceedingly technical for most researchers in the applied social sciences. In this paper, we provide some mathematical derivations concerning Rouder's approximate simulation results for the two Bayesian hypothesis tests that he considered. The key idea is to consider the probability distribution of the Bayes factor, which is regarded as being a random variable across repeated sampling. This paper therefore offers an intuitive perspective to the literature and we believe it is a valid contribution towards understanding the practice of optional stopping in the context of Bayesian hypothesis testing.

A Note on Hypothesis Testing with Random Sample Sizes and its Relationship to Bayes Factors

Frequentist and Bayesian hypothesis testing are often viewed as “two separate worlds” by practitioners. While theoretical relationships of course exist, our goal here is to demonstrate a practical example where one must be careful conducting frequentist hypothesis testing, and in that context illustrate a practical equivalence between Bayesian and frequentist testing. In particular, if the sample size is random (hardly unusual in prac tical problems where the sample size may be “all available experimental units”), then choosing an α level in advance such as 0.05 and using it for every possible sample size is inadmissible. In other words, one can find a dif ferent overall procedure which has the same overall type I error but greater power. Not coincidentally, this alternative procedure is based on Bayesian testing procedures.

A Decision-Making Method for Machinery Abnormalities Based on Neural Network Prediction and Bayesian Hypothesis Testing

For anomaly identification of predicted data in machinery condition monitoring, traditional threshold methods have problems during residual testing. It is difficult to make decisions when the residuals are close to the threshold and fluctuate. This paper proposes a Bayesian dynamic thresholding method that combines Bayesian inference with neural network signal prediction. The method makes full use of historical prior data to build an anomaly identification and warning model applicable under single variable or multidimensional variables. A long short-term memory signal prediction model is established, and then a Bayesian hypothesis testing-based anomaly identification strategy is presented to quantify the probability of anomaly occurrence and issue early warnings for anomalies beyond a certain probability. The model was applied to open data sets of a pumping station and actual operating data of a nuclear power turbine. The results indicate that the model successfully predicts the failure probability and failure time. The effectiveness of the proposed method is verified.

Child reward neurocircuitry and parental substance use history: Findings from the Adolescent Brain Cognitive Development Study

BackgroundSubstance use research has focused on family history of alcohol use disorders but less on other addictions in biological family members. We examined how parental substance use history relates to reward system functioning, specifically nucleus accumbens (NAcc) and putamen activation at age 9–10 in the Adolescent Brain Cognitive Development (ABCD) Study. This research hopes to address limitations in prior literature by focusing analyses on a large, substance-naïve sample. MethodWe included ABCD participants with valid Monetary Incentive Delay task fMRI Baseline data and parent substance use history at project baseline from Data Release 2.0 (N = 10,622). Parent-history-positive (PH+) participants had one or both biological parents with a history of two+problems with alcohol (n = 741; PH+A) and/or other drugs (n = 638; PH+D). Of participants who were parent-history-negative (PH-) for alcohol and/or drugs, a stratified random sample based on six sociodemographic variables was created and matched to the PH+group (PH-A n = 699; PH-D n = 615). The contrast of interest was anticipation of a large reward vs. neutral response. ResultsPH+A youth had more activation in the right NAcc during large reward anticipation than PH-A. PH+D youth showed enhanced left putamen activation during large reward anticipation than PH-D youth. Bayesian hypothesis testing showed moderate evidence (BF > 3) in favor of the null hypothesis. ConclusionThese findings suggest that pre-adolescents whose biological parents had a history of substance-related problems show small differences in reward processing compared to their PH- peers.

A Critical Evaluation of the FBST ev for Bayesian Hypothesis Testing

The “Full Bayesian Significance Test e-value”, henceforth FBST ev, has received increasing attention across a range of disciplines including psychology. We show that the FBST ev leads to four problems: (1) the FBST ev cannot quantify evidence in favor of a null hypothesis and therefore also cannot discriminate “evidence of absence” from “absence of evidence”; (2) the FBST ev is susceptible to sampling to a foregone conclusion; (3) the FBST ev violates the principle of predictive irrelevance, such that it is affected by data that are equally likely to occur under the null hypothesis and the alternative hypothesis; (4) the FBST ev suffers from the Jeffreys-Lindley paradox in that it does not include a correction for selection. These problems also plague the frequentist p-value. We conclude that although the FBST ev may be an improvement over the p-value, it does not provide a reasonable measure of evidence against the null hypothesis.

The Pearson Bayes factor: An analytic formula for computing evidential value from minimal summary statistics

Summary In Bayesian hypothesis testing, evidence for a statistical model is quantified by the Bayes factor, which represents the relative likelihood of observed data under that model compared to another competing model. In general, computing Bayes factors is difficult, as computing the marginal likelihood of data under a given model requires integrating over a prior distribution of model parameters. In this paper, I capitalize on a particular choice of prior distribution that allows the Bayes factor to be expressed without integral representation, and I develop a simple formula – the Pearson Bayes factor – that requires only minimal summary statistics as commonly reported in scientific papers, such as the t or F score and the degrees of freedom. In addition to presenting this new result, I provide several examples of its use and report a simulation study validating its performance. Importantly, the Pearson Bayes factor gives applied researchers the ability to compute exact Bayes factors from minimal summary data, and thus easily assess the evidential value of any data for which these summary statistics are provided, even when the original data is not available.

The use of local and nonlocal priors in Bayesian test‐based monitoring for single‐arm phase II clinical trials

Bayesian sequential monitoring is widely used in adaptive phase II studies where the objective is to examine whether an experimental drug is efficacious. Common approaches for Bayesian sequential monitoring are based on posterior or predictive probabilities and Bayesian hypothesis testing procedures using Bayes factors. In the first part of the paper, we briefly show the connections between test-based (TB) and posterior probability-based (PB) sequential monitoring approaches. Next, we extensively investigate the choice of local and nonlocal priors for the TB monitoring procedure. We describe the pros and cons of different priors in terms of operating characteristics. We also develop a user-friendly Shiny application to implement the TB design.

Signal anomaly identification strategy based on Bayesian inference for nuclear power machinery

In the machinery industry, signal anomalies are generally identified using the threshold method, which exhibits shortcomings in setting reasonable thresholds, in decision-making when signals approach thresholds or fluctuate, and in quantification of fault confidence. In this paper, a long short-term memory (LSTM) model is established to predict the time-series signals. For prediction residual, a novel decision-making strategy of signal anomaly identification based on Bayesian inference is then proposed that considers data uncertainty. Various signal abnormality conditions are analyzed, and a Bayesian hypothesis test approach is developed to determine the signal status and quantify the fault probability. After fully mining the prior information of the residuals to reduce the influence of randomness, estimates of the key parameters, namely residual mean and variance, are determined by obtaining the posterior distribution based on the normal-inverse-gamma distribution. In two nuclear power machinery examples, all potential signal anomalies are identified by the proposed method. The results of a comparative analysis with existing methods demonstrate that the proposed method can issue an alarm several hours in advance and provide a fault probability, which improves the accuracy and reliability of prediction.

Seabird bycatch vulnerability in pelagic longline fisheries based on modelling of a long-term dataset

SummaryThe incidental mortality of seabirds in fisheries remains a serious global concern. Obtaining unbiased and accurate estimates of bycatch rates is a priority for seabird bycatch mitigation and demographic research. For measuring the capture risk of seabird interactions in fisheries, the rate of carcass retrieval from hauled gear is commonly used. However, reliability can be limited by a lack of direct capture observations and the substantial pre-haul bycatch losses known to occur, meaning incidence of seabird bycatch is underestimated. To solve this problem, a new measure (bycatch vulnerability) that links an observed interaction directly to the underlying capture event is proposed to represent the capture risk of fishery interactions by seabirds. The new measure is not affected by subsequent bycatch loss. To illustrate how to estimate and analyse bycatch vulnerability, a case study based on a long-term dataset of seabird interactions and capture confirmation is provided. Bayesian modelling and hypothesis testing were conducted to identify important bycatch risk factors. Competition was found to play a central role in determining seabird bycatch vulnerability. More competitive environments were riskier for seabirds, and larger and thus more competitive species were more at risk than smaller sized and less competitive species. Species foraging behaviour also played a role. On the other hand, no additional effect of physical oceanic condition and spatio-temporal factors on bycatch vulnerability was detected. Bycatch vulnerability is recommended as a replacement for the commonly used bycatch rate or carcass retrieval rate to measure the capture risk of an interaction. Combined with a normalized contact rate, bycatch vulnerability offers an unbiased estimate of seabird bycatch rate in pelagic longline fisheries.

A Bayesian test for the 4.2 ka BP abrupt climatic change event in southeast Europe and southwest Asia using structural time series analysis of paleoclimate data

It has been proposed that there was an abrupt climatic change event around 4.2 ka BP that affected societies and even has been linked to the collapse of empires. Subsequent studies have reached conclusions that both support and contradict the proposed event; yet nevertheless, 4.2 ka BP has now been adopted as the stratigraphic boundary point between the Middle and Upper Holocene. Time series plots of paleoclimate studies that claim to support the abrupt climate change hypothesis show differing temporal patterns so, in this study, we apply the Bayesian structural time series (BSTS) approach using the CausalImpact package to test data from southeast Europe and southwest Asia for which it is claimed that they demonstrate a climatic anomaly around 4.2 ka BP. To do this, each “affected” time series is synthetically reconstructed using “unaffected” series as predictors in a fully Bayesian framework by the BSTS method and then forecast beyond the assumed starting point of the event. A Bayesian hypothesis test is then applied to differences between each synthetic and real time series to test the impact of the event against the forecast data. While our results confirm that some studies cited in support of the 4.2 ka BP event hypothesis do indeed hold true, we also show that a number of other studies fail to demonstrate any credible effect. We observe spatial and data patterning in our results, and we speculate that this climatic deterioration may have been a consequence of an asymmetrical northward expansion or migration of the Northern Hemisphere Hadley cell. Furthermore, we observe that while the signals are generally not credible, types of proxy data from the Mesopotamia region and east are consistent with aeolian dust storms.

How to use and report Bayesian hypothesis tests.

This article provides guidance on interpreting and reporting Bayesian hypothesis tests, to aid their understanding. To use and report a Bayesian hypothesis test, predicted effect sizes must be specified. The article will provide guidance in specifying effect sizes of interest (which also will be of relevance to those using frequentist statistics). First, if a minimally interesting effect size can be specified, a null interval is defined as the effects smaller in magnitude than the minimally interesting effect. Then the proportion of the posterior distribution that falls in the null interval indicates the plausibility of the null interval hypothesis. Second, if a rough scale of effect can be determined, a Bayes factor can indicate evidence for a model representing that scale of effect versus a model of the null hypothesis. Both methods allow data to count against a theory that predicts a difference. By contrast, nonsignificance does not count against such a theory. Various examples are provided including the suitability of Bayesian analyses for demonstrating the absence of conscious perception under putative subliminal conditions, and its presence in supraliminal conditions.

Distinguishing W′ signals at hadron colliders using neural networks

We investigate a neural network-based hypothesis test to distinguish different $W'$ and charged scalar resonances through the $\ell+\require{cancel}\cancel{E}_T$ channel at hadron colliders. This is traditionally challenging due to a four-fold ambiguity at proton-proton colliders, such as the Large Hadron Collider. Of the neural network approaches we studied, we find a multi-class classifier based on a fully-connected neural network trained upon 2D histograms made from kinematic variables of the final state $\ell$ to be the most powerful. Furthermore, by considering the 1-jet processes, we demonstrate that one can generalize to multiple $2D$ histograms to represent different variable pairs. Finally, as a comparison to traditional approaches, we compare our method with Bayesian hypothesis testing and discuss the pros and cons of each approach. The neural network scheme presented in this paper is a powerful tool that can help probe the properties of charged resonances.

Beneath the surface: Unearthing within-person variability and mean relations with Bayesian mixed models.

Mixed-effects models are becoming common in psychological science. Although they have many desirable features, there is still untapped potential. It is customary to view homogeneous variance as an assumption to satisfy. We argue to move beyond that perspective, and to view modeling within-person variance as an opportunity to gain a richer understanding of psychological processes. The technique to do so is based on the mixed-effects location scale model that can simultaneously estimate mixed-effects submodels to both the mean (location) and within-person variance (scale). We develop a framework that goes beyond assessing the submodels in isolation of one another and introduce a novel Bayesian hypothesis test for mean-variance correlations in the distribution of random effects. We first present a motivating example, which makes clear how the model can characterize mean-variance relations. We then apply the method to reaction times (RTs) gathered from 2 cognitive inhibition tasks. We find there are more individual differences in the within-person variance than the mean structure, as well as a complex web of structural mean-variance relations. This stands in contrast to the dominant view of within-person variance (i.e., "noise"). The results also point toward paradoxical within-person, as opposed to between-person, effects: several people had slower and less variable incongruent responses. This contradicts the typical pattern, wherein larger means tend to be associated with more variability. We conclude with future directions, spanning from methodological to theoretical inquires, that can be answered with the presented methodology. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Attack Detection Method based on Bayesian Hypothesis Testing Principle in CPS

Cyber physical system (CPS) introduces the concepts and methods of communication networks into traditional industrial processes, and realizes the control of physical processes by information flow, which is a key issue in national defense, military and industrial production. Many application problems provide solutions. However, this also makes the original pure industrial system environment face more security risks. Considering the security of the CPS control layer, based on the principle of Bayesian hypothesis testing, a detection method for the tampering of the measurement data of the control layer is proposed. This method uses the prior knowledge of the parameters to make the model still usable under the condition of a small sample amount of data. At the same time, the judgment conclusions made can accurately give the probability value of the attack behavior, which can more intuitively explain the possibility of the current statement. Compared with the previous traditional hypothesis testing method, this method is in line with the background of the CPS attack, so it has practical advantages.

BFpack: Flexible Bayes Factor Testing of Scientific Theories in R

There have been considerable methodological developments of Bayes factors for hypothesis testing in the social and behavioral sciences, and related fields. This development is due to the flexibility of the Bayes factor for testing multiple hypotheses simultaneously, the ability to test complex hypotheses involving equality as well as order constraints on the parameters of interest, and the interpretability of the outcome as the weight of evidence provided by the data in support of competing scientific theories. The available software tools for Bayesian hypothesis testing are still limited however. In this paper we present a new R package called BFpack that contains functions for Bayes factor hypothesis testing for the many common testing problems. The software includes novel tools for (i) Bayesian exploratory testing (e.g., zero vs positive vs negative effects), (ii) Bayesian confirmatory testing (competing hypotheses with equality and/or order constraints), (iii) common statistical analyses, such as linear regression, generalized linear models, (multivariate) analysis of (co)variance, correlation analysis, and random intercept models, (iv) using default priors, and (v) while allowing data to contain missing observations that are missing at random.