Bayesian Significance Test Research Articles

Full Bayesian Significance Testing for Neural Networks

Significance testing aims to determine whether a proposition about the population distribution is the truth or not given observations. However, traditional significance testing often needs to derive the distribution of the testing statistic, failing to deal with complex nonlinear relationships. In this paper, we propose to conduct Full Bayesian Significance Testing for neural networks, called nFBST, to overcome the limitation in relationship characterization of traditional approaches. A Bayesian neural network is utilized to fit the nonlinear and multi-dimensional relationships with small errors and avoid hard theoretical derivation by computing the evidence value. Besides, nFBST can test not only global significance but also local and instance-wise significance, which previous testing methods don't focus on. Moreover, nFBST is a general framework that can be extended based on the measures selected, such as Grad-nFBST, LRP-nFBST, DeepLIFT-nFBST, LIME-nFBST. A range of experiments on both simulated and real data are conducted to show the advantages of our method.

A Bayesian robustness measure in significance tests for equivalence tests

In this article, the local sensitivity of non linear prior quantities in Bayesian significance tests with respect to the choice of a prior distribution is considered. We propose sensitivity indices using the Gâteaux derivative to evaluate the rate of change of statistical functionals defined over the space of prior probability measures. These sensitivity indices are easy to interpret and calculate. We apply the proposed methodology to equivalence tests for two independent binomial proportions to quantify the local sensitivity of quantities in significance tests, such as adaptive significance level and power, with respect to the choice of the prior distribution. We present an application in sensory analysis and consumer research to illustrate the proposed methodology.

A new Bayesian discrepancy measure

The aim of this article is to make a contribution to the Bayesian procedure of testing precise hypotheses for parametric models. For this purpose, we define the Bayesian Discrepancy Measure that allows one to evaluate the suitability of a given hypothesis with respect to the available information (prior law and data). To summarise this information, the posterior median is employed, allowing a simple assessment of the discrepancy with a fixed hypothesis. The Bayesian Discrepancy Measure assesses the compatibility of a single hypothesis with the observed data, as opposed to the more common comparative approach where a hypothesis is rejected in favour of a competing hypothesis. The proposed measure of evidence has properties of consistency and invariance. After presenting the definition of the measure for a parameter of interest, both in the absence and in the presence of nuisance parameters, we illustrate some examples showing its conceptual and interpretative simplicity. Finally, we compare a test procedure based on the Bayesian Discrepancy Measure, with the Full Bayesian Significance Test, a well-known Bayesian testing procedure for sharp hypotheses.

Social media consumption and depressive symptoms during the COVID-19 lockdown: the mediating effect of physical activity.

Social media platforms played a critical role during the COVID-19 pandemic. This study aimed to explore: (1) the changes in social media consumption patterns, physical activity levels/sedentary behavior, and depressive symptoms, and (2) how the changes in social media consumption patterns predict the changes in depressive symptoms while investigating the mediating role of changes in physical activity levels/sedentary behavior between before, and after the COVID-19 lockdown among U.S. adults with different age clusters. A total of 695 U.S. participants completed an online questionnaire via MTurk, and participants were asked to recall their social media consumption patterns, physical activity/sedentary behavior, depressive symptoms in January and May of 2020 while covariates included non-physical activity health behavior including diet quality, alcohol consumption, smoking, and sleep quality. The results of Bayesian significance testing of changes showed that the older participants tended to spend more time with content-focused social media platforms during the lockdown. While significantly increased sitting time was reported by all age clusters, no significant changes were found in activity levels. Additionally, the middle-aged and older participants reported significantly higher depressive symptoms. The findings of a multigroup structural analysis showed the significant mediating effect of moderate-to-vigorous physical activity on the relationship between changes in social media consumption and depressive symptoms. This study highlights the need for targeting specific social media platforms for older adults and the importance of moderate-to-vigorous physical activity to alleviate the mental health issues resulting from social media consumption. The result of this study also highlights the need for sport-based intervention programs in the future and the need for more social media campaigns at the institution/organization levels established by public health stakeholders and policy makers to promote physical activity and maximize population perception and reach during the pandemic.

Adaptive Significance Levels in Tests for Linear Regression Models: The e-Value and P-Value Cases

The full Bayesian significance test (FBST) for precise hypotheses is a Bayesian alternative to the traditional significance tests based on p-values. The FBST is characterized by the e-value as an evidence index in favor of the null hypothesis (H). An important practical issue for the implementation of the FBST is to establish how small the evidence against H must be in order to decide for its rejection. In this work, we present a method to find a cutoff value for the e-value in the FBST by minimizing the linear combination of the averaged type-I and type-II error probabilities for a given sample size and also for a given dimensionality of the parameter space. Furthermore, we compare our methodology with the results obtained from the test with adaptive significance level, which presents the capital-P P-value as a decision-making evidence measure. For this purpose, the scenario of linear regression models with unknown variance under the Bayesian approach is considered.

Measure-Theoretic Musings Cannot Salvage the Full Bayesian Significance Test as a Measure of Evidence

Ly and Wagenmakers (Computational Brain & Behavior:1–8, in press) critiqued the Full Bayesian Significance Test (FBST) and the associated statistic FBST ev: similar to the frequentist p-value, FBST ev cannot quantify evidence for the null hypothesis, allows sampling to a foregone conclusion, and suffers from the Jeffreys-Lindley paradox. In response, Kelter (Computational Brain & Behavior:1–11, 2022) suggested that the critique is based on a measure-theoretic premise that is often inappropriate in practice, namely the assignment of non-zero prior mass to a point-null hypothesis. Here we argue that the key aspects of our initial critique remain intact when the point-null hypothesis is replaced either by a peri-null hypothesis or by an interval-null hypothesis; hence, the discussion on the validity of a point-null hypothesis is a red herring. We suggest that it is tempting yet fallacious to test a hypothesis by estimating a parameter that is part of a different model. By rejecting any null hypothesis before it is tested, FBST is begging the question. Although FBST may be useful as a measure of surprise under a single model, we believe that the concept of evidence is inherently relative; consequently, evidence for competing hypotheses ought to be quantified by examining the relative adequacy of their predictions. This philosophy is fundamentally at odds with the FBST.

The evidence interval and the Bayesian evidence value: On a unified theory for Bayesian hypothesis testing and interval estimation.

Interval estimation is one of the most frequently used methods in statistical science, employed to provide a range of credible values a parameter is located in after taking into account the uncertainty in the data. However, while this interpretation only holds for Bayesian interval estimates, these suffer from two problems. First, Bayesian interval estimates can include values which have not been corroborated by observing the data. Second, Bayesian interval estimates and hypothesis tests can yield contradictory conclusions. In this paper a new theory for Bayesian hypothesis testing and interval estimation is presented. A new interval estimate is proposed, the Bayesian evidence interval, which is inspired by the Pereira-Stern theory of the full Bayesian significance test (FBST). It is shown that the evidence interval is a generalization of existing Bayesian interval estimates, that it solves the problems of standard Bayesian interval estimates and that it unifies Bayesian hypothesis testing and parameter estimation. The Bayesian evidence value is introduced, which quantifies the evidence for the (interval) null and alternative hypothesis. Based on the evidence interval and the evidence value, the (full) Bayesian evidence test (FBET) is proposed as a new, model-independent Bayesian hypothesis test. Additionally, a decision rule for hypothesis testing is derived which shows the relationship to a widely used decision rule based on the region of practical equivalence and Bayesian highest posterior density intervals and to the e-value in the FBST. In summary, the proposed method is universally applicable, computationally efficient, and while the evidence interval can be seen as an extension of existing Bayesian interval estimates, the FBET is a generalization of the FBST and contains it as a special case. Together, the theory developed provides a unification of Bayesian hypothesis testing and interval estimation and is made available in the R package fbst.

Changes in Physical Activity and Depressive Symptoms During COVID-19 Lockdown: United States Adult Age Groups.

This study investigates: (1) the changes in three major health-related factors—physical activity, non-physical-activity health behavior (i.e., diet quality, alcohol consumption, smoking, sleep quality), and depressive symptoms, and (2) how changes in physical activity were associated with changes in one’s depressive symptoms among young adults, middle-aged adults, and older adults while controlling non-physical-activity health behavior and sociodemographic characteristics among young, middle-aged, and older adults before and after the COVID-19 outbreak lockdown in the United States. A total of 695 participants completed an online questionnaire via MTurk, and participants were asked to recall their physical activity, depressive symptoms, and non-physical-activity health behavior status in January and May of 2020. The IPAQ-SF was used to evaluate individuals’ physical activity, while the CES-D-10 was used to assess depressive symptoms. Covariates included non-physical-activity health behavior and sociodemographic factors. A Bayesian significance testing of changes was used to examine significant changes in physical activity, non-physical-activity behavior, and depressive symptoms in each age group while Bayesian regression analysis was employed to examine how the changes in physical activity were associated with respondents’ depressive symptoms while controlling for individual NHB and sociodemographic characteristics. The results showed that the participants tended to maintain their physical activity levels after the lockdown despite significant increases in sitting time among young and older adults. Decreases in moderate physical activity frequency were associated with a higher level of depressive symptoms (R2 = 17.1%). Although young and middle-aged cohorts experienced fewer differences in depressive symptoms compared to their counterparts in the older group, we found no significant heterogeneity effects in the relationships of interest across all age groups. Considering different influences of physical activity on depressive symptoms depending on different levels of activity and ages, more randomized clinical trials with program-based intervention studies should be conducted with different physical activity programs for different age populations.

Fbst: An R package for the Full Bayesian Significance Test for testing a sharp null hypothesis against its alternative via the e value

Hypothesis testing is a central statistical method in psychology and the cognitive sciences. However, the problems of null hypothesis significance testing (NHST) and p values have been debated widely, but few attractive alternatives exist. This article introduces the fbst R package, which implements the Full Bayesian Significance Test (FBST) to test a sharp null hypothesis against its alternative via the e value. The statistical theory of the FBST has been introduced more than two decades ago and since then the FBST has shown to be a Bayesian alternative to NHST and p values with both theoretical and practical highly appealing properties. The algorithm provided in the fbst package is applicable to any Bayesian model as long as the posterior distribution can be obtained at least numerically. The core function of the package provides the Bayesian evidence against the null hypothesis, the e value. Additionally, p values based on asymptotic arguments can be computed and rich visualizations for communication and interpretation of the results can be produced. Three examples of frequently used statistical procedures in the cognitive sciences are given in this paper, which demonstrate how to apply the FBST in practice using the fbst package. Based on the success of the FBST in statistical science, the fbst package should be of interest to a broad range of researchers and hopefully will encourage researchers to consider the FBST as a possible alternative when conducting hypothesis tests of a sharp null hypothesis.

On the Measure-Theoretic Premises of Bayes Factor and Full Bayesian Significance Tests: a Critical Reevaluation

The Full Bayesian Significance Test (FBST) and the Bayesian evidence value recently have received increasing attention across a variety of sciences including psychology. Ly and Wagenmakers (2021) have provided a critical evaluation of the method and concluded that it suffers from four problems which are mostly attributed to the asymptotic relationship of the Bayesian evidence value to the frequentist p-value. While Ly and Wagenmakers (2021) tackle an important question about the best way of statistical hypothesis testing in the cognitive sciences, it is shown in this paper that their arguments are based on a specific measure-theoretic premise. The identified problems hold only under a specific class of prior distributions which are required only when adopting a Bayes factor test. However, the FBST explicitly avoids this premise, which resolves the problems in practical data analysis. In summary, the analysis leads to the more important question whether precise point null hypotheses are realistic for scientific research, and a shift towards the Hodges-Lehmann paradigm may be an appealing solution when there is doubt on the appropriateness of a precise hypothesis.

A Critical Evaluation of the FBST ev for Bayesian Hypothesis Testing

The “Full Bayesian Significance Test e-value”, henceforth FBST ev, has received increasing attention across a range of disciplines including psychology. We show that the FBST ev leads to four problems: (1) the FBST ev cannot quantify evidence in favor of a null hypothesis and therefore also cannot discriminate “evidence of absence” from “absence of evidence”; (2) the FBST ev is susceptible to sampling to a foregone conclusion; (3) the FBST ev violates the principle of predictive irrelevance, such that it is affected by data that are equally likely to occur under the null hypothesis and the alternative hypothesis; (4) the FBST ev suffers from the Jeffreys-Lindley paradox in that it does not include a correction for selection. These problems also plague the frequentist p-value. We conclude that although the FBST ev may be an improvement over the p-value, it does not provide a reasonable measure of evidence against the null hypothesis.

Cointegration and Unit Root Tests: A Fully Bayesian Approach.

To perform statistical inference for time series, one should be able to assess if they present deterministic or stochastic trends. For univariate analysis, one way to detect stochastic trends is to test if the series has unit roots, and for multivariate studies it is often relevant to search for stationary linear relationships between the series, or if they cointegrate. The main goal of this article is to briefly review the shortcomings of unit root and cointegration tests proposed by the Bayesian approach of statistical inference and to show how they can be overcome by the Full Bayesian Significance Test (FBST), a procedure designed to test sharp or precise hypothesis. We will compare its performance with the most used frequentist alternatives, namely, the Augmented Dickey–Fuller for unit roots and the maximum eigenvalue test for cointegration.

The e-value: a fully Bayesian significance measure for precise statistical hypotheses and its research program

This article gives a survey of the e-value, a statistical significance measure a.k.a. the evidence rendered by observational data, X, in support of a statistical hypothesis, H, or, the other way around, the epistemic value of H given X. The e-value and the accompanying FBST, the Full Bayesian Significance Test, constitute the core of a research program that was started at IME-USP, is being developed by over 20 researchers worldwide, and has, so far, been referenced by over 200 publications. The e-value and the FBST comply with the best principles of Bayesian inference, including the likelihood principle, complete invariance, asymptotic consistency, etc. Furthermore, they exhibit powerful logic or algebraic properties in situations where one needs to compare or compose distinct hypotheses that can be formulated either in the same or in different statistical models. Moreover, they effortlessly accommodate the case of sharp or precise hypotheses, a situation where alternative methods often require ad hoc and convoluted procedures. Finally, the FBST has outstanding robustness and reliability characteristics, outperforming traditional tests of hypotheses in many practical applications of statistical modeling and operations research.

EXPONENTIATED DISCRETE WEIBULL DISTRIBUTION FOR CENSORED DATA

This paper further develops the statistical inference procedure of the exponentiated discrete Weibull distribution (EDW) for data with the presence of censoring. This generalization of the discrete Weibull distribution has the advantage of being suitable to model non-monotone failure rates, such as those with bathtub and unimodal distributions. Inferences about EDW distribution are presented using both frequentist and bayesian approaches. In addition, the classical Likelihood Ratio Test and a Full Bayesian Significance Test (FBST) were performed to test the parameters of EDW distribution. The method presented is applied to simulated data and illustrated with a real dataset regarding patients diagnosed with head and neck cancer.

Forensic Speaker Comparison Using Evidence Interval in Full Bayesian Significance Test

This paper describes the application of a full Bayesian significance test (FBST) to compute evidence intervals in forensic speaker comparison (FSC). In the FBST approach, the challenge is to apply the test to a large number of observations and to formulate an equation to solve the test quickly. The contribution of the present work is that it proposes an application of the FBST to FSC and develops a method to calculate the FBST for the distribution of expected values (mean) with unknown variance without using Monte Carlo Markov chains (MCMC). Comparisons with other interval inference methodologies indicate that the evidence interval size is 49% greater than that computed with the Gosset approach. The evidence interval presented 71% fewer classification errors than the punctual inference did for the signal-to-noise ratio (SNR) of 17 dB.

Unobserved component model with multi-interval input interventions: An application to crude oil production

In this paper, a multi-interval input intervention unobserved component model (MIII-UCM) is developed to model the crude oil production series. The focus of this study is on the development of a model that could be used to model and evaluate the effects of the military crackdown, presidential amnesty program, and Niger Delta avenger insurgency respectively. The monthly crude oil production dataset from the web database of the central bank of Nigeria (CBN) was used. The proposed UCM-intervention model with significant unobserved stochastic level, deterministic trigonometric seasonal, stochastic irregular term and pulse intervention functions based on the Akaike's information criteria (AIC), Schwartz's Bayesian Information Criteria (BIC), goodness-of-fit and significance tests captured the structural variability of the time series used in this study. The results showed that step intervention functions yield negative nonsignificant effects. Generally, the military crackdown on militant groups, the presidential amnesty program, and Niger Delta avenger insurgency presented a negative nonsignificant shift on crude oil production.

A Bayesian analysis of a change in the mean of independent normal sequence with contaminated observation

In this paper, we consider a Bayesian analysis of a change in the mean of independent gaussian samples in the presence of a single outlier. An unconditional Bayesian significance test for testing change versus no change is performed under consideration of non informative prior distribution of the parameters. From a numerical study using the Gibbs sampler algorithm, the effect of a contaminated observation on the performance of the Bayesian significance test of change is studied. Keywords: Gaussian models; Change-point; HPD region sets; p-value; Outliers. AMS 2010 Mathematics Subject Classification Objects: 91B84; 62F15; 62F03

An approximate Bayesian significance test for genomic evaluations.

Genomic information can be used to study the genetic architecture of some trait. Not only the size of the genetic effect captured by molecular markers and their position on the genome but also the mode of inheritance, which might be additive or dominant, and the presence of interactions are interesting parameters. When searching for interacting loci, estimating the effect size and determining the significant marker pairs increases the computational burden in terms of speed and memory allocation dramatically. This study revisits a rapid Bayesian approach (fastbayes). As a novel contribution, a measure of evidence is derived to select markers with effect significantly different from zero. It is based on the credibility of the highest posterior density interval next to zero in a marginalized manner. This methodology is applied to simulated data resembling a dairy cattle population in order to verify the sensitivity of testing for a given range of type‐I error levels. A real data application complements this study. Sensitivity and specificity of fastbayes were similar to a variational Bayesian method, and a further reduction of computing time could be achieved. More than 50% of the simulated causative variants were identified. The most complex model containing different kinds of genetic effects and their pairwise interactions yielded the best outcome over a range of type‐I error levels. The validation study showed that fastbayes is a dual‐purpose tool for genomic inferences – it is applicable to predict future outcome of not‐yet phenotyped individuals with high precision as well as to estimate and test single‐marker effects. Furthermore, it allows the estimation of billions of interaction effects.

A discussion on significance indices for contingency tables under small sample sizes.

Hypothesis testing in contingency tables is usually based on asymptotic results, thereby restricting its proper use to large samples. To study these tests in small samples, we consider the likelihood ratio test (LRT) and define an accurate index for the celebrated hypotheses of homogeneity, independence, and Hardy-Weinberg equilibrium. The aim is to understand the use of the asymptotic results of the frequentist Likelihood Ratio Test and the Bayesian FBST (Full Bayesian Significance Test) under small-sample scenarios. The proposed exact LRT p-value is used as a benchmark to understand the other indices. We perform analysis in different scenarios, considering different sample sizes and different table dimensions. The conditional Fisher’s exact test for 2 × 2 tables and the Barnard’s exact test are also discussed. The main message of this paper is that all indices have very similar behavior, except for Fisher and Barnard tests that has a discrete behavior. The most powerful test was the asymptotic p-value from the likelihood ratio test, suggesting that is a good alternative for small sample sizes.

Model choice in separate families: A comparison between the FBST and the Cox test

ABSTRACTThe important problem of discriminating between separate families of distributions is the theme of this work. The Bayesian significance test, FBST, is compared with the celebrated Cox test. The three families most used in survival analysis, lognormal, gamma and Weibull, are considered for the discrimination. A convex combination—with unknown weights—of the three densities is used for this discrimination. After these weights have been estimated, the one with the highest value indicates the best statistical model among the three. Another important feature considered is the parameterization used. All the three densities are written as a function of the common population mean and variance. Including the weights, the number of parameters is reduced from eight (two of each density and two of the convex combination) to four (two from the common mean and variance plus two of the weights). Some numerical results from simulations are given. In these simulations, the results of FBST are compared with those obtained with the Cox test. Two real examples properly illustrate the procedures.