Bayesian Metric Research Articles

Beyond the Classical Type I Error: Bayesian Metrics for Bayesian Designs Using Informative Priors

There is growing interest in Bayesian clinical trial designs with informative prior distributions, for example for extrapolation of adult data to pediatrics, or use of external controls. While the classical Type I error is commonly used to evaluate such designs, it cannot be strictly controlled and it is acknowledged that other metrics may be more appropriate. We focus on two common situations—borrowing control data or information on the treatment contrast—and discuss several fully probabilistic metrics to evaluate the risk of false positive conclusions. Each metric requires specification of a design prior, which can differ from the analysis prior and permits understanding of the behavior of a Bayesian design under scenarios where the analysis prior differs from the true data generation process. The metrics include the average Type I error and the pre-posterior probability of a false positive result. When borrowing control data, our empirical cases demonstrate that the average Type I error is asymptotically controlled (in certain cases strictly) when the analysis and design prior coincide. We illustrate use of these Bayesian metrics with real applications, and discuss how they could facilitate discussions between sponsors, regulators and other stakeholders about the appropriateness of Bayesian borrowing designs for pivotal studies.

Interpreting frequentist hypothesis tests: insights from Bayesian inference

Randomized controlled trials are one of the best ways of quantifying the effectiveness of medical interventions. Therefore, when the authors of a randomized superiority trial report that differences in the primary outcome between the intervention group and the control group are “significant” (i.e., P ≤ 0.05), we might assume that the intervention has an effect on the outcome. Similarly, when differences between the groups are “not significant,” we might assume that the intervention does not have an effect on the outcome. Nevertheless, both assumptions are frequently incorrect.In this article, we explore the relationship that exists between real treatment effects and declarations of statistical significance based on P values and confidence intervals. We explain why, in some circumstances, the chance an intervention is ineffective when P ≤ 0.05 exceeds 25% and the chance an intervention is effective when P > 0.05 exceeds 50%.Over the last decade, there has been increasing interest in Bayesian methods as an alternative to frequentist hypothesis testing. We provide a robust but nontechnical introduction to Bayesian inference and explain why a Bayesian posterior distribution overcomes many of the problems associated with frequentist hypothesis testing.Notwithstanding the current interest in Bayesian methods, frequentist hypothesis testing remains the default method for statistical inference in medical research. Therefore, we propose an interim solution to the “significance problem” based on simplified Bayesian metrics (e.g., Bayes factor, false positive risk) that can be reported along with traditional P values and confidence intervals. We calculate these metrics for four well-known multicentre trials. We provide links to online calculators so readers can easily estimate these metrics for published trials. In this way, we hope decisions on incorporating the results of randomized trials into clinical practice can be enhanced, minimizing the chance that useful treatments are discarded or that ineffective treatments are adopted.

Detection of clouds in multiple wind velocity fields using ground-based infrared sky images

To improve the performance of an intra-hour global solar irradiance forecasting algorithm, it is important to detect multiple layers of clouds. Horizontal atmospheric wind shear causes wind velocity fields to have different directions and speeds. In images of clouds acquired using ground-based sky imagers, clouds may be moving in different wind layers. The information provided by a solar forecasting algorithm is necessary to optimize and schedule the solar generation resources and storage devices in a smart grid. This investigation studies the performance of unsupervised learning techniques when detecting the number of cloud layers in infrared sky images. The images are acquired using an innovative infrared sky imager mounted on a solar tracker. Different mixture models are used to infer the distribution of the cloud features. Multiple Bayesian metrics and a sequential hidden Markov model are implemented to find the optimal number of clusters in the mixture models, and their performances are compared. The motion vectors are computed using a probabilistic implementation of the Lucas-Kanade algorithm. The correlations between the cloud motion vectors and temperatures are analyzed to discover the method that leads to the most accurate results. The findings point that a sequential hidden Markov model outperforms the detection accuracy of standard Bayesian model selection metrics.

Kidney transplant program specific reporting and transplant metrics.

Kidney transplantation is a heavily regulated medical procedure with the Secretary of HHS ultimately responsible for oversight and authority derived from the NOTA and the Final Rule. Transplant Programs undergo publicly reported evaluations every 6 months based on outcomes from a 2-and-a-half-year period. The current Bayesian metrics for kidney transplant programs were created such that over ten percentage of programs are deemed underperformers, or 'flag', every 6 months. Newly suggested transplant metrics have been released for public comment in Summer 2021. In addition to graft outcomes, waiting list mortality and organ acceptance rate ratios are proposed. Under the newly proposed kidney transplant metrics, over 10% of programs are expected to be deemed underperformers or 'flagged'. Transplant Center flagging is well correlated with decreased transplantation due to the transplant centres move to more conservative organ and patient acceptance. Death on the waiting list is a proposed metric over which transplant centres have little influence. In the USA, the harsh regulation continued by Health Resources and Services Administration (HRSA) through the national organ procurement and transplant network (OPTN) and Scientific Registry for Transplant Recipients (SRTR) leads directly to high organ discard rates and limitations to transplanting patients with perceived unadjusted risks. Instead of loosening regulation in a highly functioning industry that achieves remarkable outcomes in end stage kidney patients, the OPTN with the SRTR persist in increasing potential penalties through more proposed metrics that continue to deem 10% of US kidney transplant programs as underperformers. HRSA must establish a reasonable regulatory environment that allows for innovation and increased transplant opportunities for US end-stage renal disease patients.

How best to quantify replication success? A simulation study on the comparison of replication success metrics.

To overcome the frequently debated crisis of confidence, replicating studies is becoming increasingly more common. Multiple frequentist and Bayesian measures have been proposed to evaluate whether a replication is successful, but little is known about which method best captures replication success. This study is one of the first attempts to compare a number of quantitative measures of replication success with respect to their ability to draw the correct inference when the underlying truth is known, while taking publication bias into account. Our results show that Bayesian metrics seem to slightly outperform frequentist metrics across the board. Generally, meta-analytic approaches seem to slightly outperform metrics that evaluate single studies, except in the scenario of extreme publication bias, where this pattern reverses.

Effect estimates of COVID-19 non-pharmaceutical interventions are non-robust and highly model-dependent

ObjectiveTo compare the inference regarding the effectiveness of the various non-pharmaceutical interventions (NPIs) for COVID-19 obtained from different SIR models. Study design and settingWe explored two models developed by Imperial College that considered only NPIs without accounting for mobility (model 1) or only mobility (model 2), and a model accounting for the combination of mobility and NPIs (model 3). Imperial College applied models 1 and 2 to 11 European countries and to the USA, respectively. We applied these models to 14 European countries (original 11 plus another 3), over two different time horizons. ResultsWhile model 1 found that lockdown was the most effective measure in the original 11 countries, model 2 showed that lockdown had little or no benefit as it was typically introduced at a point when the time-varying reproduction number was already very low. Model 3 found that the simple banning of public events was beneficial, while lockdown had no consistent impact. Based on Bayesian metrics, model 2 was better supported by the data than either model 1 or model 3 for both time horizons. ConclusionInferences on effects of NPIs are non-robust and highly sensitive to model specification. In the SIR modeling framework, the impacts of lockdown are uncertain and highly model-dependent.

A Bayesian analysis for the Wilcoxon signed-rank statistic

ABSTRACTA Bayesian analysis is provided for the Wilcoxon signed-rank statistic (T+). The Bayesian analysis is based on a sign-bias parameter φ on the (0, 1) interval. For the case of a uniform prior probability distribution for φ and for small sample sizes (i.e., 6 ⩽ n ⩽ 25), values for the statistic T+ are computed that enable probabilistic statements about φ. For larger sample sizes, approximations are provided for the asymptotic likelihood function P(T+|φ) as well as for the posterior distribution P(φ|T+). Power analyses are examined both for properly specified Gaussian sampling and for misspecified non Gaussian models. The new Bayesian metric has high power efficiency in the range of 0.9–1 relative to a standard t test when there is Gaussian sampling. But if the sampling is from an unknown and misspecified distribution, then the new statistic still has high power; in some cases, the power can be higher than the t test (especially for probability mixtures and heavy-tailed distributions). The new Bayesian analysis is thus a useful and robust method for applications where the usual parametric assumptions are questionable. These properties further enable a way to do a generic Bayesian analysis for many non Gaussian distributions that currently lack a formal Bayesian model.

Food-Web Complexity in Guaymas Basin Hydrothermal Vents and Cold Seeps.

In the Guaymas Basin, the presence of cold seeps and hydrothermal vents in close proximity, similar sedimentary settings and comparable depths offers a unique opportunity to assess and compare the functioning of these deep-sea chemosynthetic ecosystems. The food webs of five seep and four vent assemblages were studied using stable carbon and nitrogen isotope analyses. Although the two ecosystems shared similar potential basal sources, their food webs differed: seeps relied predominantly on methanotrophy and thiotrophy via the Calvin-Benson-Bassham (CBB) cycle and vents on petroleum-derived organic matter and thiotrophy via the CBB and reductive tricarboxylic acid (rTCA) cycles. In contrast to symbiotic species, the heterotrophic fauna exhibited high trophic flexibility among assemblages, suggesting weak trophic links to the metabolic diversity of chemosynthetic primary producers. At both ecosystems, food webs did not appear to be organised through predator-prey links but rather through weak trophic relationships among co-occurring species. Examples of trophic or spatial niche differentiation highlighted the importance of species-sorting processes within chemosynthetic ecosystems. Variability in food web structure, addressed through Bayesian metrics, revealed consistent trends across ecosystems. Food-web complexity significantly decreased with increasing methane concentrations, a common proxy for the intensity of seep and vent fluid fluxes. Although high fluid-fluxes have the potential to enhance primary productivity, they generate environmental constraints that may limit microbial diversity, colonisation of consumers and the structuring role of competitive interactions, leading to an overall reduction of food-web complexity and an increase in trophic redundancy. Heterogeneity provided by foundation species was identified as an additional structuring factor. According to their biological activities, foundation species may have the potential to partly release the competitive pressure within communities of low fluid-flux habitats. Finally, ecosystem functioning in vents and seeps was highly similar despite environmental differences (e.g. physico-chemistry, dominant basal sources) suggesting that ecological niches are not specifically linked to the nature of fluids. This comparison of seep and vent functioning in the Guaymas basin thus provides further supports to the hypothesis of continuity among deep-sea chemosynthetic ecosystems.

Uncertain deduction and conditional reasoning.

There has been a paradigm shift in the psychology of deductive reasoning. Many researchers no longer think it is appropriate to ask people to assume premises and decide what necessarily follows, with the results evaluated by binary extensional logic. Most every day and scientific inference is made from more or less confidently held beliefs and not assumptions, and the relevant normative standard is Bayesian probability theory. We argue that the study of “uncertain deduction” should directly ask people to assign probabilities to both premises and conclusions, and report an experiment using this method. We assess this reasoning by two Bayesian metrics: probabilistic validity and coherence according to probability theory. On both measures, participants perform above chance in conditional reasoning, but they do much better when statements are grouped as inferences, rather than evaluated in separate tasks.

An Adaptive Response Surface Method Using Bayesian Metric and Model Bias Correction Function

The Bayesian metric was used to select the best available response surface in the literature. One of the major drawbacks of this method is the lack of a rigorous method to quantify data uncertainty, which is required as an input. In addition, the accuracy of any response surface is inherently unpredictable. This paper employs the Gaussian process based model bias correction method to quantify the data uncertainty and subsequently improve the accuracy of a response surface model. An adaptive response surface updating algorithm is then proposed for a large-scale problem to select the best response surface. The proposed methodology is demonstrated by a mathematical example and then applied to a vehicle design problem.

Protein Folding Is Mechanistically Robust

Markov state models (MSMs) have proven to be useful tools in simulating large and slowly-relaxing biological systems like proteins. MSMs model proteins through dynamics on a discrete-state energy landscape, allowing molecules to effectively sample large regions of phase space. In this work, we use aspects of MSMs to ask: is protein folding mechanistically robust? We first provide a definition of mechanism in the context of Markovian models, and we later use perturbation theory and the concept of parametric sloppiness to show that parts of the MSM eigenspectrum are resistant to perturbation. We introduce a new, to our knowledge, Bayesian metric by which eigenspectrum robustness can be evaluated, and we discuss the implications of mechanistic robustness and possible new applications of MSMs to understanding biophysical phenomena.

Bayesian metrics for comparing response surface models of data with uncertainty

For many optimization applications a complicated computational simulation is replaced with a simpler response surface model. These models are built by fitting a limited number of evaluations of the full simulation with a simple function that captures the trends in the evaluated data. In many cases the values of the data at the evaluation points have some uncertainty. This paper uses Bayesian model selection to derive two objective metrics that can be used to determine which response surface model provides the most appropriate representation of the evaluated data given the associated uncertainty. These metrics are shown to be consistent with modelling intuition based on Occam's principle. The uncertainty may be due to numerical error, approximations, uncertain input conditions, or to higher order effects in the simulation that do not need to be fit by the response surface. Two metrics, Q and G, are derived in this paper. The metric Q assumes that a good estimate of the simulation uncertainty is available. The metric G assumes the uncertainty, although present, is unknown. Application of these metrics in one and two dimensions are demonstrated.