Prior-data Conflict Research Articles

Testing the fit of data and external sets via an imprecise Sargan-Hansen test

In empirical sciences such as psychology, the term cumulative science mostly refers to the integration of theories, while external (prior) information may also be used in statistical inference. This external information can be in the form of statistical moments and is subject to various types of uncertainty, e.g., because it is estimated, or because of qualitative uncertainty due to differences in study design or sampling. Before using it in statistical inference, it is therefore important to test whether the external information fits a new data set, taking into account its uncertainties. As a frequentist approach, the Sargan-Hansen test from the generalized method of moments framework is used in this paper. It tests, given a statistical model, whether data and point-wise external information are in conflict. A separability result is given that simplifies the Sargan-Hansen test statistic in most cases. The Sargan-Hansen test is then extended to the imprecise scenario with (estimated) external sets using stochastically ordered credal sets. Furthermore, an exact small sample version is derived for normally distributed variables. As a Bayesian approach, two prior-data conflict criteria are discussed as a test for the fit of external information to the data. Two simulation studies are performed to test and compare the power and type I error of the methods discussed. Different small sample scenarios are implemented, varying the moments used, the level of significance, and other aspects. The results show that both the Sargan-Hansen test and the Bayesian criteria control type I errors while having sufficient or even good power. To facilitate the use of the methods by applied scientists, easy-to-use R functions are provided in the R script in the supplementary materials.

Digital twins and Bayesian dynamic borrowing: Two recent approaches for incorporating historical control data.

Recent years have seen an increasing interest in incorporating external control data for designing and evaluating randomized clinical trials (RCT). This may decrease costs and shorten inclusion times by reducing sample sizes. For small populations, with limited recruitment, this can be especially important. Bayesian dynamic borrowing (BDB) has been a popular choice as it claims to protect against potential prior data conflict. Digital twins (DT) has recently been proposed as another method to utilize historical data. DT, also known as PROCOVA™, is based on constructing a prognostic score from historical control data, typically using machine learning. This score is included in a pre-specified ANCOVA as the primary analysis of the RCT. The promise of this idea is power increase while guaranteeing strong type 1 error control. In this paper, we apply analytic derivations and simulations to analyze and discuss examples of these two approaches. We conclude that BDB and DT, although similar in scope, have fundamental differences which need be considered in the specific application. The inflation of the type 1 error is a serious issue for BDB, while more evidence is needed of a tangible value of DT for real RCTs.

Detecting and diagnosing prior and likelihood sensitivity with power-scaling

Determining the sensitivity of the posterior to perturbations of the prior and likelihood is an important part of the Bayesian workflow. We introduce a practical and computationally efficient sensitivity analysis approach using importance sampling to estimate properties of posteriors resulting from power-scaling the prior or likelihood. On this basis, we suggest a diagnostic that can indicate the presence of prior-data conflict or likelihood noninformativity and discuss limitations to this power-scaling approach. The approach can be easily included in Bayesian workflows with minimal effort by the model builder and we present an implementation in our new R package priorsense. We further demonstrate the workflow on case studies of real data using models varying in complexity from simple linear models to Gaussian process models.

A conservative approach to leveraging external evidence for effective clinical trial design.

Prior probabilities of clinical hypotheses are not systematically used for clinical trial design yet, due to a concern that poor priors may lead to poor decisions. To address this concern, a conservative approach to Bayesian trial design is illustrated here, requiring that the operational characteristics of the primary trial outcome are stronger than the prior. This approach is complementary to current Bayesian design methods, in that it insures against prior-data conflict by defining a sample size commensurate to a discrete design prior. This approach is ethical, in that it requires designs appropriate to achieving pre-specified levels of clinical equipoise imbalance. Practical examples are discussed, illustrating design of trials with binary or time to event endpoints. Moderate increases in phase II study sample size are shown to deliver strong levels of overall evidence for go/no-go clinical development decisions. Levels of negative evidence provided by group sequential confirmatory designs are found negligible, highlighting the importance of complementing efficacy boundaries with non-binding futility criteria.

SAM: Self-adapting mixture prior to dynamically borrow information from historical data in clinical trials.

Mixture priors provide an intuitive way to incorporate historical data while accounting for potential prior-data conflict by combining an informative prior with a noninformative prior. However, prespecifying the mixing weight for each component remains a crucial challenge. Ideally, the mixing weight should reflect the degree of prior-data conflict, which is often unknown beforehand, posing a significant obstacle to the application and acceptance of mixture priors. To address this challenge, we introduce self-adapting mixture (SAM) priors that determine the mixing weight using likelihood ratio test statistics or Bayes factors. SAM priors are data-driven and self-adapting, favoring the informative (noninformative) prior component when there is little (substantial) evidence of prior-data conflict. Consequently, SAM priors achieve dynamic information borrowing. We demonstrate that SAM priors exhibit desirable properties in both finite and large samples and achieve information-borrowing consistency. Moreover, SAM priors are easy to compute, data-driven, and calibration-free, mitigating the risk of data dredging. Numerical studies show that SAM priors outperform existing methods in adopting prior-data conflicts effectively. We developed R package "SAMprior" and web application that are freely available at CRAN and www.trialdesign.org to facilitate the use of SAMpriors.

Leveraging external evidence using Bayesian hierarchical model and propensity score in the presence of covariates

In recent decades, there has been growing interest in leveraging external data information for clinical development as it improves the efficiency of the design and inference of clinical trials when utilized properly and more importantly, alleviates potential ethical and recruitment challenges. When it is of interest to augment the concurrent study's control arm using external control data, the potential outcome heterogeneity across data sources, also known as prior-data conflict, should be accounted for. In addition, in the outcome modeling, inclusion of prognostic covariates that may have impact on the outcome can avoid efficiency loss or potential bias. In this paper, we propose a Bayesian hierarchical modeling strategy incorporating covariate-adjusted meta-analytic predictive approach (cMAP) and also introduce a propensity score (PS) based sequential procedure that integrates the cMAP. In the simulation study, the proposed methods are found to have advantages in the estimation, power, and type I error control over the standard methods such as PS matching alone and hierarchical modeling that ignores the covariates. An illustrative example is used to illustrate the procedure.

Adaptively leveraging external data with robust meta-analytical-predictive prior using empirical Bayes.

The robust meta-analytical-predictive (rMAP) prior is a popular method to robustly leverage external data. However, a mixture coefficient would need to be pre-specified based on the anticipated level of prior-data conflict. This can be very challenging at the study design stage. We propose a novel empirical Bayes robust MAP (EB-rMAP) prior to address this practical need and adaptively leverage external/historical data. Built on Box's prior predictive p-value, the EB-rMAP prior framework balances between model parsimony and flexibility through a tuning parameter. The proposed framework can be applied to binomial, normal, and time-to-event endpoints. Implementation of the EB-rMAP prior is also computationally efficient. Simulation results demonstrate that the EB-rMAP prior is robust in the presence of prior-data conflict while preserving statistical power. The proposed EB-rMAP prior is then applied to a clinical dataset that comprises 10 oncology clinical trials, including the prospective study.

How Valuable Are Small Measurement Datasets in Supplementing Occupational Exposure Models? A Numerical Study Using the Advanced Reach Tool

The Advanced REACH Tool (ART) is the most detailed exposure model currently available for estimating inhalation exposures to dusts, vapours, and aerosols under a broad range of exposure scenarios. The ART follows a Bayesian approach, making use of a calibrated source-receptor model to provide central estimates of exposures and information on exposure variability from meta-analyses in the literature. Uniquely amongst exposure models, the ART provides a facility to update the baseline estimates from the mechanistic model and variance components using measurement data collected on the exposure scenario; however, in practical use, this facility is little used. In this paper, the full capability of the ART tool is demonstrated using a small number of carefully chosen case studies that each had a sufficient breadth of personal exposure measurement data to support a measurement-led exposure assessment. In total, six cases studies are documented, three where the estimate from the source-receptor model of the ART was consistent with measurement data, and a further three case studies where the source-receptor model of the ART was inconsistent with measurement data, resulting in a prior-data conflict. A simulation study was designed that involved drawing subsets of between two and ten measurements from the available measurement dataset, with estimates of the geometric mean (GM) and 90th percentile of exposures from the posterior distribution of ART compared against measurement-based estimates of these summaries. Results from this work indicate that very substantial reductions in the uncertainty associated with estimates of the GM and 90th percentile could be achieved with as few as two measurements, with results in detail sensitive to both the measurements themselves and worker and company labels associated with the measurements. For case studies involving prior-data conflicts, the estimates of the GM and 90th percentile rapidly changed as measurement data were used to update the prior. However, results suggest that the current statistical model of the ART does not allow a complete resolution of a prior-data conflict.

Bayesian Inference for the Weights in Logarithmic Pooling

Combining distributions is an important issue in decision theory and Bayesian inference. Logarithmic pooling is a popular method to aggregate expert opinions by using a set of weights that reflect the reliability of each information source. However, the resulting pooled distribution depends heavily on set of weights given to each opinion/prior and thus careful consideration must be given to the choice of weights. In this paper we review and extend the statistical theory of logarithmic pooling, focusing on the assignment of the weights using a hierarchical prior distribution. We explore several statistical applications, such as the estimation of survival probabilities, meta-analysis and Bayesian melding of deterministic models of population growth and epidemics. We show that it is possible learn the weights from data, although identifiability issues may arise for some configurations of priors and data. Furthermore, we show how the hierarchical approach leads to posterior distributions that are able to accommodate prior-data conflict in complex models.

Weakly informative priors and prior-data conflict checking for likelihood-free inference

Weakly informative priors and prior-data conflict checking for likelihood-free inference

Probability of Study Success (PrSS) Evaluation Based on Multiple Endpoints in Late Phase Oncology Drug Development

Phase 2 Oncology clinical trials are increasingly designed with multiple primary endpoints, such as progression-free survival (PFS) and overall survival (OS). While this gives a clearer picture of treatment benefit than trials with a single endpoint, it complicates the decision to begin a registration (phase 3) trial following a phase 2 trial. Methods for calculating the probability of study success (PrSS) assume the distribution of OS is well understood, but rarely consider the case of multiple primary endpoints. We introduce the BAMBOO method (BAyesian Model that Brings phase 2 cOmposite endpOints) to address this gap. We propose modeling the phase 2 log hazard ratios for PFS and OS as a bivariate normal random variable, with the unknown correlation parameter estimated with a meta-analysis or via simulation. The PrSS of a phase 3 study with multiple primary endpoints study can then be obtained using the posterior predictive distribution of this bivariate normal random variable. We provide methods to include additional surrogate endpoints, such as objective response rate, by extending the model to a multivariate normal of arbitrary dimension. Simulation results suggest the proposed method can better address the known prior-data conflict issue comparing to existing approaches, while borrowing historical data with caution is generally recommended under high discrepancy and small sample size. Finally, we provide an example of how BAMBOO can aid in planning a phase 3 trial.

Iterative importance sampling with Markov chain Monte Carlo sampling in robust Bayesian analysis

Bayesian inference under a set of priors, called robust Bayesian analysis, allows for estimation of parameters within a model and quantification of epistemic uncertainty in quantities of interest by bounded (or imprecise) probability. Iterative importance sampling can be used to estimate bounds on the quantity of interest by optimizing over the set of priors. A method for iterative importance sampling when the robust Bayesian inference relies on Markov chain Monte Carlo (MCMC) sampling is proposed. To accommodate the MCMC sampling in iterative importance sampling, a new expression for the effective sample size of the importance sampling is derived, which accounts for the correlation in the MCMC samples. To illustrate the proposed method for robust Bayesian analysis, iterative importance sampling with MCMC sampling is applied to estimate the lower bound of the overall effect in a previously published meta-analysis with a random effects model. The performance of the method compared to a grid search method and under different degrees of prior-data conflict is also explored.

A hierarchical prior for generalized linear models based on predictions for the mean response.

There has been increased interest in using prior information in statistical analyses. For example, in rare diseases, it can be difficult to establish treatment efficacy based solely on data from a prospective study due to low sample sizes. To overcome this issue, an informative prior to the treatment effect may be elicited. We develop a novel extension of the conjugate prior of Chen and Ibrahim (2003) that enables practitioners to elicit a prior prediction for the mean response for generalized linear models, treating the prediction as random. We refer to the hierarchical prior as the hierarchical prediction prior (HPP). For independent and identically distributed settings and the normal linear model, we derive cases for which the hyperprior is a conjugate prior. We also develop an extension of the HPP in situations where summary statistics from a previous study are available. The HPP allows for discounting based on the quality of individual level predictions, and simulation results suggest that, compared to the conjugate prior and the power prior, the HPP efficiency gains (e.g., lower mean squared error) where predictions are incompatible with the data. An efficient Monte Carlo Markov chain algorithm is developed. Applications illustrate that inferences under the HPP are more robust to prior-data conflict compared to selected nonhierarchical priors.

Bayesian sample size determination for diagnostic accuracy studies.

The development of a new diagnostic test ideally follows a sequence of stages which, among other aims, evaluate technical performance. This includes an analytical validity study, a diagnostic accuracy study, and an interventional clinical utility study. In this article, we propose a novel Bayesian approach to sample size determination for the diagnostic accuracy study, which takes advantage of information available from the analytical validity stage. We utilize assurance to calculate the required sample size based on the target width of a posterior probability interval and can choose to use or disregard the data from the analytical validity study when subsequently inferring measures of test accuracy. Sensitivity analyses are performed to assess the robustness of the proposed sample size to the choice of prior, and prior‐data conflict is evaluated by comparing the data to the prior predictive distributions. We illustrate the proposed approach using a motivating real‐life application involving a diagnostic test for ventilator associated pneumonia. Finally, we compare the properties of the approach against commonly used alternatives. The results show that, when suitable prior information is available, the assurance‐based approach can reduce the required sample size when compared to alternative approaches.

Bayesian adaptive randomization design incorporating propensity score-matched historical controls.

Incorporating historical control data to augment the control arm in randomized controlled trials (RCTs) is one way of increasing their efficiency and feasibility when adequate RCTs cannot be conducted. In recent work, a Bayesian adaptive randomization design incorporating historical control data has been proposed to reduce sample size according to the amount of information that could be borrowed, assessed at interim assessment in respect to prior-data conflict. However, the approach does not distinguish between the two sources of prior-data conflict: (1) imbalance in measured covariates, and (2) imbalance in unmeasured covariates. In this paper, we propose an extension of the Bayesian adaptive randomization design to incorporate propensity score-matched historical controls. At interim assessment, historical controls similar to the concurrent controls in terms of measured covariates are selected using propensity score matching. Then, final sample size of the control arm is adjusted according to the extent of borrowing from the matched historical controls quantified by effective historical sample size. The conditional power prior approach and commensurate prior approach are adopted for designing the prior, and addressing prior-data conflict due to unmeasured covariate imbalance. Simulation results show that the proposed method yields reduced bias in treatment effect estimates, type I error at the nominal level, and reduced sample size while maintaining statistical power. Even when residual imbalance exists due to unmeasured covariates, the proposed method borrowed more information without risking substantially inflated type I error and bias, providing meaningful implications for use of historical controls to facilitate the conduct of adequate RCTs.

Innovative Adaptive Study Design in Transfusion-Dependent Beta-Thalassemia: Bayesian Design with Concurrent Randomization and Borrowing from Historical Data

BackgroundClinical development of new therapies in transfusion-dependent beta-thalassemia has several challenges. Patient enrollment in rare diseases requires multi-center multi-country studies, and the lack of reliable surrogate endpoint for dose selection requires powering for clinical endpoints usually used in Phase 3 trials. An acceptable endpoint from a regulatory perspective which is based on responders analysis, such as proportion of patients experiencing ≥50% reduction in Red Blood Cell (RBC) transfusion burden and a reduction of ≥2 units, requires 12 weeks screening period to establish baseline transfusion burden for reliable comparison. Importantly, higher randomization ratio of treatment:placebo can improve patients' motivation to enroll into a trial, but it is less statistically efficient and requires higher sample size.We designed a Phase-2b, double-blind, randomized, placebo controlled, multi-center study with Vamifeport (NCT04938635) to assess the efficacy and safety of multiple doses of a new therapy in adults with transfusion-dependent beta-thalassemia. The proposed design follows the Bayesian framework with borrowing from published historical control data. The historical control data is used to construct an informative prior for the control arm to reduce the burden of patients randomized to a control arm and improve the trial's efficiency in performing dose selection.Study Design and MethodsAdults (18 to 65 y.o.) with documented diagnosis of β-thalassemia or hemoglobin E / β-thalassemia will be randomized to three doses of the investigational drug or placebo plus best supportive care. RBC transfusion dependence is defined as at least 6 RBC Units in the 24 weeks prior to randomization and no transfusion-free period for ≥35 days during that period. The primary endpoint is the proportion of patients experiencing ≥33% reduction of RBC units from baseline and a reduction of ≥2 units assessed from week 13 to week 24. The key secondary endpoints include proportion of patients experiencing ≥33% reduction from week 37 to week 48; proportion of patients experiencing ≥50% reduction over any consecutive 12-week interval from week 1 to week 48 and the mean change from baseline in RBC transfusions (units) from week 13 to week 24. The primary and key-secondary analysis will be conducted in a hierarchical fashion to account for multiplicity.We proposed a Bayesian design with the use of noninformative, or weakly informative, priors for the active dose arms while using a robustified informative prior for the control arm. Historical control data will be “borrowed” in an informative prior for the control arm rate from the Phase 3 trial - BELIEVE. The robustification is required in order to control the level of borrowing depending on the level of prior-data conflict.Prior-data conflict can arise from multiple sources like population heterogeneity between the historical and current study. Therefore, the selection of historical data (BELIEVE trial) addresses similarity in inclusion / exclusion criteria, standard of care etc. The robustification of the informative prior does not take into account prior-data conflict in terms of population or study characteristics but directly focuses on the informative prior of the parameter of interest and the corresponding likelihood of the current data. For example, in the BELIEVE study, out of 112 patients randomized to the control arm, 5 patients (4.5%) had a ≥33% reduction in transfusion burden over 24 weeks. A prior-data conflict may arise if the Phase-2b trial of interest here, suggests that the proportion is substantially different that 4.5% and this can inflate the frequentist Type-I or Type-II error rates examined via simulations.We evaluated Type-I error rates of the proposed design with 5000 Monte-Carlo runs for each scenario of the response rates. Using informative prior with no prior-data conflict the type-I error with no robustification is ≈ 2.4%. As the prior-data conflict increases, without robustification, the type-I error cannot be controlled. However, with a robustification weight of 0.5 the type-I errors can be controlled in line with regulatory requirements.DiscussionA proposed Bayesian design with robustified informative prior for the control arm helps reduce patients' burden of randomization to control arm and reduce overall sample size for a rare disease trial when recruitment and trial duration are challenging. DisclosuresMuehlemann: Vifor Pharma AG: Consultancy. Mukherjee: Vifor Pharma AG: Consultancy. Taher: Bristol Myers Squibb: Consultancy, Research Funding; Vifor Pharma: Consultancy, Research Funding; Agios Pharmaceuticals: Consultancy; Ionis Pharmaceuticals: Consultancy, Research Funding; Novartis: Consultancy, Research Funding. Gudmundsdottir: Vifor Pharma AG: Current Employment. Morin: Vifor Pharma AG: Current Employment. Richard: Vifor Pharma AG: Current Employment.

Deep historical borrowing framework to prospectively and simultaneously synthesize control information in confirmatory clinical trials with multiple endpoints

ABSTRACT In current clinical trial development, historical information is receiving more attention as it provides utility beyond sample size calculation. Meta-analytic-predictive (MAP) priors and robust MAP priors have been proposed for prospectively borrowing historical data on a single endpoint. To simultaneously synthesize control information from multiple endpoints in confirmatory clinical trials, we propose to approximate posterior probabilities from a Bayesian hierarchical model and estimate critical values by deep learning to construct pre-specified strategies for hypothesis testing. This feature is important to ensure study integrity by establishing prospective decision functions before the trial conduct. Simulations are performed to show that our method properly controls family-wise error rate and preserves power as compared with a typical practice of choosing constant critical values given a subset of null space. Satisfactory performance under prior-data conflict is also demonstrated. We further illustrate our method using a case study in Immunology.

Avoiding prior–data conflict in regression models via mixture priors

The Bayesian‐80 model consists of the prior–likelihood pair. A prior–data conflict arises whenever the prior allocates most of its mass to regions of the parameter space where the likelihood is relatively low. Once a prior–data conflict is diagnosed, what to do next is a hard question to answer. We propose an automatic prior elicitation that involves a two‐component mixture of a diffuse and an informative prior distribution that favours the first component if a conflict emerges. Using various examples, we show that these mixture priors can be useful in regression models as a device for regularizing the estimates and retrieving useful inferential conclusions.

Commentary on “Empirical profile Bayesian estimation for extrapolation of historical adult data to pediatric drug development”

Bayesian borrowing has been a long-standing concept, but the difficulty has been that if the current data deviates from the historical information, often referred to as the prior-data conflict, then the simple traditional Bayesian approach will introduce bias and lead to a false positive decision

Checking for model failure and for prior-data conflict with the constrained multinomial model

Multinomial models can be difficult to use when constraints are placed on the probabilities. An exact model checking procedure for such models is developed based on a uniform prior on the full multinomial model. For inference, a nonuniform prior can be used and a consistency theorem is proved concerning a check for prior-data conflict with the chosen prior. Applications are presented and a new elicitation methodology is developed for multinomial models with ordered probabilities.