Posterior Variance Research Articles

From LATE to ATE: A Bayesian approach

We develop a Bayesian model that produces a posterior distribution of the marginal treatment effect (MTE) function. The method provides researchers with a principled way to extrapolate from the observed moments using flexible assumptions, thereby allowing researchers to generate plausible ranges of important and potentially policy-relevant quantities of interest. We then use the model to propose a natural decomposition of the posterior variance into “statistical uncertainty,” i.e., variance that stems from the imprecise estimation of the observed moments, and “extrapolation uncertainty,” i.e., variance that stems from uncertainty in how to extrapolate away from the observed moments. We conclude by showing that under our preferred priors, even in an experiment as large as the Oregon Health Insurance Experiment, the main source of uncertainty in the ATE comes from uncertainty in the true values of the observed moments.

Efficient ensemble uncertainty estimation in Gaussian processes regression

Abstract Reliable uncertainty measures are required when using data-based machine learning interatomic potentials (MLIPs) for atomistic simulations. In this work, we propose for sparse Gaussian process regression (GPR) type MLIPs a stochastic uncertainty measure akin to the query-by-committee approach often used in conjunction with neural network based MLIPs. The uncertainty measure is coined ‘label noise’ ensemble uncertainty as it emerges from adding noise to the energy labels in the training data. We find that this method of calculating an ensemble uncertainty is as well calibrated as the one obtained from the closed-form expression for the posterior variance when the sparse GPR is treated as a projected process. Comparing the two methods, our proposed ensemble uncertainty is, however, faster to evaluate than the closed-form expression. Finally, we demonstrate that the proposed uncertainty measure acts better to support a Bayesian search for optimal structure of Au20 clusters.

Deep learning coupled Bayesian inference method for measuring the elastoplastic properties of SS400 steel welds by nanoindentation experiment

Nanoindentation experiment has shown broad application prospects due to its ability to measure the mechanical properties of various materials at multiple scales. In this paper, a deep learning coupled Bayesian inverse approach is proposed for measuring the elastoplastic parameters of SS400 steel welds by nanoindentation experiment. The nanoindentation experiments were performed on the SS400 steel welds, including base metal (BM), weld zone (WZ), and heat affected zone (HAZ), and the experiment load–displacement (P-h) curves were obtained. The hyper-parameters tunable artificial neural network (ANN) was established to correlate elastoplastic parameters with indentation P-h curves. Based on Bayesian inference theory, the posterior density function for estimating the unknown material parameters was established. Transitional Markov chain Monte Carlo was used for sampling from the posterior density function, and the elastoplastic properties in different regions of SS400 steel welds were identified. The advantage of the established measuring method is that the hyper-parameters optimized ANN model can provide the very accurate forward relationship between material properties and indentation P-h curves. Besides, the inverse Bayesian framework can quantify the potential uncertainty of the identified elastoplastic parameters. The measured elastoplastic properties of the base metal of SS400 steel show good agreement with tensile experiment data, of which the maximum measuring error is less than 12%. The measured elastoplastic properties in WZ and HAZ are also proved to be effective. The uncertainty of the identified elastoplastic parameters of SS400 steel welds can be quantified by posterior marginal distribution, using Mean and Variance values. The results proved that the proposed inverse measuring method is reliable and effective.

Uncertainty propagation of initial conditions in thermal models

The operation of machine tools often demands a highly accurate knowledge of the tool center point’s (TCP) position. The displacement of the TCP over time can be inferred from coupled thermal models, which comprise a set of geometrically coupled heat equations. Each of these equations represents the temperature in part of the machine, and they are often formulated on complicated geometries. The accuracy of the TCP prediction depends highly on the accuracy of the model parameters, such as heat exchange parameters, and the initial temperature. Thus it is of utmost interest to determine the influence of these parameters on the TCP displacement prediction. In turn, the accuracy of the parameter estimate is essentially determined by the measurement accuracy and the sensor placement. Determining the accuracy of a given sensor configuration is a key prerequisite of optimal sensor placement. In this paper, we develop a thermal model for the discretization of realistic machine tool. On top of this model we propose two numerical algorithms to evaluate any given thermal sensor configuration with respect to its accuracy. We compute the posterior variances from the posterior covariance matrix with respect to an uncertain initial temperature field. The full matrix is dense and potentially very large, depending on the size of the discretized model. We first present a way to compute this approximation, which relies on the computation of the model sensitivities with respect to the initial values. Additionally, we present a low-rank tensor method which exploits the underlying system structure. We compare the efficiency of both algorithms with respect to runtime and memory requirements and discuss their respective advantages with regard to optimal sensor placement problems.

Active Learning for a Recursive Non-Additive Emulator for Multi-Fidelity Computer Experiments

Computer simulations have become essential for analyzing complex systems, but high-fidelity simulations often come with significant computational costs. To tackle this challenge, multi-fidelity computer experiments have emerged as a promising approach that leverages both low-fidelity and high-fidelity simulations, enhancing both the accuracy and efficiency of the analysis. In this article, we introduce a new and flexible statistical model, the Recursive Non-Additive (RNA) emulator, that integrates the data from multi-fidelity computer experiments. Unlike conventional multi-fidelity emulation approaches that rely on an additive auto-regressive structure, the proposed RNA emulator recursively captures the relationships between multi-fidelity data using Gaussian process priors without making the additive assumption, allowing the model to accommodate more complex data patterns. Importantly, we derive the posterior predictive mean and variance of the emulator, which can be efficiently computed in a closed-form manner, leading to significant improvements in computational efficiency. Additionally, based on this emulator, we introduce four active learning strategies that optimize the balance between accuracy and simulation costs to guide the selection of the fidelity level and input locations for the next simulation run. We demonstrate the effectiveness of the proposed approach in a suite of synthetic examples and a real-world problem. An R package RNAmf for the proposed methodology is provided on CRAN.

Genomic-inferred cross-selection methods for multi-trait improvement in a recurrent selection breeding program

The major drawback to the implementation of genomic selection in a breeding program lies in long-term decrease in additive genetic variance, which is a trade-off for rapid genetic improvement in short term. Balancing increase in genetic gain with retention of additive genetic variance necessitates careful optimization of this trade-off. In this study, we proposed an integrated index selection approach within the genomic inferred cross-selection (GCS) framework to maximize genetic gain across multiple traits. With this method, we identified optimal crosses that simultaneously maximize progeny performance and maintain genetic variance for multiple traits. Using a stochastic simulated recurrent breeding program over a 40-years period, we evaluated different GCS methods along with other factors, such as the number of parents, crosses, and progeny per cross, that influence genetic gain in a pulse crop breeding program. Across all breeding scenarios, the posterior mean variance consistently enhances genetic gain when compared to other methods, such as the usefulness criterion, optimal haploid value, mean genomic estimated breeding value, and mean index selection value of the superior parents. In addition, we provide a detailed strategy to optimize the number of parents, crosses, and progeny per cross that can potentially maximize short- and long-term genetic gain in a public breeding program.

Technical note: Posterior uncertainty estimation via a Monte Carlo procedure specialized for 4D-Var data assimilation

Abstract. Through the Bayesian lens of four-dimensional variational (4D-Var) data assimilation, uncertainty in model parameters is traditionally quantified through the posterior covariance matrix. However, in modern settings involving high-dimensional and computationally expensive forward models, posterior covariance knowledge must be relaxed to deterministic or stochastic approximations. In the carbon flux inversion literature, (Chevallier et al., 2007) proposed a stochastic method capable of approximating posterior variances of linear functionals of the model parameters that is particularly well suited for large-scale Earth-system 4D-Var data assimilation tasks. This note formalizes this algorithm and clarifies its properties. We provide a formal statement of the algorithm, demonstrate why it converges to the desired posterior variance quantity of interest, and provide additional uncertainty quantification allowing incorporation of the Monte Carlo sampling uncertainty into the method's Bayesian credible intervals. The methodology is demonstrated using toy simulations and a realistic carbon flux inversion observing system simulation experiment.

Joint Bayesian estimation of process and measurement noise statistics in nonlinear Kalman filtering

Bayesian filtering aims at sequentially estimating the states and parameters in nonlinear dynamical systems, and finds applications in a variety of engineering fields. Extended and unscented Kalman filter are popular choices for this purpose as they typically achieve a good trade-off between computational complexity and accuracy, while still quantifying posterior uncertainty. However, the accuracy of the identified states and parameters, both in terms of their posterior mean and variance, is heavily influenced by the assumed statistics of both the process and measurement noise terms. Therefore, learning the noise characteristics becomes an essential aspect of Bayesian filtering, but is non-trivial especially in cases where process and measurement noise statistics are jointly non-identifiable. This article proposes a methodology that decomposes measurement and process noise learning in a fully Bayesian setting. For measurement noise estimation, we improve upon a Bayesian method that leverages conjugacy of the inverse-Wishart distribution with respect to the Gaussian likelihood model. For process noise estimation, Bayesian model averaging is coupled with a mutation scheme to compare the performance of competing models while exploring high-probability regions in the space of the unknown process noise variance. The algorithm is first tested on simple numerical models for both state estimation and parameter learning, highlighting the superior accuracy of the proposed measurement noise correction and the benefits of decoupling process and measurement noise learning in non-identifiable settings. Then we demonstrate the usefulness of this algorithm in more challenging problems, namely identification of a multi-degree-of-freedom system under unmeasured excitation, and modeling of the highly nonlinear response of a full-scale bridge pier using experimental data.

Expert’s reputation concern and consumer information

Serious information asymmetry exists between consumers and expert sellers in the credit-goods market. This study develops a generic model to investigate the relationship between consumer information and expert fraud in the credence goods market. The unique features of the model are that (i) the state of the consumer is multidimensional, (ii) the expert seller has concerns about their reputation, and (iii) information asymmetry exists between the consumer and the expert seller regarding the necessity of the provided treatment ex-ante and ex-post. On average, the equilibrium amount of the recommended treatment is greater than the necessary level. The contingent effect of any consumer information on expert overtreatment can be evaluated by analyzing the posterior variance of the number of problems updated by the information. Consumer information on only part of the problem hardly reduces expert overtreatment in certain situations. The unconditional effect of consumer information on expert seller fraud varies considerably with state distribution.

SECRET: Statistical Emulation for Computational Reverse Engineering and Translation with applications in healthcare

There have been impressive advances in the physical and mathematical modelling of complex physiological systems in the last few decades, with the potential to revolutionise personalised healthcare with patient-specific evidence-based diagnosis, risk assessment and treatment decision support using digital twins. However, practical progress and genuine clinical impact hinge on successful model calibration, parameter estimation and uncertainty quantification, which calls for novel innovative adaptions and methodological extensions of contemporary state-of-the-art inference techniques from Statistics and Machine Learning. In the present study, we focus on two computational fluid-dynamics (CFD) models of the blood systemic and pulmonary circulation. We discuss state-of-the-art emulation techniques based on deep learning and Gaussian processes, which are coupled with established inference techniques based on greedy optimisation, simulated annealing, Markov Chain Monte Carlo, History Matching and rejection sampling for computationally fast inference of unknown parameters of the CFD models from blood flow and pressure data. The inference task was set as a competitive challenge which the participants had to conduct within a limited time frame representative of clinical requirements. The performance of the methods was assessed independently and objectively by the challenge organisers, based on a ground truth that was unknown to the method developers. Our results indicate that for the systemic challenge, in which an idealised case of noise-free data was considered, the relative deviation from the ground-truth in parameter space ranges from 10−5% (highest-performing method) to 3% (lowest-performing method). For the pulmonary challenge, for which noisy data was generated, the performance ranges from 0.9% to 7% deviation for the parameter posterior mean, and from 35% to 570% deviation for the parameter posterior variance.

Optimal Sensor Placement with Minimal Predicted Posterior Variance

The performance of each monitoring system critically depends on the sensor layout and typical optimization criteria are based on the accessibility to the structure on site, experience with similar structures, and an optimal signal-to-noise ratio. More advanced criteria aim to optimize the feature extraction process, and this paper aims to develop a sensor placement strategy for optimal damage diagnosis. The approach is based on Bayesian updating and the optimization goal is to find a sensor layout that is the most sensitive to small changes in the examined system parameters. It is shown that such layouts manifest themselves through a minimum variance in the posterior distribution. Using Kalman filters, the posterior variance can be “predicted” based on the measurement error and numerical models, so no data from and no simulations in the damaged state are required. By employing a polynomial chaos expansion (PCE)-based Kalman filter, the method becomes applicable for structures with many measurement channels, multiple examined system parameters, and for a large number of sensor combinations. Gaussian distributions are assumed for both system inputs and outputs. For proof of concept, the sensor placement is optimized on a numerical cantilever beam, demonstrating that the method can efficiently find the optimal sensor layout. Moreover, arbitrary measurement quantities (inclinations, vibrations, etc.) or damage-sensitive features (statistical values, modal parameters, etc.) can be used as observations, and multiple system parameters can be considered at the same time. Among other factors, an optimal sensor layout leads to large measurement responses, a small measurement error, and more advanced statistical quantities that will be elaborated on in detail.

Cloud inversion analysis of surrounding rock parameters for underground powerhouse based on PSO-BP optimized neural network and web technology

Aiming at the shortcomings of the BP neural network in practical applications, such as easy to fall into local extremum and slow convergence speed, we optimized the initial weights and thresholds of the BP neural network using the particle swarm optimization (PSO). Additionally, cloud computing service, web technology, cloud database and numerical simulation were integrated to construct an intelligent feedback analysis cloud program for underground engineering safety monitoring based on the PSO-BP algorithm. The program could conveniently, quickly, and intelligently carry out numerical analysis of underground engineering and dynamic feedback analysis of surrounding rock parameters. The program was applied to the cloud inversion analysis of the surrounding rock parameters for the underground powerhouse of the Shuangjiangkou Hydropower Station. The calculated displacement simulated with the back-analyzed parameters matches the measured displacement very well. The posterior variance evaluation shows that the posterior error ratio is 0.045 and the small error probability is 0.999. The evaluation results indicate that the intelligent feedback analysis cloud program has high accuracy and can be applied to engineering practice.

Adaptive Robust Control of Uncertain Euler-Lagrange Systems Using Gaussian Processes.

This article proposes a novel adaptive robust control approach based on Gaussian processes (GPs) for the high-precision tracking problem of uncertain Euler-Lagrange (EL) systems with time-varying external disturbances. Given a prior dynamic model, the GP regression (GPR) technique is employed to obtain a nonparametric data-based uncertainty model, including its probabilistic confidence intervals. Based on the adaptive sliding mode control (ASMC) framework, the posterior means of GPs are utilized for dynamic compensation, whereas the posterior variances are applied to adjust the feedback gains. This proposed control strategy is robust against significant system uncertainty with low feedback gains. A novel adaptive law for updating hyperparameters based on tracking error feedback is presented, thereby improving the performance of both tracking control and GP modeling simultaneously. Compared to existing likelihood-based optimization methods, this hyperparameter adaptive law enables data-efficient and fast uncertainty learning for control applications. The proposed control strategy guarantees the semiglobal asymptotic convergence to zero tracking error with a specified probability. Simulations using an underwater robot model demonstrate that the utilization of GPs and hyperparameter adaptive law significantly improves the performance of tracking control and uncertainty learning.

Total Least Squares Estimation in Hedonic House Price Models

In real estate valuation using the Hedonic Price Model (HPM) estimated via Ordinary Least Squares (OLS) regression, subjectivity and measurement errors in the independent variables violate the Gauss–Markov theorem assumption of a non-random coefficient matrix, leading to biased parameter estimates and incorrect precision assessments. In this contribution, the Errors-in-Variables model equipped with Total Least Squares (TLS) estimation is proposed to address these issues. It fully considers random errors in both dependent and independent variables. An iterative algorithm is provided, and posterior accuracy estimates are provided to validate its effectiveness. Monte Carlo simulations demonstrate that TLS provides more accurate solutions than OLS, significantly improving the root mean square error by over 70%. Empirical experiments on datasets from Boston and Wuhan further confirm the superior performance of TLS, which consistently yields a higher coefficient of determination and a lower posterior variance factor, which shows its more substantial explanatory power for the data. Moreover, TLS shows comparable or slightly superior performance in terms of prediction accuracy. These results make it a compelling and practical method to enhance the HPM.

Quantum-assisted Hilbert-space Gaussian process regression

Gaussian processes are probabilistic models that are commonly used as functional priors in machine learning. Due to their probabilistic nature, they can be used to capture prior information on the statistics of noise, smoothness of the functions, and training data uncertainty. However, their computational complexity quickly becomes intractable as the size of the data set grows. We propose a Hilbert-space approximation-based quantum algorithm for Gaussian process regression to overcome this limitation. Our method consists of a combination of classical basis function expansion with quantum computing techniques of quantum principal component analysis, conditional rotations, and Hadamard and tests. The quantum principal component analysis is used to estimate the eigenvalues, while the conditional rotations and the Hadamard and tests are employed to evaluate the posterior mean and variance of the Gaussian process. Our method provides polynomial computational complexity reduction over the classical method. Published by the American Physical Society 2024

Online detection of the incidence via transfer learning

AbstractThe counting process has abundant applications in reality, and Poisson process monitoring actually has received extensive attention and research. However, conventional methods experience poor performance when shifts appears early and only small number of historical observations in Phase I can be used for estimation. To overcome it, we creatively propose a new online monitoring algorithm under the transfer learning framework, which utilizes the information from observations of additional data sources so that the target process can be described better. By making the utmost of the somewhat correlated data from other domains, which is measured by a bivariate Gamma distributed statistic presented by us, the explicit properties (e.g., posterior probability mass function, posterior expectation, and posterior variance) are also strictly proved. Furthermore, based on the above theoretical results, we design two computationally efficient control schemes in Phase II, that is a control chart based on the cumulative distribution function for large shifts and an exponentially weighted moving average control chart for small shifts. For a better understanding of the more practical applications and transferability matter, we provide some optimal values for parameter setting. Extensive numerical simulations and a case of skin cancer incidence in America verify the superiorities of our approach.

Bayesian reconstruction of Cartesian product graph signals with general patterns of missing data

In this paper, we address the challenge of signal reconstruction on Cartesian product graphs, which frequently arises in applications where node-level data may be corrupted by equipment failure or difficult to reliably collect. A recent approach to Graph Signal Reconstruction (GSR) is to formulate it as a Bayesian inference problem, where the underlying signal is presumed to be smooth with respect to the graph topology. However, the resulting linear system derived from this model specification is often intractable to solve directly for general Cartesian product graphs. To overcome this limitation, we introduce two robust and efficient iterative procedures for computing the posterior mean. The first method is a stationary iterative technique based on matrix splitting, while the second employs a conjugate gradient approach with a symmetric graph-spectral preconditioner. We provide a theoretical and numerical comparison of the convergence behaviour of these algorithms, demonstrating that the optimal choice depends on the model hyperparameters. Finally, we propose a novel supervised learning-based technique, enhanced by an active learning strategy, for estimating the posterior marginal variance. This circumvents the need to instantiate, invert or decompose large matrices directly in memory and results in an R2 statistic of over 0.95 on a synthetic test data set. This paper contributes to the development of advanced signal reconstruction techniques applicable in various domains, including large-scale sensor networks and social network analysis.

Parallelization of adaptive Bayesian cubature using multimodal optimization algorithms

PurposeBayesian cubature (BC) has emerged to be one of most competitive approach for estimating the multi-dimensional integral especially when the integrand is expensive to evaluate, and alternative acquisition functions, such as the Posterior Variance Contribution (PVC) function, have been developed for adaptive experiment design of the integration points. However, those sequential design strategies also prevent BC from being implemented in a parallel scheme. Therefore, this paper aims at developing a parallelized adaptive BC method to further improve the computational efficiency.Design/methodology/approachBy theoretically examining the multimodal behavior of the PVC function, it is concluded that the multiple local maxima all have important contribution to the integration accuracy as can be selected as design points, providing a practical way for parallelization of the adaptive BC. Inspired by the above finding, four multimodal optimization algorithms, including one newly developed in this work, are then introduced for finding multiple local maxima of the PVC function in one run, and further for parallel implementation of the adaptive BC.FindingsThe superiority of the parallel schemes and the performance of the four multimodal optimization algorithms are then demonstrated and compared with the k-means clustering method by using two numerical benchmarks and two engineering examples.Originality/valueMultimodal behavior of acquisition function for BC is comprehensively investigated. All the local maxima of the acquisition function contribute to adaptive BC accuracy. Parallelization of adaptive BC is realized with four multimodal optimization methods.

Predictive coding networks for temporal prediction.

One of the key problems the brain faces is inferring the state of the world from a sequence of dynamically changing stimuli, and it is not yet clear how the sensory system achieves this task. A well-established computational framework for describing perceptual processes in the brain is provided by the theory of predictive coding. Although the original proposals of predictive coding have discussed temporal prediction, later work developing this theory mostly focused on static stimuli, and key questions on neural implementation and computational properties of temporal predictive coding networks remain open. Here, we address these questions and present a formulation of the temporal predictive coding model that can be naturally implemented in recurrent networks, in which activity dynamics rely only on local inputs to the neurons, and learning only utilises local Hebbian plasticity. Additionally, we show that temporal predictive coding networks can approximate the performance of the Kalman filter in predicting behaviour of linear systems, and behave as a variant of a Kalman filter which does not track its own subjective posterior variance. Importantly, temporal predictive coding networks can achieve similar accuracy as the Kalman filter without performing complex mathematical operations, but just employing simple computations that can be implemented by biological networks. Moreover, when trained with natural dynamic inputs, we found that temporal predictive coding can produce Gabor-like, motion-sensitive receptive fields resembling those observed in real neurons in visual areas. In addition, we demonstrate how the model can be effectively generalized to nonlinear systems. Overall, models presented in this paper show how biologically plausible circuits can predict future stimuli and may guide research on understanding specific neural circuits in brain areas involved in temporal prediction.

Structural reliability analysis with extremely small failure probabilities: A quasi-Bayesian active learning method

The concept of Bayesian active learning has recently been introduced from machine learning to structural reliability analysis. Although several specific methods have been successfully developed, significant efforts are still needed to fully exploit their potential and to address existing challenges. This work proposes a quasi-Bayesian active learning method, called ‘Quasi-Bayesian Active Learning Cubature’, for structural reliability analysis with extremely small failure probabilities. The method is established based on a cleaver use of the Bayesian failure probability inference framework. To reduce the computational burden associated with the exact posterior variance of the failure probability, we propose a quasi posterior variance instead. Then, two critical elements for Bayesian active learning, namely the stopping criterion and the learning function, are developed subsequently. The stopping criterion is defined based on the quasi posterior coefficient of variation of the failure probability, whose numerical solution scheme is also tailored. The learning function is extracted from the quasi posterior variance, with the introduction of an additional parameter that allows multi-point selection and hence parallel distributed processing. By testing on four numerical examples, it is empirically shown that the proposed method can assess extremely small failure probabilities with desired accuracy and efficiency.