Bayesian Neural Network Model Research Articles

A probabilistic forecast for multi-year ENSO using bayesian convolutional neural network

Abstract A robust El Niño Southern Oscillation (ENSO) prediction is essential for monitoring the global climate, regional monsoons, and weather extremes. Despite dedicated efforts spanning decades, the precise prediction of ENSO events through numerical modelling beyond a couple of seasonal lead times remains a daunting challenge. The advent of deep learning-based approaches marks a transformative era in climate and weather prediction. However, many machine learning-based studies attempting ENSO prediction are confined to singular estimates, lacking adequate quantification of uncertainty in learned parameters and overlooking the crucial need for a nuanced understanding of ENSO prediction confidence. Here, we introduce a deep learning-based Bayesian Convolutional Neural Network (BCNN) model that provides robust probabilistic predictions for ENSO with a lead time of up to 9-10 months across all seasons. The Bayesian layers within the convolutional neural network maintain the capability to predict a distribution of learned parameters. Augmented with bias correction, our model reproduces the amplitude of the Nino 3.4 index with fidelity for lead up to 9-10 months. The inherent capacity for uncertainty modelling enhances the reliability of BNNs, making them particularly valuable in operational services. This research holds substantial socio-economic implications as it enhances our forecasting capabilities and rigorously quantifies forecast uncertainties, providing valuable insights for planning and policymaking.

Reliable Prediction of Solar Photovoltaic Power and Module Efficiency using Bayesian Surrogate Assisted Explainable Data-Driven Model

This study proposes a Bayesian surrogate-driven explainable deep neural network model to predict and interpret the module efficiency and maximum output power of three commercially available photovoltaic modules: monocrystalline silicon, polycrystalline silicon, and amorphous silicon during the winter season. In addition, the influence of the photovoltaic material and ambient conditions on the predicted power and module efficiency is investigated. The model inputs include solar irradiance, module material (monocrystalline, polycrystalline, and amorphous silicon), season of the year (months), and time of day (morning to evening). Experiments were conducted on these three modules during the winter using data from Rawalpindi, Pakistan. These data were then fed into the deep neural network model to train and yield predictions. Several preprocessing techniques such as logarithmic transformation, square root, cube root, reciprocal, and exponential transformations are employed to improve the linearity of distributed data. For hyper-parameters optimization, Gaussian process, gradient boost regression trees, and random forest are used. The results show that the optimal deep neural network using random forest surrogate model along with square root and exponential transformation is able to predict the maximum power output and module efficiency of the considered photovoltaic modules for the entire investigated period with a correlation coefficient, R2 = 0.998. The least accurate model (R2=0.991) is a Gaussian process using simple data distribution. These results hold valuable insights for predicting and optimizing the performance of other solar photovoltaic modules in various climates and different seasons of the year.

Dynamic uncertainty evaluation of cylindricity error based on Bayesian deep neural network propagation method

Most of the existing methods for measuring and evaluating product geometric tolerances are for static processes, which use lower dimensions, are more complex to calculate, and have poor traceability and dynamic manageability and control of the measurement process. In this paper, a dynamic evaluation model of cylindricity error uncertainty based on the Bayesian deep neural network propagation method is proposed. Firstly, the input random variable sampling sample containing error uncertainty is generated by proposing the improved Monte Carlo sampling method (IMCS), which has the characteristics of homogeneity and validity; then the principle and construction process of a Bayesian neural network model are introduced to generate the propagation model of multidimensional random error variables; further, the Gaussian mixture model is trained by the Monte Carlo method, and the probability density function of the target response is generated by using the Gaussian mixture model. Through numerical experiments, test analysis of bearing outer rings, and comparison with ISO standard methods, the results show that the model based on the dynamic Bayesian deep neural network propagation method can correctly and effectively assess the uncertainty dynamically, and the whole estimation process is stable.

Predicting Market Capitalization of Large Global Companies using Ordinary Least Square, Feedforward and Bayesian Neural Network Models

Predicting Market Capitalization of Large Global Companies using Ordinary Least Square, Feedforward and Bayesian Neural Network Models

Bayesian approach to energy dependence of fission product yields of 235U by data augmentation

ABSTRACT We utilized a two hidden-layer Bayesian neural network (BNN) model along with data augmentation (DA) to predict the energy dependence of fission product yields (FPY). In the BNN model, the JENDL-5 FPY data are separated into 80 % for training and 20 % for validation. Additionally, the training data are combined with experimental cumulative fission yields and calculated values by five Gaussian model. The number of units in each layer and activation function were selected carefully to reproduce the global and fine structure of the FPY data. Through comparing the results with and without DA, we found that DA is particularly valuable for specific nuclides. The evaluation results with DA demonstrate reliable and accurate predictions of the energy dependence of fission product yields of 235U.

Improving the accuracy of genomic prediction in dairy cattle using the biologically annotated neural networks framework

BackgroundBiologically annotated neural networks (BANNs) are feedforward Bayesian neural network models that utilize partially connected architectures based on SNP-set annotations. As an interpretable neural network, BANNs model SNP and SNP-set effects in their input and hidden layers, respectively. Furthermore, the weights and connections of the network are regarded as random variables with prior distributions reflecting the manifestation of genetic effects at various genomic scales. However, its application in genomic prediction has yet to be explored.ResultsThis study extended the BANNs framework to the area of genomic selection and explored the optimal SNP-set partitioning strategies by using dairy cattle datasets. The SNP-sets were partitioned based on two strategies–gene annotations and 100 kb windows, denoted as BANN_gene and BANN_100kb, respectively. The BANNs model was compared with GBLUP, random forest (RF), BayesB and BayesCπ through five replicates of five-fold cross-validation using genotypic and phenotypic data on milk production traits, type traits, and one health trait of 6,558, 6,210 and 5,962 Chinese Holsteins, respectively. Results showed that the BANNs framework achieves higher genomic prediction accuracy compared to GBLUP, RF and Bayesian methods. Specifically, the BANN_100kb demonstrated superior accuracy and the BANN_gene exhibited generally suboptimal accuracy compared to GBLUP, RF, BayesB and BayesCπ across all traits. The average accuracy improvements of BANN_100kb over GBLUP, RF, BayesB and BayesCπ were 4.86%, 3.95%, 3.84% and 1.92%, and the accuracy of BANN_gene was improved by 3.75%, 2.86%, 2.73% and 0.85% compared to GBLUP, RF, BayesB and BayesCπ, respectively across all seven traits. Meanwhile, both BANN_100kb and BANN_gene yielded lower overall mean square error values than GBLUP, RF and Bayesian methods.ConclusionOur findings demonstrated that the BANNs framework performed better than traditional genomic prediction methods in our tested scenarios, and might serve as a promising alternative approach for genomic prediction in dairy cattle.

Loss-Based Variational Bayes Prediction

We propose a new approach to Bayesian prediction that caters for models with a large number of parameters and is robust to model misspecification. Given a class of high-dimensional (but parametric) predictive models, this new approach constructs a posterior predictive using a variational approximation to a generalized posterior that is directly focused on predictive accuracy. The theoretical behavior of the new prediction approach is analyzed and a form of optimality demonstrated. Applications to both simulated and empirical data using high-dimensional Bayesian neural network and autoregressive mixture models demonstrate that the approach provides more accurate results than various alternatives, including misspecified likelihood-based predictions. Supplementary materials for this article are available online.

From drought hazard to risk: A spring wheat vulnerability assessment in the Canadian Prairies

The Canadian drought monitor (CDM) provides monthly assessments of drought severity but does not provide guidance on the potential agricultural impacts. This study extends the CDM to a comprehensive risk assessment framework by assessing the vulnerability of spring wheat to drought timing and severity in the Canadian Prairies. To achieve this, a bayesian neural network (BNN) model was developed that predicts high-resolution wheat yields and uncertainties using the CDM and normalized difference vegetation index (NDVI) data from 2010 to 2021. Model performance is comparable to previous studies, with errors ranging from 13% to 18% of mean yields and R2 values from 0.43 to 0.70 that peak in early August. The model was then applied under various drought conditions to evaluate how different drought timings and severities affect yield vulnerability. The results demonstrate that drought exposure from June to August reduces wheat yields in proportion to drought severity while abnormally dry conditions may have a positive impact of yield, particularly in August. Yield uncertainty decreases substantially from June to July while yield prediction accuracy increases, suggesting that mitigation efforts are most effective prior to July. The final component of the research involved creating operational risk maps that integrate the yield vulnerability assessment with the CDM. These maps were applied in a case study to demonstrate their utility in providing actionable, real-time information for agricultural stakeholders. Overall, this study contributes to the body of knowledge regarding crop sensitivities to varying drought conditions and represents a methodological advancement by enabling high-resolution, operational agricultural risk assessments at scale. By integrating detailed vulnerability data with drought hazard monitoring from the CDM, the information provided by these risk maps can be leveraged by agricultural stakeholders in near real-time to make well-informed decisions that mitigate the impacts of drought on wheat production, enhancing the resilience of agricultural systems to climatic variability.

Pipeline corrosion prediction and uncertainty analysis with an ensemble Bayesian neural network approach

This study developed an ensemble Bayesian Neural Network (BNN) model for pipeline corrosion prediction incorporating uncertainty analysis. The performance of the proposed BNN model was compared with different empirical and data-driven models. Shapley Additive Explanation (SHAP) values were used to quantify the importance of input factors. Results indicate that the ensemble BNN model demonstrates superior prediction accuracy, achieving the highest R2 value of 0.949, with the lowest Mean Absolute Error (MAE) of 0.33 mm and Root Mean Square Error (RMSE) of 0.449 mm on corrosion depth. It outperforms other models in predicting maximum corrosion depth and uniquely captures uncertainty information, which deterministic models cannot provide. Key factors influencing corrosion include chloride content, pH, pipe-to-soil potential, and pipeline age. Specifically, conditions such as a pipe-to-soil potential exceeding − 0.85 V, a pipeline age over 20 years, and pH values below 7 are found to significantly accelerate corrosion. The model also illustrates the effects of different influencing factors on corrosion growth uncertainty. Higher chloride content, lower pH values, and higher pipe-to-soil potentials may expand the uncertainty bounds. The proposed model presents the dynamic changes in prediction uncertainty over time, which is usually ignored in previous models. It can improve the precision and reliability of long-term predictions concerning pipeline corrosion depth. This knowledge of corrosion uncertainty is crucial for making informed decisions about pipeline inspection and maintenance strategies.

Liquid droplet entrainment in an annular flow boiling regime—A Bayesian regularization algorithm based study

Accurate prediction of the entrained liquid droplet fraction in an annular two-phase flow regime plays a crucial role for estimating the dryout type critical heat flux to identify the optimized flow characteristics in the thermal systems across different industries. Existing studies have provided different correlations based on the limited experimental data. However, these correlations are applicable to certain operating conditions. Therefore, the present study aims at applying a deep learning method, specifically an artificial neural network (ANN), to enhance the prediction of the entrained liquid droplet fraction. Experimental data from various works on annular flow, covering a wide spectrum of pressure and flow conditions, are utilized for training the ANN model. Eight input variables, viz, superficial gas velocity (JSG), superficial liquid velocity (JSL), gas viscosity (μG), liquid viscosity (μL), gas density (ρG), liquid density (ρL), pipe diameter (d) and liquid surface tension (σLV) are considered as input features. The entrained liquid droplet fraction is the single output feature. The present model employs the Bayesian regularization backpropagation algorithm for training. The present ANN model is compared against the performance of linear regression, decision tree and support vector machine algorithms, and found that the performance of the present Bayesian regularization neural network (BRNN) model is superior within ∼7.5% deviation. Further, the BRNN model is coupled with the film mass flow rate model to obtain the axial variation of the liquid film mass flow rate and good agreement is noticed when compared against the experimental data.

Probabilistic reach-avoid for Bayesian neural networks

Model-based reinforcement learning seeks to simultaneously learn the dynamics of an unknown stochastic environment and synthesise an optimal policy for acting in it. Ensuring the safety and robustness of sequential decisions made through a policy in such an environment is a key challenge for policies intended for safety-critical scenarios. In this work, we investigate two complementary problems: first, computing reach-avoid probabilities for iterative predictions made with dynamical models, with dynamics described by Bayesian neural network (BNN); second, synthesising control policies that are optimal with respect to a given reach-avoid specification (reaching a “target” state, while avoiding a set of “unsafe” states) and a learned BNN model. Our solution leverages interval propagation and backward recursion techniques to compute lower bounds for the probability that a policy's sequence of actions leads to satisfying the reach-avoid specification. Such computed lower bounds provide safety certification for the given policy and BNN model. We then introduce control synthesis algorithms to derive policies maximizing said lower bounds on the safety probability. We demonstrate the effectiveness of our method on a series of control benchmarks characterized by learned BNN dynamics models. On our most challenging benchmark, compared to purely data-driven policies the optimal synthesis algorithm is able to provide more than a four-fold increase in the number of certifiable states and more than a three-fold increase in the average guaranteed reach-avoid probability.

Optimisation and Calibration of Bayesian Neural Network for Probabilistic Prediction of Biogas Performance in an Anaerobic Lagoon.

This study aims to enhance diagnostic capabilities for optimising the performance of the anaerobic sewage treatment lagoon at Melbourne Water's Western Treatment Plant (WTP) through a novel machine learning (ML)-based monitoring strategy. This strategy employs ML to make accurate probabilistic predictions of biogas performance by leveraging diverse real-life operational and inspection sensor and other measurement data for asset management, decision making, and structural health monitoring (SHM). The paper commences with data analysis and preprocessing of complex irregular datasets to facilitate efficient learning in an artificial neural network. Subsequently, a Bayesian mixture density neural network model incorporating an attention-based mechanism in bidirectional long short-term memory (BiLSTM) was developed. This probabilistic approach uses a distribution output layer based on the Gaussian mixture model and Monte Carlo (MC) dropout technique in estimating data and model uncertainties, respectively. Furthermore, systematic hyperparameter optimisation revealed that the optimised model achieved a negative log-likelihood (NLL) of 0.074, significantly outperforming other configurations. It achieved an accuracy approximately 9 times greater than the average model performance (NLL = 0.753) and 22 times greater than the worst performing model (NLL = 1.677). Key factors influencing the model's accuracy, such as the input window size and the number of hidden units in the BiLSTM layer, were identified, while the number of neurons in the fully connected layer was found to have no significant impact on accuracy. Moreover, model calibration using the expected calibration error was performed to correct the model's predictive uncertainty. The findings suggest that the inherent data significantly contribute to the overall uncertainty of the model, highlighting the need for more high-quality data to enhance learning. This study lays the groundwork for applying ML in transforming high-value assets into intelligent structures and has broader implications for ML in asset management, SHM applications, and renewable energy sectors.

Application of Response Surface-Corrected Finite Element Model and Bayesian Neural Networks to Predict the Dynamic Response of Forth Road Bridges under Strong Winds.

With the rapid development of big data, the Internet of Things (IoT), and other technological advancements, digital twin (DT) technology is increasingly being applied to the field of bridge structural health monitoring. Achieving the precise implementation of DT relies significantly on a dual-drive approach, combining the influence of both physical model-driven and data-driven methodologies. In this paper, two methods are proposed to predict the displacement and dynamic response of structures under strong winds, namely, a Bayesian Neural Network (BNN) model based on Bayesian inference and a finite element model (FEM) method modified based on genetic algorithms (GAs) and multi-objective optimization (MOO) using response surface methodology (RSM). The characteristics of these approaches in predicting the dynamic response of large-span bridges are explored, and a comparative analysis is conducted to evaluate their differences in computational accuracy, efficiency, model complexity, interpretability, and comprehensiveness. The characteristics of the two methods were evaluated using data collected on the Forth Road Bridge (FRB) as an example under unusual weather conditions with strong wind action. This work proposes a dual-driven approach, integrating machine learning and FEM with GNSS and Earth Observation for Structural Health Monitoring (GeoSHM), to bridge the gap in the limited application of dual-driven methods primarily applied for small- and medium-sized bridges to large-span bridge structures. The research results show that the BNN model achieved higher R2 values for predicting the Y and Z displacements (0.9073 and 0.7969, respectively) compared to the FEM model (0.6167 and 0.6283). The BNN model exhibited significantly faster computation, taking only 20 s, while the FEM model required 5 h. However, the physical model provided higher interpretability and the ability to predict the dynamic response of the entire structure. These findings help to promote the further integration of these two approaches to obtain an accurate and comprehensive dual-driven approach for predicting the structural dynamic response of large-span bridge structures affected by strong wind loading.

Integrating Dropout and Kullback-Leibler Regularization in Bayesian Neural Networks for improved uncertainty estimation in Regression

The objective of the study is to enhance uncertainty prediction in regression problems by introducing a revolutionary Bayesian Neural Network (BNN) model. Experimental results reveal significant improvements in uncertainty prediction and point forecasts with the integrated BNN model compared to the plain BNN. Performance metrics, including mean squared error (MSE), mean absolute error (MAE), and R-squared (R²), demonstrate superior results for the proposed BNN. The experimental results show that for plain BNN, MSE is 87.3, MAE is 6.62 and R2 is −0.0492, whereas for the proposed BNN model MSE was found to be 44.64, MAE is 4.4 and R2 is 0.46. This research brings a fresh approach to Bayesian Neural Networks by incorporating both dropout and KL regularization techniques, resulting in a powerful tool for handling regression tasks with certainty. By combining these techniques, the study enhances model stability, avoids overfitting, and achieves more reliable uncertainty estimation. This study adds to our knowledge of uncertainty-aware machine learning models and offers a valuable solution for accurately assessing uncertainty in various applications.•The innovative BNN model merges the power of Bayesian principles with the effectiveness of dropout and KL regularization.•To test and refine our model, the study utilizes the Boston Housing dataset for both training and evaluation purposes.

Uncertainty quantification in multiaxial fatigue life prediction using Bayesian neural networks

In practical engineering scenarios, unforeseen failures may result in significant costs and risks. Assessing the uncertainty related to multiaxial fatigue life prediction is crucial for engineering applications. However, establishing a reliable multiaxial fatigue life prediction model remains challenging due to uncertainties in physical models, material properties, and measurement data. This paper proposes an effective Bayesian Neural Network (BNN) model for quantifying the uncertainty in multiaxial fatigue life prediction. Fatigue parameters in the multiaxial fatigue life prediction model are used as inputs for the BNN. The BNN method utilizes No-U-turn Sampler (NUTS) and Automatic Differentiation Variational Inference (ADVI) to address the inference problem in the model and perform life predictions for unknown non-proportional loading paths and different materials. Validation using experimental data from six different materials confirms the effectiveness of the BNN model. Experiments show that the performance of BNN models is beneficial in most cases compared to classical ML models, assessed through different error standards, statistical tests, Taylor diagrams, uncertainty analysis, and scatter index analysis. The proposed method provides a clear understanding of the uncertainty in multiaxial fatigue life prediction, offering a promising approach for simulating fatigue life under multiaxial loading in engineering applications.

Aviation Safety Risk Analysis Based on Bayesian Network Modeling

The occurrence of serious flight accidents not only brings huge economic losses to airlines but also poses a great threat to passengers' lives. The occurrence of serious flight accidents will not only bring great economic losses to airlines but also cause great threats to the lives of passengers. Therefore, in this paper, based on the flight records of different flight crews, flight routes, airports, and specific flight conditions, aircraft safety risk analysis is conducted to calculate and evaluate the risk propensity. Firstly, the data are preprocessed, and then the QAR data are clustered to extract the key data items affecting the safe flight of the aircraft; then the data are transformed to the attached data, and a hierarchical clustering model is established based on the K-means clustering algorithm, which classifies the factors affecting the safety quality of the aircraft flight into the three main aspects of environmental factors, crew maneuvering, and aircraft state, and the key parameters are weighted by the Random Forest algorithm. weights of the key parameters. Then, based on the 3 criterion, the outliers exceeding 3 were selected as the dependent variables leading to the occurrence of deviation; based on the three-layer model, a Bayesian neural network model based on the three-layer model, and then determine the Bayesian nodes; finally, calculate the Bayesian node probability, and then provide a reasonable quantitative description of the flight maneuver.

A Latent Factor-Based Bayesian Neural Networks Model in Cloud Platform for Used Car Price Prediction

The selling price of a used car can be predicted based on its historical information. Accurate and reasonable used car price evaluation will be able to promote the healthy progress of the used car industry. Current used car price prediction models are troubled by data quality and the inability to provide estimates of uncertain information, and the prediction accuracy cannot meet the needs of real-world scenarios. In this work, we propose a price prediction model based on a Bayesian neural networks with latent factor (LFBNN) in a cloud platform, which is capable of performing latent factor extraction operations on structured data of used cars as a way to remove the effect of noise in the dataset on the model performance. Moreover, the weight parameters of the Bayesian neural networks (NNs) are represented as probability distributions, which is equivalent to introducing uncertainty and acting as a regularization effect compared to a NN with fixed weights, thus making it possible to alleviate the overfitting problem. The LFBNN model uses a cloud platform for data transmission and storage, enabling it to run faster. Compared with the current benchmark models, the model proposed in this work achieves excellent experimental results on two real datasets. The experimental results on the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Car_{1}$</tex-math></inline-formula> dataset are 1.498 for mse, 1.224 for rmse, and 0.995 for MAE, and 1.519 for mse, 1.232 for rmse, and 1.002 for MAE on the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Car_{2}$</tex-math></inline-formula> dataset.

Direct short-term net load forecasting in renewable integrated microgrids using machine learning: A comparative assessment

Modern microgrids require accurate net load forecasting (NLF) for optimal operation and management at high shares of renewable energy sources. Machine learning (ML) principles can be used to develop precise and reliable NLF models. This paper evaluates the performance of different ML models, that are optimally trained using supervised learning regimes, for direct short-term net load forecasting (STNLF) in renewable microgrids. Different categories of ML models, such as neural network, ensemble, linear regression, nearest neighbor, and support vector machine were used. The comparative assessment was conducted utilizing historical net load, meteorological, and time-related categorical data acquired from the renewable integrated microgrid of the University of Cyprus in Nicosia, Cyprus. The results showed that all STNLF ML models achieved normalized root mean square error (nRMSE) values below 10%. Amongst the investigated models, the Bayesian neural network (BNN) presented the highest forecasting accuracy, exhibiting a daily average error of 3.58%. In addition, the BNN model yielded robust forecasts regardless of the season and weather conditions. Finally, the results demonstrated that optimally constructed ML models can be applied to provide STNLF in renewable integrated microgrids, which can be used by microgrid operators to efficiently control and manage their assets.

Addressing posterior collapse by splitting decoders in variational recurrent autoencoders

Variational recurrent autoencoder model (VRAE) is an appealing technique for capturing the variabilities underlying complex sequential data, which is realized by introducing high-level latent random variables as hidden states. Existing models suffer from the well-known ‘posterior collapse’ problem, meaning that a powerful autoregressive decoder equipped in the model itself could capture all the variabilities and hence leave the latent variables learning nothing from the data. From the perspective of model training, the posterior collapse problem can result in a very low Kullback–Leibler divergence (KL-divergence) value, which means the posteriors of the latent variables tend to be just the priors. In this paper, we address this problem by proposing a Bayesian variational recurrent neural network (BVRNN) model, in which two additional decoders are added into the original VRAE. These extra decoders can force the latent variables to learn meaningful knowledge during the training process. We conduct experiments on MNIST and Fashion-MNIST dataset. The experimental results show that the proposed model outperforms several baseline models. We further adapt the proposed model to a very challenging task in natural language processing, namely Named-Entity Recognition (NER). Experimental results show that our model is competitive to the state-of-the-art models on NER.

GlAIcomics: a deep neural network classifier for spectroscopy-augmented mass spectrometric glycans data

Carbohydrate sequencing is a formidable task identified as a strategic goal in modern biochemistry. It relies on identifying a large number of isomers and their connectivity with high accuracy. Recently, gas phase vibrational laser spectroscopy combined with mass spectrometry tools have been proposed as a very promising sequencing approach. However, its use as a generic analytical tool relies on the development of recognition techniques that can analyse complex vibrational fingerprints for a large number of monomers. In this study, we used a Bayesian deep neural network model to automatically identify and classify vibrational fingerprints of several monosaccharides. We report high performances of the obtained trained algorithm (GlAIcomics), that can be used to discriminate contamination and identify a molecule with a high degree of confidence. It opens the possibility to use artificial intelligence in combination with spectroscopy-augmented mass spectrometry for carbohydrates sequencing and glycomics applications.