Bayesian Learning Model Research Articles

Exploring cognitive individuality and the underlying creativity in statistical learning and phase entrainment.

Statistical learning starts at an early age and is intimately linked to brain development and the emergence of individuality. Through such a long period of statistical learning, the brain updates and constructs statistical models, with the model's individuality changing based on the type and degree of stimulation received. However, the detailed mechanisms underlying this process are unknown. This paper argues three main points of statistical learning, including 1) cognitive individuality based on "reliability" of prediction, 2) the construction of information "hierarchy" through chunking, and 3) the acquisition of "1-3Hz rhythm" that is essential for early language and music learning. We developed a Hierarchical Bayesian Statistical Learning (HBSL) model that takes into account both reliability and hierarchy, mimicking the statistical learning processes of the brain. Using this model, we conducted a simulation experiment to visualize the temporal dynamics of perception and production processes through statistical learning. By modulating the sensitivity to sound stimuli, we simulated three cognitive models with different reliability on bottom-up sensory stimuli relative to top-down prior prediction: hypo-sensitive, normal-sensitive, and hyper-sensitive models. We suggested that statistical learning plays a crucial role in the acquisition of 1-3 Hz rhythm. Moreover, a hyper-sensitive model quickly learned the sensory statistics but became fixated on their internal model, making it difficult to generate new information, whereas a hypo-sensitive model has lower learning efficiency but may be more likely to generate new information. Various individual characteristics may not necessarily confer an overall advantage over others, as there may be a trade-off between learning efficiency and the ease of generating new information. This study has the potential to shed light on the heterogeneous nature of statistical learning, as well as the paradoxical phenomenon in which individuals with certain cognitive traits that impede specific types of perceptual abilities exhibit superior performance in creative contexts.

The Impact of Risk Cycles on Business Cycles: A Historical View

Abstract We investigate the effects of financial risk cycles on business cycles, using a panel spanning 73 countries since 1900. Agents use a Bayesian learning model to form their beliefs about risk. We construct a proxy of these beliefs and show that perceived low risk encourages risk-taking, augmenting growth at the cost of accumulating financial vulnerabilities, and, therefore, a reversal in growth follows. The reversal is particularly pronounced when the low-risk environment persists and credit growth is excessive. Global risk cycles have a stronger effect on growth than local risk cycles via their impact on capital flows, investment, and debt-issuer quality. Authors have furnished an Internet Appendix, which is available on the Oxford University Press Web site next to the link to the final published paper online.

Learning from Polls During Electoral Campaigns

Voters’ beliefs about the strength of political parties are a central part of many foundational political science theories. In this article, we present a dynamic Bayesian learning model that allows us to study how voters form these beliefs by learning from pre-election polls over the course of an election campaign. In the model, belief adaptation to new polls can vary due to the perceived precision of the poll or the reliance on prior beliefs. We evaluate the implications of our model using two experiments. We find that respondents update their beliefs assuming that the polls are relatively imprecise but still weigh them more strongly than their priors. Studying implications for motivational learning by partisans, we find that varying adaptation works through varying reliance on priors and not necessarily by discrediting a poll’s precision. The findings inform our understanding of the consequences of learning from polls during political campaigns and motivational learning in general.

Microblogging Replies and Opinion Polarization: A Natural Experiment

In recent years, there has been a heated discussion on opinion polarization on social media platforms. Extant research attributes the emergence of echo chambers to higher exposure to information from users’ existing social networks, which consists of like-minded others and argues that the provision of information from outside users’ networks could alleviate opinion polarization. In this paper, we formulate a hierarchical Bayesian learning model to investigate the impact of replies, one of the main channels for information outside of users’ networks, on opinion polarization. We leverage a unique natural experiment contained in the data from a leading microblogging website in China in which the reply function was shut down for three days. This setting allows us to identify the impact of replies from that of peer microblogs. We found that shutting down reply function reduced sentiment polarization on the microblogging site. In addition, this effect was more significant for individuals with higher social media participation. The results of this study shed light on marketing campaign strategies as well as the ways in which platform design can reduce polarization.

Robust Incremental Bayesian Learning Based Online Flux Linkage Estimation for PMSM Drives

For the permanent magnet synchronous machine (PMSM), parameter estimation can be greatly affected by the measurement uncertainty, but few efforts are made to reduce the uncertainty level for estimation performance improvement. Therefore, this article proposes an efficient and robust incremental Bayesian learning approach for PMSM parameter estimation. The measurement uncertainty is evaluated to guide the selection of informative measurements, and the estimation uncertainty is provided to indicate the confidence in using the estimated results. Specifically, a Bayesian learning strategy with a layered noise model is proposed for nonlinear flux linkage estimation. The measurement uncertainty level is estimated from the proposed Bayesian learning model, which is utilized to adaptively select the most informative data and delete the noninformative data for parameter estimation. This contributes to improving estimation accuracy and computation efficiency. Moreover, the estimation uncertainty is also determined by the proposed model, which can be used to indicate if the estimated results can be trusted and utilized in practical applications. The proposed approach is evaluated on a laboratory interior PMSM under various operating conditions.

Atypical prediction error learning is associated with prodromal symptoms in individuals at clinical high risk for psychosis

Reductions in the auditory mismatch negativity (MMN) have been well-demonstrated in schizophrenia rendering it a promising biomarker for understanding the emergence of psychosis. According to the predictive coding theory of psychosis, MMN impairments may reflect disturbances in hierarchical information processing driven by maladaptive precision-weighted prediction errors (pwPEs) and enhanced belief updating. We applied a hierarchical Bayesian model of learning to single-trial EEG data from an auditory oddball paradigm in 31 help-seeking antipsychotic-naive high-risk individuals and 23 healthy controls to understand the computational mechanisms underlying the auditory MMN. We found that low-level sensory and high-level volatility pwPE expression correlated with EEG amplitudes, coinciding with the timing of the MMN. Furthermore, we found that prodromal positive symptom severity was associated with increased expression of sensory pwPEs and higher-level belief uncertainty. Our findings provide support for the role of pwPEs in auditory MMN generation, and suggest that increased sensory pwPEs driven by changes in belief uncertainty may render the environment seemingly unpredictable. This may predispose high-risk individuals to delusion-like ideation to explain this experience. These results highlight the value of computational models for understanding the pathophysiological mechanisms of psychosis.

Uncertainty-aware mixed-variable machine learning for materials design

Data-driven design shows the promise of accelerating materials discovery but is challenging due to the prohibitive cost of searching the vast design space of chemistry, structure, and synthesis methods. Bayesian optimization (BO) employs uncertainty-aware machine learning models to select promising designs to evaluate, hence reducing the cost. However, BO with mixed numerical and categorical variables, which is of particular interest in materials design, has not been well studied. In this work, we survey frequentist and Bayesian approaches to uncertainty quantification of machine learning with mixed variables. We then conduct a systematic comparative study of their performances in BO using a popular representative model from each group, the random forest-based Lolo model (frequentist) and the latent variable Gaussian process model (Bayesian). We examine the efficacy of the two models in the optimization of mathematical functions, as well as properties of structural and functional materials, where we observe performance differences as related to problem dimensionality and complexity. By investigating the machine learning models’ predictive and uncertainty estimation capabilities, we provide interpretations of the observed performance differences. Our results provide practical guidance on choosing between frequentist and Bayesian uncertainty-aware machine learning models for mixed-variable BO in materials design.

Gradient boosting and bayesian network machine learning models predict aflatoxin and fumonisin contamination of maize in Illinois - First USA case study.

Mycotoxin contamination of corn results in significant agroeconomic losses and poses serious health issues worldwide. This paper presents the first report utilizing machine learning and historical aflatoxin and fumonisin contamination levels in-order-to develop models that can confidently predict mycotoxin contamination of corn in Illinois, a major corn producing state in the USA. Historical monthly meteorological data from a 14-year period combined with corresponding aflatoxin and fumonisin contamination data from the State of Illinois were used to engineer input features that link weather, fungal growth, and aflatoxin production in combination with gradient boosting (GBM) and bayesian network (BN) modeling. The GBM and BN models developed can predict mycotoxin contamination with overall 94% accuracy. Analyses for aflatoxin and fumonisin with GBM showed that meteorological and satellite-acquired vegetative index data during March significantly influenced grain contamination at the end of the corn growing season. Prediction of high aflatoxin contamination levels was linked to high aflatoxin risk index in March/June/July, high vegetative index in March and low vegetative index in July. Correspondingly, high levels of fumonisin contamination were linked to high precipitation levels in February/March/September and high vegetative index in March. During corn flowering time in June, higher temperatures range increased prediction of high levels of fumonisin contamination, while high aflatoxin contamination levels were linked to high aflatoxin risk index. Meteorological events prior to corn planting in the field have high influence on predicting aflatoxin and fumonisin contamination levels at the end of the year. These early-year events detected by the models can directly assist farmers and stakeholders to make informed decisions to prevent mycotoxin contamination of Illinois grown corn.

Inferential power in identifying frequency-dependent social learning strengthened by increasing behavioural options.

Two-option choice experimental designs are the most commonly employed framework for identifying evidence of social learning or social learning strategies in captive and wild populations. In nature, however, animals often choose from more than two behaviours, and multiple innovations may arise simultaneously. Studies of animal social learning are often constrained by small sample sizes, which limit researchers' ability to convincingly identify the proposed social learning strategy responsible for behavioural choice. In this study, I examine whether expanding behavioural options from k=2 to k > 2 and increasing sample size affects inferential power in identifying social learning strategies. I focus on three frequency-dependent learning strategies: conformist transmission, unbiased transmission and anti-conformist transmission. I simulate 100 datasets for 72 parameter combinations, yielding 7200 simulations. I evaluate number of options (k=2, 3, 4, 5), population size (n=5, 10, 25, 50, 100, 250) and the logarithmic strength of frequency dependence (log(f)=log(1∕3), log(1), log(3)). I then fit a Bayesian social learning model to simulated data to evaluate the percent of the posterior consistent with type of frequency dependence, posterior standard deviations, highest posterior density intervals and posterior medians relative to the true simulated value of log(f). I show that increasing the number of options an animal can choose from increases the accuracy and certainty of identifying the type and magnitude of frequency-dependent social learning. These effects are particularly pronounced at small to intermediate sample sizes, which are common in empirical studies of animal social learning. These findings suggest that knowing what an animal did not choose is equally important as knowing what an animal did choose when identifying social learning strategies. By strategically increasing the number of behaviours from which an animal can choose, researchers can increase inferential power in identifying social learning strategies without increasing sample size, that is, adding additional animals or collecting more data.

Performance Comparison of Bayesian Deep Learning Model and Traditional Bayesian Neural Network in Short-Term PV Interval Prediction

The intermittence and fluctuation of renewable energy bring significant uncertainty to the power system, which enormously increases the operational risks of the power system. The development of efficient interval prediction models can provide data support for decision making and help improve the economy and reliability of energy interconnection operation. The performance of Bayesian deep learning models and Bayesian shallow neural networks in short-term interval prediction of photovoltaic power is compared in this study. Specifically, an LSTM Approximate Bayesian Neural Network model (ABNN-I) is built on the basis of the deep learning and Monte Carlo Dropout method. Meanwhile, a Feedforward Bayesian Neural Network (ABNN-II) model is introduced by Feedforward Neural Network and the Markov Chain Monte Carlo method. To better compare and verify the interval prediction capability of the ABNN models, a novel clustering method with three-dimensional features which include the number of peaks and valleys, the average power value, and the non-stationary measurement coefficient is proposed for generating sunny and non-sunny clustering sets, respectively. Results show that the ABNN-I model has an excellent performance in the field of photovoltaic short-term interval forecasting. At a 95% confidence level, the interval coverage from ABNN-I to ABNN-II can be increased by up to 3.1% and the average width of the interval can be reduced by 56%. Therefore, with the help of the high computational capacity of deep learning and the inherent ability to quantify uncertainty of the interval forecast from Bayesian methods, this research provides high-quality interval prediction results for photovoltaic power prediction and solves the problem of difficult modeling for over-fitting that exists in the training process, especially on the non-sunny clustering sets.

Uncertainty quantification in super-resolution guided wave array imaging using a variational Bayesian deep learning approach

Super-resolution guided wave array imaging has shown to be a feasible tool to detect and image sub-wavelength defects. However, research gaps remain in effectively quantifying and understanding the uncertainties in super-resolution imaging. In this study, we present a Bayesian deep learning approach to quantify and interpret various uncertainties in super-resolution guided wave array imaging. Specifically, we implement a Monte Carlo (MC) dropout scheme in the multi-scale deep learning models for approximate Bayesian inference to effectively quantify uncertainties in super-resolution subwavelength defect imaging. Furthermore, we decompose the total predictive uncertainty into distinct uncertainty sources: aleatoric uncertainty inherent in the data and epistemic uncertainty associated with the Bayesian deep learning model. From the experimental study, we observe that the two types of uncertainty (aleatoric and epistemic) in super-resolution guided wave array imaging can be successfully quantified using the multi-scale Bayesian deep learning approach. We further discuss the effectiveness of the Bayesian deep learning approach in small-data cases and compare the super-resolution imaging performance with a non-Bayesian (deterministic) deep learning approach.

Application of a Bayesian Network Learning Model to Predict Longitudinal Trajectories of Executive Function Difficulties in Elementary School Students.

Executive function is the mental ability to modulate behavior or thinking to accomplish a task. This is developmentally important for children’s academic achievements and ability to adjust to school. We classified executive function difficulties (EFDs) in longitudinal trajectories in Korean children from 7 to 10 years old. We found predictors of EFDs using latent class growth analysis and Bayesian network learning methods with Panel Study data. Three types of latent class models of executive function difficulties were identified: low, intermediate, and high EFDs. The modeling performance of the high EFD group was excellent (AUC = .91), and the predictors were the child’s gender, temperamental emotionality, happiness, DSM (Diagnostic and Statistical Manual of Mental Disorders) anxiety problems, and the mother’s depression as well as coparenting conflict recognized by the mother. The results show that using latent class growth analysis and Bayesian network learning are helpful in classifying the longitudinal EFD patterns in elementary school students. Furthermore, school-age EFD is affected by emotional problems in parents and children that continue from early life. These findings can support children’s development and prevent risk by preclassifying children who may experience persistent EFD and tracing causes.

Structural health monitoring data cleaning based on Bayesian robust tensor learning

Data cleaning is necessary to obtain high-quality monitoring data, which consists of two parts: anomaly identification and repair. A high-quality dataset is always used to repair data anomalies in current research, which may be difficult to obtain in practice because of the large number of data anomalies existing in real data. Besides, outliers, baseline shifts, and noise in dataset can lead to wrong missing restoration results. Therefore, the anomaly identification and location are needed to remove outliers, baseline shifts, and noise to regard the repair of these anomalies as missing data restoration. However, the identification of multiple data anomalies is time consuming and challenging. To this end, we propose a Bayesian robust tensor learning model to reconstruct the monitoring data tensor by extracting potential data features of the spatiotemporal tensor. A mixture of Laplace distribution with generalized inverse Gaussian distribution is used to model noise to enhance robustness. The tensor formed from the data containing anomalies can be input directly into the model without the identification and location of the data anomalies in advance. All data anomalies are repaired during the tensor reconstruction process. In addition, outliers, baseline shifts, and noise do not interfere with the missing restoration results owing to the robustness of the model. Furthermore, different cleaning strategies are proposed for dynamic and static response data. The data performance is verified using the real monitoring data of a concrete box girder bridge. The results show that the proposed method exhibits good data-cleaning performance when dealing with data containing a large number of data anomalies. In addition, as the abnormal rate increases, the proposed method still exhibits good data-cleaning performance.

Automated Posture Positioning for High Precision 3D Scanning of a Freeform Design using Bayesian Optimization

Three-dimensional scanning is widely used for the dimension measurements of physical objects with freeform designs. The output point cloud is flexible enough to provide a detailed geometric description for these objects. However, geometric accuracy and precision are still debatable for this scanning process. Uncertainties are ubiquitous in geometric measurement due to many physical factors. One potential factor is the object’s posture in the scanning region. The posture of target positioning on the scanning platform could influence the normal of the scanning points, which could further affect the measurement variances. This paper first investigates the geometric and spatial factors that could potentially influence scanning variance. This functional relationship is modeled as a Bayesian extreme learning model, which is later utilized to find the object’s optimal posture for variance reduction. A Bayesian optimization approach is proposed to solve this minimization problem. Case studies are presented to validate the proposed methodology.

Comparison of Bayesian, Frequentist and Machine learning models for predicting the two-year mortality of patients diagnosed with squamous cell carcinoma of the oral cavity

BackgroundStatistical models developed in frequentist and Bayesian context along with machine learning algorithms can encompass the multifactorial effect of the prognostic factors in predicting the outcome. This paper is aimed to compare the effect estimates and predictive performance of Bayesian, frequentist and machine learning algorithm in predicting the two-year mortality of patients diagnosed with squamous cell carcinoma (SCC) of oral cavity. Materials and methodsLogistic regression (LR), Binary Discriminant analysis (BDA), Naïve Bayes (NB), Bayesian regression (BLR), K nearest neighbor (KNN), Artificial neural network (ANN) and Random Forest (RF) models were built. The effect estimate of each prognostic factor was estimated and compared by LR and BLR model. 10-fold cross-validation was performed for internal validation of the models. The predictive performance of the models was assessed and compared. ResultsBLR model had lower and narrower effect estimates in comparison to LR model. Age and smoking are the biggest prognostic risk factors for SCC whereas surgery had the best response amongst the mode of treatment. Random forest had an AUROC of 0.86 (0.82, 0.90) whereas it was estimated to be 0.77 (0.71, 0.82) for both BLR and LR models. ConclusionsBLR model had better precision in estimating the effect size of prognostic factors and can be an alternative for predicting mortality in patients with SCC of oral cavity. Machine learning classifiers had the best predictive ability as compared to statistical models.

The impact of risk cycles on business cycles: a historical view

We investigate the effects of financial risk cycles on business cycles, using a panel spanning 73 countries since 1900. Agents use a Bayesian learning model to form their beliefs on risk. We construct a proxy of these beliefs and show that perceived low risk encourages risk-taking, augmenting growth at the cost of accumulating financial vulnerabilities, and therefore, a reversal in growth follows. The reversal is particularly pronounced when the low-risk environment persists and credit growth is excessive. Global-risk cycles have a stronger effect on growth than local-risk cycles via their impact on capital flows, investment, and debt-issuer quality.

The origin and evolution of open habitats in North America inferred by Bayesian deep learning models

Some of the most extensive terrestrial biomes today consist of open vegetation, including temperate grasslands and tropical savannas. These biomes originated relatively recently in Earth’s history, likely replacing forested habitats in the second half of the Cenozoic. However, the timing of their origination and expansion remains disputed. Here, we present a Bayesian deep learning model that utilizes information from fossil evidence, geologic models, and paleoclimatic proxies to reconstruct paleovegetation, placing the emergence of open habitats in North America at around 23 million years ago. By the time of the onset of the Quaternary glacial cycles, open habitats were covering more than 30% of North America and were expanding at peak rates, to eventually become the most prominent natural vegetation type today. Our entirely data-driven approach demonstrates how deep learning can harness unexplored signals from complex data sets to provide insights into the evolution of Earth’s biomes in time and space.

A multi-output deep learning model based on Bayesian optimization for sequential train delays prediction

ABSTRACT Accurate train arrival delay predictions can provide timely information for passengers and train dispatchers. Previous work mainly focused on predicting the delay of a single train, which is not enough to assist dispatchers, because making more comprehensive decisions considering more trains needs more future delay information of a group of trains. Therefore, this paper proposes a Bayesian optimization-based multi-output deep learning model, which includes a fully connected neural network (FCNN) and two long–short-term memory (LSTM) components, to predict the arrival delays of multiple trains simultaneously. The proposed model is calibrated and validated with the train operation data from the Wuhan–Guangzhou (W-G) high-speed railway. The test results show that the mean absolute error and the root mean square error of the proposed model is 1.379 and 2.021 min, over the four subsequent trains. Moreover, the proposed model outperforms the standard train delay prediction benchmark model.

Uncertainty utilization in fault detection using Bayesian deep learning

Up to now, extensive literature on the usage of deep learning in manufacturing can be found. Though, actual usage of deep learning in manufacturing sites is somehow restrained by the quality of the obtained data, especially for machine failure cases. This article proposes an approach for utilizing the prediction uncertainty information generated by Bayesian deep learning models to improve decision-making in fault detection. Inference is carried out using Automatic Differentiation Variational Inference (ADVI), and the resultant prediction uncertainty information is utilized to enhance fault detection. The proposed approach is applied to an open-source dataset and a real case study on Vertical Continuous Plating (VCP) of printed circuit boards. The experiments show that the performance of the proposed scheme is considerably beneficial compared to classical deep learning models.

Machine Learning for Discovery of New ADORA Modulators.

Adenosine (ADO) is an extracellular signaling molecule generated locally under conditions that produce ischemia, hypoxia, or inflammation. It is involved in modulating a range of physiological functions throughout the brain and periphery through the membrane-bound G protein-coupled receptors, called adenosine receptors (ARs) A1AR, A2AAR, A2BAR, and A3AR. These are therefore important targets for neurological, cardiovascular, inflammatory, and autoimmune diseases and are the subject of drug development directed toward the cyclic adenosine monophosphate and other signaling pathways. Initially using public data for A1AR agonists we generated and validated a Bayesian machine learning model (Receiver Operator Characteristic of 0.87) that we used to identify molecules for testing. Three selected molecules, crisaborole, febuxostat and paroxetine, showed initial activity in vitro using the HEK293 A1AR Nomad cell line. However, radioligand binding, β-arrestin assay and calcium influx assay did not confirm this A1AR activity. Nevertheless, several other AR activities were identified. Febuxostat and paroxetine both inhibited orthosteric radioligand binding in the µM range for A2AAR and A3AR. In HEK293 cells expressing the human A2AAR, stimulation of cAMP was observed for crisaborole (EC50 2.8 µM) and paroxetine (EC50 14 µM), but not for febuxostat. Crisaborole also increased cAMP accumulation in A2BAR-expressing HEK293 cells, but it was weaker than at the A2AAR. At the human A3AR, paroxetine did not show any agonist activity at 100 µM, although it displayed binding with a Ki value of 14.5 µM, suggesting antagonist activity. We have now identified novel modulators of A2AAR, A2BAR and A3AR subtypes that are clinically used for other therapeutic indications, and which are structurally distinct from previously reported tool compounds or drugs.