Bayesian Generative Model Research Articles

Calibrating Bayesian generative machine learning for Bayesiamplification

Abstract Recently, combinations of generative and Bayesian deep learning have been introduced in particle physics for both fast detector simulation and inference tasks.
These neural networks aim to quantify the uncertainty on the generated distribution originating from limited training statistics.
The interpretation of a distribution-wide uncertainty however remains ill-defined.
We show a clear scheme for quantifying the calibration of Bayesian generative machine learning models.
For a Continuous Normalizing Flow applied to a low-dimensional toy example, we evaluate the calibration of Bayesian uncertainties from either a mean-field Gaussian weight posterior, or Monte Carlo sampling network weights, to gauge their behaviour on unsteady distribution edges. 
Well calibrated uncertainties can then be used to roughly estimate the number of uncorrelated truth samples that are equivalent to the generated sample and clearly indicate data amplification for smooth features of the distribution.

Deep Variational Network Toward Blind Image Restoration.

Blind image restoration (IR) is a common yet challenging problem in computer vision. Classical model-based methods and recent deep learning (DL)-based methods represent two different methodologies for this problem, each with their own merits and drawbacks. In this paper, we propose a novel blind image restoration method, aiming to integrate both the advantages of them. Specifically, we construct a general Bayesian generative model for the blind IR, which explicitly depicts the degradation process. In this proposed model, a pixel-wise non-i.i.d. Gaussian distribution is employed to fit the image noise. It is with more flexibility than the simple i.i.d. Gaussian or Laplacian distributions as adopted in most of conventional methods, so as to handle more complicated noise types contained in the image degradation. To solve the model, we design a variational inference algorithm where all the expected posteriori distributions are parameterized as deep neural networks to increase their model capability. Notably, such an inference algorithm induces a unified framework to jointly deal with the tasks of degradation estimation and image restoration. Further, the degradation information estimated in the former task is utilized to guide the latter IR process. Experiments on two typical blind IR tasks, namely image denoising and super-resolution, demonstrate that the proposed method achieves superior performance over current state-of-the-arts.

Beta-CoRM: A Bayesian approach for n-gram profiles analysis

n-gram profiles have been successfully and widely used to analyse long sequences of potentially differing lengths for clustering or classification. Mainly, machine learning algorithms have been used for this purpose but, despite their predictive performance, these methods cannot discover hidden structures or provide a full probabilistic representation of the data. A novel class of Bayesian generative models designed for n-gram profiles used as binary attributes have been designed to address this. The flexibility of the proposed modelling allows to consider a straightforward approach to feature selection in the generative model. Furthermore, a slice sampling algorithm is derived for a fast inferential procedure, which is applied to synthetic and real data scenarios and shows that feature selection can improve classification accuracy.

Stochastic image spectroscopy: a discriminative generative approach to hyperspectral image modelling and classification

This paper introduces a new latent variable probabilistic framework for representing spectral data of high spatial and spectral dimensionality, such as hyperspectral images. We use a generative Bayesian model to represent the image formation process and provide interpretable and efficient inference and learning methods. Surprisingly, our approach can be implemented with simple tools and does not require extensive training data, detailed pixel-by-pixel labeling, or significant computational resources. Numerous experiments with simulated data and real benchmark scenarios show encouraging image classification performance. These results validate the unique ability of our framework to discriminate complex hyperspectral images, irrespective of the presence of highly discriminative spectral signatures.

A novel Bayesian generative approach for estimating tumor dynamics from published studies.

Tumor growth inhibition (TGI) modeling attempts to describe the time course changes in tumor size for patients undergoing cancer therapy. TGI models present several advantages over traditional exposure-response models that are based explicitly on clinical end points and have become a popular tool in the pharmacometrics community. Unfortunately, the data required to fit TGI models, namely longitudinal tumor measurements, are sparse or often not available in literature or publicly accessible databases. On the contrary, common end points such as progression-free survival (PFS) and objective response rate (ORR) are directly derived from longitudinal tumor measurements and are routinely published. To this end, a Bayesian generative model relating underlying tumor dynamics to summary PFS and ORR data is introduced to learn TGI model parameters using only published summary data. The parameterized model can describe the tumor dynamics, quantify treatment effect, and account for differences in the study population. The utility of this model is shown by applying it to several published studies, and learned parameters are combined to simulate an in silico trial of a novel combination therapy in a novel setting.

Chaotic neural dynamics facilitate probabilistic computations through sampling

Cortical neurons exhibit highly variable responses over trials and time. Theoretical works posit that this variability arises potentially from chaotic network dynamics of recurrently connected neurons. Here, we demonstrate that chaotic neural dynamics, formed through synaptic learning, allow networks to perform sensory cue integration in a sampling-based implementation. We show that the emergent chaotic dynamics provide neural substrates for generating samples not only of a static variable but also of a dynamical trajectory, where generic recurrent networks acquire these abilities with a biologically plausible learning rule through trial and error. Furthermore, the networks generalize their experience in the stimulus-evoked samples to the inference without partial or all sensory information, which suggests a computational role of spontaneous activity as a representation of the priors as well as a tractable biological computation for marginal distributions. These findings suggest that chaotic neural dynamics may serve for the brain function as a Bayesian generative model.

Generative Bayesian modeling to nowcast the effective reproduction number from line list data with missing symptom onset dates.

The time-varying effective reproduction number Rt is a widely used indicator of transmission dynamics during infectious disease outbreaks. Timely estimates of Rt can be obtained from reported cases counted by their date of symptom onset, which is generally closer to the time of infection than the date of report. Case counts by date of symptom onset are typically obtained from line list data, however these data can have missing information and are subject to right truncation. Previous methods have addressed these problems independently by first imputing missing onset dates, then adjusting truncated case counts, and finally estimating the effective reproduction number. This stepwise approach makes it difficult to propagate uncertainty and can introduce subtle biases during real-time estimation due to the continued impact of assumptions made in previous steps. In this work, we integrate imputation, truncation adjustment, and Rt estimation into a single generative Bayesian model, allowing direct joint inference of case counts and Rt from line list data with missing symptom onset dates. We then use this framework to compare the performance of nowcasting approaches with different stepwise and generative components on synthetic line list data for multiple outbreak scenarios and across different epidemic phases. We find that under reporting delays realistic for hospitalization data (50% of reports delayed by more than a week), intermediate smoothing, as is common practice in stepwise approaches, can bias nowcasts of case counts and Rt, which is avoided in a joint generative approach due to shared regularization of all model components. On incomplete line list data, a fully generative approach enables the quantification of uncertainty due to missing onset dates without the need for an initial multiple imputation step. In a real-world comparison using hospitalization line list data from the COVID-19 pandemic in Switzerland, we observe the same qualitative differences between approaches. The generative modeling components developed in this work have been integrated and further extended in the R package epinowcast, providing a flexible and interpretable tool for real-time surveillance.

A multi-demand operating system underlying diverse cognitive tasks

The existence of a multiple-demand cortical system with an adaptive, domain-general, role in cognition has been proposed, but the underlying dynamic mechanisms and their links to cognitive control abilities are poorly understood. Here we use a probabilistic generative Bayesian model of brain circuit dynamics to determine dynamic brain states across multiple cognitive domains, independent datasets, and participant groups, including task fMRI data from Human Connectome Project, Dual Mechanisms of Cognitive Control study and a neurodevelopment study. We discover a shared brain state across seven distinct cognitive tasks and found that the dynamics of this shared brain state predicted cognitive control abilities in each task. Our findings reveal the flexible engagement of dynamic brain processes across multiple cognitive domains and participant groups, and uncover the generative mechanisms underlying the functioning of a domain-general cognitive operating system. Our computational framework opens promising avenues for probing neurocognitive function and dysfunction.

SHARE-Topic: Bayesian interpretable modeling of single-cell multi-omic data

Multi-omic single-cell technologies, which simultaneously measure the transcriptional and epigenomic state of the same cell, enable understanding epigenetic mechanisms of gene regulation. However, noisy and sparse data pose fundamental statistical challenges to extract biological knowledge from complex datasets. SHARE-Topic, a Bayesian generative model of multi-omic single cell data using topic models, aims to address these challenges. SHARE-Topic identifies common patterns of co-variation between different omic layers, providing interpretable explanations for the data complexity. Tested on data from different technological platforms, SHARE-Topic provides low dimensional representations recapitulating known biology and defines associations between genes and distal regulators in individual cells.

Bayesian semiparametric model for sequential treatment decisions with informative timing.

We develop a Bayesian semiparametric model for the impact of dynamic treatment rules on survival among patients diagnosed with pediatric acute myeloid leukemia (AML). The data consist of a subset of patients enrolled in a phase III clinical trial in which patients move through a sequence of four treatment courses. At each course, they undergo treatment that may or may not include anthracyclines (ACT). While ACT is known to be effective at treating AML, it is also cardiotoxic and can lead to early death for some patients. Our task is to estimate the potential survival probability under hypothetical dynamic ACT treatment strategies, but there are several impediments. First, since ACT is not randomized, its effect on survival is confounded over time. Second, subjects initiate the next course depending on when they recover from the previous course, making timing potentially informative of subsequent treatment and survival. Third, patients may die or drop out before ever completing the full treatment sequence. We develop a generative Bayesian semiparametric model based on Gamma Process priors to address these complexities. At each treatment course, the model captures subjects' transition to subsequent treatment or death in continuous time. G-computation is used to compute a posterior over potential survival probability that is adjusted for time-varying confounding. Using our approach, we estimate the efficacy of hypothetical treatment rules that dynamically modify ACT based on evolving cardiac function.

Spatio-temporal modeling of co-dynamics of smallpox, measles, and pertussis in pre-healthcare Finland.

Infections are known to interact as previous infections may have an effect on risk of succumbing to a new infection. The co-dynamics can be mediated by immunosuppression or modulation, shared environmental or climatic drivers, or competition for susceptible hosts. Research and statistical methods in epidemiology often concentrate on large pooled datasets, or high quality data from cities, leaving rural areas underrepresented in literature. Data considering rural populations are typically sparse and scarce, especially in the case of historical data sources, which may introduce considerable methodological challenges. In order to overcome many obstacles due to such data, we present a general Bayesian spatio-temporal model for disease co-dynamics. Applying the proposed model on historical (1820-1850) Finnish parish register data, we study the spread of infectious diseases in pre-healthcare Finland. We observe that measles, pertussis, and smallpox exhibit positively correlated dynamics, which could be attributed to immunosuppressive effects or, for example, the general weakening of the population due to recurring infections or poor nutritional conditions.

A Bayesian non-asymptotic extreme value model for daily rainfall data

Motivated by the analysis of extreme rainfall data, we introduce a general Bayesian hierarchical model for estimating the probability distribution of extreme values of intermittent random sequences, a common problem in geophysical and environmental science settings. The approach presented here is derived under a set of assumptions different than those used in traditional extreme value (EV) theory. A hierarchical model structure is adopted for capturing the underlying variability in the yearly distribution of event magnitudes and occurrences, which are described through a latent temporal process. Focusing on daily rainfall extremes, the structure of the proposed model lends itself to incorporating prior geo-physical understanding of the rainfall process. By means of an extensive simulation study, we show that this methodology can significantly reduce quantile estimation uncertainty with respect to Bayesian formulations of traditional asymptotic EV methods, particularly in the case of relatively small samples. On the other hand, a negative bias is observed in estimated rainfall quantiles. We evaluate the tradeoff between bias and variance for extreme rainfall quantiles and show that the approach proposed here can be advantageous depending on the sample size available for training. The benefits of the approach are illustrated with an application to a large data set of 479 long daily rainfall historical records from across the conterminous United States. By comparing measures of in-sample and out-of-sample predictive accuracy, we find that the model structure developed here, combined with the use of all available observations for inference, can improve robustness with respect to overfitting to the specific sample in case of short datasets.

Spatially Dependent Bayesian Modeling of Geostatistics Data and Its Application for Tuberculosis (TB) in China

Geostatistics data in regions always have highly spatial heterogeneous, yet the regional features of the data itself cannot be ignored. In this paper, a novel latent Bayesian spatial model is proposed, which incorporates the spatial dependence of different subjects and the spatial random effects to further analysis the influence of spatial effect. The model is verified to be compatible with the integrated nested Laplace approximation (INLA) framework and is fitted using INLA and stochastic partial differential equation (SPDE). The posterior marginal distribution of parameters is estimated with high precision. Additionally, a practical application of the model is presented using tuberculosis (TB) incidence data in China from 2015 to 2019. The results show that the fitting accuracy of our model is higher than the general Bayesian spatial model using INLA-SPDE.

Incorporating (variational) free energy models into mechanisms: the case of predictive processing under the free energy principle

The issue of the relationship between predictive processing (PP) and the free energy principle (FEP) remains a subject of debate and controversy within the research community. Many researchers have expressed doubts regarding the actual integration of PP with the FEP, questioning whether the FEP can truly contribute significantly to the mechanistic understanding of PP or even undermine such integration altogether. In this paper, I present an alternative perspective. I argue that, from the viewpoint of the constraint-based mechanisms approach, the FEP imposes an important constraint, namely variational free energy, on the mechanistic architecture proposed by PP. According to the constraint-based mechanisms approach, high-level cognitive mechanisms are integral parts of extensive heterarchical networks that govern the physiology and behavior of agents. Consequently, mechanistic explanations of cognitive phenomena should incorporate constraints and flows of free energy as relevant components, given that the implemented constraints operate as long as free energy is available. Within this framework, I contend that the FEP provides a relevant constraint for explaining at least some biological cognitive mechanisms described in terms of Bayesian generative models that minimize prediction errors.

Continual learning with Bayesian model based on a fixed pre-trained feature extractor

Deep learning has shown its human-level performance in various applications. However, current deep learning models are characterized by catastrophic forgetting of old knowledge when learning new classes. This poses a challenge such as in intelligent diagnosis systems where initially only training data of a limited number of diseases are available. In this case, updating the intelligent system with data of new diseases would inevitably downgrade its performance on previously learned diseases. Inspired by the process of learning new knowledge in human brains, we propose a Bayesian generative model for continual learning built on a fixed pre-trained feature extractor. In this model, knowledge of each old class can be compactly represented by a collection of statistical distributions, e.g., with Gaussian mixture models, and naturally kept from forgetting in continual learning over time. Unlike existing class-incremental learning methods, the proposed approach is not sensitive to the continual learning process and can be additionally well applied to the data-incremental learning scenario. Experiments on multiple medical and natural image classification tasks reveal that the proposed approach outperforms state-of-the-art approaches that even keep some images of old classes during continual learning of new classes.

Reliable affordances: A generative modeling approach for test-retest reliability of the affordances task.

The affordances task serves as an important tool for the assessment of cognition and visuomotor functioning, and yet its test-retest reliability has not been established. In the affordances task, participants attend to a goal-directed task (e.g., classifying manipulable objects such as cups and pots) while suppressing their stimulus-driven, irrelevant reactions afforded by these objects (e.g., grasping their handles). This results in cognitive conflicts manifesting at the task level and the response level. In the current study, we assessed the reliability of the affordances task for the first time. While doing so, we referred to the "reliability paradox," according to which behavioral tasks that produce highly replicable group-level effects often yield low test-retest reliability due to the inadequacy of traditional correlation methods in capturing individual differences between participants. Alongside the simple test-retest correlations, we employed a Bayesian generative model that was recently demonstrated to result in a more precise estimation of test-retest reliability. Two hundred and ninety-five participants completed an online version of the affordances task twice, with a one-week gap. Performance on the online version replicated results obtained under in-lab administrations of the task. While the simple correlation method resulted in weak test-retest measures of the different effects, the generative model yielded a good reliability assessment. The current results support the utility of the affordances task as a reliable behavioral tool for the assessment of group-level and individual differences in cognitive and visuomotor functioning. The results further support the employment of generative modeling in the study of individual differences.

Real-Time Neural Dense Elevation Mapping for Urban Terrain With Uncertainty Estimations

Having good knowledge of terrain information is essential for improving the performance of various downstream tasks on complex terrains, especially for the locomotion and navigation of legged robots. We present a novel framework for neural urban terrain reconstruction with uncertainty estimations. It generates dense robot-centric elevation maps online from sparse LiDAR observations. We design a novel pre-processing and point features representation approach that ensures high robustness and computational efficiency when integrating multiple point cloud frames. A generative Bayesian model then recovers the detailed terrain structures while simultaneously providing the pixel-wise reconstruction uncertainty. We evaluate the proposed pipeline through both simulation and real-world experiments. Our approach achieves high-quality terrain reconstruction with real-time performance on a mobile platform, and the uncertainty estimates may further benefit the downstream tasks of legged robots.

Cross-Modal Variational Auto-Encoder for Content-Based Micro-Video Background Music Recommendation

In this paper, we propose a cross-modal variational auto-encoder (CMVAE) for content-based micro-video background music recommendation. CMVAE is a hierarchical Bayesian generative model that matches relevant background music to a micro-video by projecting these two multimodal inputs into a shared low-dimensional latent space, where the alignment of two corresponding embeddings of a matched video-music pair is achieved by cross-generation. Moreover, the multimodal information is fused by the product-of-experts (PoE) principle, where the semantic information in visual and textual modalities of the micro-video are weighted according to their variance estimations such that the modality with a lower noise level is given more weights. Therefore, the micro-video latent variables contain less irrelevant information that results in a more robust model generalization. Furthermore, we establish a large-scale content-based micro-video background music recommendation dataset, TT-150k, composed of approximately 3,000 different background music clips associated to 150,000 micro-videos from different users. Extensive experiments on the established TT-150k dataset demonstrate the effectiveness of the proposed method. A qualitative assessment of CMVAE by visualizing some recommendation results is also included.

Sentiment Analysis of Weibo Comments based on LDA Model

The essence of LDA (Latent Dirichlet Allocation) model is a generative Bayesian probability model that contains three layers of words, topics and corpus (sometimes called document set). Under the LDA algorithm theory, each document represents a probability distribution formed by some topics, and each topic represents a probability distribution formed by many words. Therefore, the model fitting results will present the core keywords and specific probabilities of each topic, and researchers can interpret the meaning of the document according to the model results. In this paper, we hope to use the LDA model as the basis for the emotional analysis of microblog comments.

Social Inference from Relational Visual Information

Humans easily make social evaluations from visual input, such as recognizing a social interaction or deciding who is a friend and who is a foe. Prior efforts to model this ability have suggested that visual features and motion cues alone cannot account for human performance in social tasks, like distinguishing between helping and hindering interactions. On the other hand, generative Bayesian inference models, which make predictions based on simulations of agents’ social or physical goals, accurately predict human judgments, but are computationally very expensive and often impossible to implement in natural visual stimuli. Inspired by developmental work, we hypothesize that introducing inductive biases in visual models would allow them to make more human-like social judgments. Specifically, in this study, we investigate if relational representations of visual stimuli using graph neural networks can predict human social interaction judgments. We use the PHASE dataset, consisting of 2D animations of two agents and two objects resembling real-life social interactions, rated as friendly, neutral, or adversarial. We propose a graph neural networks (GNN) based architecture, that takes in graph representations of a video and predicts the relationship between the agents. We collected human ratings for each of the 400 videos and found that our GNN model aligns with human judgments significantly better than a baseline visual model with the same visual/motion information but without the graph structure (79% vs 62% prediction of human judgements). Intriguingly, explicitly adding relational information to the baseline model does not improve its performance, suggesting that graphical representations in particular are important to modeling human judgements. Taken together, these results suggest that relational graphical representations of visual information can help artificial vision systems make more human-like social judgments without incurring the computational cost of Bayesian models, and provide insights into the computations that humans employ while making visual social judgments.