Probabilistic Generative Model Research Articles

Topic modelling as a method for framing analysis of news coverage of the Russia-Ukraine war in 2022–2023

This study critically analyses the representation of the Russia-Ukraine war in Western (the Euronews) and Eastern (the Kyiv Post) media discourses. It examines how media organisations shape narratives through strategic framing. Employing the Natural Language Processing technique – Topic Modelling – with a generative probabilistic model LDA and a transformer-based language model BERT, the study reveals generic frames elaborated by more specific extensions, shedding light on media portrayal of economy, public opinion, security & defence, external regulations, policy evaluation, and health & safety sectors. Through Named Entity Recognition with roBERTa, Sentiment Analysis with distilBERT, and Corpus Linguistics methods with LancsBox X, interpretation of these overarching frames provides a comprehensive analysis of the nuances in narratives, societal perceptions and policy decisions amidst the ongoing war.

The Gods as Latent Causes: A Statistical Inference Theory of Religion

ABSTRACT The statistical brain hypothesis posits that the brain constructs probabilistic models of the environment. Here we examine whether this perspective can provide any insight on religion. We propose that religious ideas represent an attempt to explain away residuals, that is, to explain discrepancies between observations and predictions. The framework postulates probabilistic generative models where gods are described by latent variables whose possible states correspond to the actions available to the gods. As examined in the paper, our proposal offers a plausible interpretation of typical religious phenomena such as miracles, omens, and divination. Moreover, it captures important characteristics of religious beliefs including the notion that gods control multiple spheres of reality, are organized hierarchically, and control aspects that are salient for believers. Besides offering an intriguing new perspective on religion, the paper corroborates the possibility that the statistical brain hypothesis represents a unifying theory of the mind.

Learning probability distributions of sensory inputs with Monte Carlo predictive coding.

It has been suggested that the brain employs probabilistic generative models to optimally interpret sensory information. This hypothesis has been formalised in distinct frameworks, focusing on explaining separate phenomena. On one hand, classic predictive coding theory proposed how the probabilistic models can be learned by networks of neurons employing local synaptic plasticity. On the other hand, neural sampling theories have demonstrated how stochastic dynamics enable neural circuits to represent the posterior distributions of latent states of the environment. These frameworks were brought together by variational filtering that introduced neural sampling to predictive coding. Here, we consider a variant of variational filtering for static inputs, to which we refer as Monte Carlo predictive coding (MCPC). We demonstrate that the integration of predictive coding with neural sampling results in a neural network that learns precise generative models using local computation and plasticity. The neural dynamics of MCPC infer the posterior distributions of the latent states in the presence of sensory inputs, and can generate likely inputs in their absence. Furthermore, MCPC captures the experimental observations on the variability of neural activity during perceptual tasks. By combining predictive coding and neural sampling, MCPC can account for both sets of neural data that previously had been explained by these individual frameworks.

Retraction Note: Deep belief network-based probabilistic generative model for detection of robotic manipulator failure execution

Retraction Note: Deep belief network-based probabilistic generative model for detection of robotic manipulator failure execution

A Data-Driven Approach for Game Evaluation Using Latent Dirichlet Allocation Method Based on Players’ Reviews

Garena is a global game developer and publisher. Garena provides users with access to popular and engaging online games for mobile and PC, developed, curated, and localized for each market. The Battle Royale genre is relatively new, and this research will evaluate Free Fire, one of the games in this genre made by Garena. Analyzing end-user reviews is considered important for evaluating software quality. Researchers need to understand which aspects need to be evaluated based on player reviews on Google Play and how the model's performance is generated using feedback from players who have played Free Fire. In this study, researchers use the Latent Dirichlet Allocation (LDA) method to model topics and generate clusters in discussions for each topic. LDA is a generative probabilistic model of a corpus. This research on topic modeling using Google Play reviews and LDA has identified the topics users are most concerned with. The research shows three main aspects: bugs, graphics and performance, and game rules/punishment policy, as aspects that need to be evaluated based on player reviews on Google Play.

Multi-adversarial autoencoders: Stable, faster and self-adaptive representation learning

The variational autoencoder (VAE) and generative adversarial networks (GAN) are two prominent approaches to achieving a probabilistic generative model by way of an autoencoder and a two-player minimax game. While VAEs often suffer from over-simplified posterior approximations, the adversarial autoencoder (AAE) has shown promise by adopting GAN to match the variational posterior to an arbitrary prior through adversarial training. Both VAEs and GANs face significant challenges such as training stability, mode collapse, and difficulty in extracting meaningful latent representations. In this paper, we propose the Multi-adversarial Autoencoder (MAAE), which extends the AAE framework by incorporating multiple discriminators and enabling soft-ensemble feedback. By adaptively regulating the collective feedback from multiple discriminators, MAAE captures a balance between fitting the data distribution and performing accurate inference and accelerates training stability while extracting meaningful and interpretable latent representations. Experimental evaluations on MNIST, CIFAR10, and CelebA datasets demonstrate significant improvements in latent representation, quality of generated samples, log-likelihood, and a pairwise comparison metric, with comparisons to recent methods.

Emergent time scales of epistasis in protein evolution

We introduce a data-driven epistatic model of protein evolution, capable of generating evolutionary trajectories spanning very different time scales reaching from individual mutations to diverged homologs. Our in silico evolution encompasses random nucleotide mutations, insertions and deletions, and models selection using a fitness landscape, which is inferred via a generative probabilistic model for protein families. We show that the proposed framework accurately reproduces the sequence statistics of both short-time (experimental) and long-time (natural) protein evolution, suggesting applicability also to relatively data-poor intermediate evolutionary time scales, which are currently inaccessible to evolution experiments. Our model uncovers a highly collective nature of epistasis, gradually changing the fitness effect of mutations in a diverging sequence context, rather than acting via strong interactions between individual mutations. This collective nature triggers the emergence of a long evolutionary time scale, separating fast mutational processes inside a given sequence context, from the slow evolution of the context itself. The model quantitatively reproduces epistatic phenomena such as contingency and entrenchment, as well as the loss of predictability in protein evolution observed in deep mutational scanning experiments of distant homologs. It thereby deepens our understanding of the interplay between mutation and selection in shaping protein diversity and functions, allows one to statistically forecast evolution, and challenges the prevailing independent-site models of protein evolution, which are unable to capture the fundamental importance of epistasis.

Ab initio uncertainty quantification in scattering analysis of microscopy.

Estimating parameters from data is a fundamental problem in physics, customarily done by minimizing a loss function between a model and observed statistics. In scattering-based analysis, it is common to work in the reciprocal space. Researchers often employ their domain expertise to select a specific range of wave vectors for analysis, a choice that can vary depending on the specific case. We introduce another paradigm that defines a probabilistic generative model from the beginning of data processing and propagates the uncertainty for parameter estimation, termed the ab initio uncertainty quantification (AIUQ). As an illustrative example, we demonstrate this approach with differential dynamic microscopy (DDM) that extracts dynamical information through minimizing a loss function for the squared differences of the Fourier-transformed intensities, at a selected range of wave vectors. We first show that the conventional way of estimation in DDM is equivalent to fitting a temporal variogram in the reciprocal space using a latent factor model as the generative model. Then we derive the maximum marginal likelihood estimator, which optimally weighs the information at all wave vectors, therefore eliminating the need to select the range of wave vectors. Furthermore, we substantially reduce the computational cost of computing the likelihood function without approximation, by utilizing the generalized Schur algorithm for Toeplitz covariances. Simulated studies of a wide range of dynamical systems validate that the AIUQ method improves estimation accuracy and enables model selection with automated analysis. The utility of AIUQ is also demonstrated by three distinct sets of experiments: first in an isotropic Newtonian fluid, pushing limits of optically dense systems compared to multiple particle tracking; next in a system undergoing a sol-gel transition, automating the determination of gelling points and critical exponent; and lastly, in discerning anisotropic diffusive behavior of colloids in a liquid crystal. These studies demonstrate that the new approach does not require manually selecting the wave vector range and enables automated analysis.

Stacking models of brain dynamics to improve prediction of subject traits in fMRI

Abstract Beyond structural and time-averaged functional connectivity brain measures, modelling the way brain activity dynamically unfolds can add important information to our understanding and characterisation of individual cognitive traits. One approach to leveraging this information is to extract features from models of brain network dynamics to predict individual traits. However, these predictions are susceptible to variability due to factors such as variation in model estimation induced by the choice of hyperparameters. We suggest that, rather than merely being statistical noise, this variability may be useful in providing complementary information that can be leveraged to improve prediction accuracy. To leverage this variability, we propose the use of stacking, a prediction-driven approach for model selection. Specifically, we combine predictions developed from multiple hidden Markov models—a probabilistic generative model of network dynamics that identifies recurring patterns of brain activity—to demonstrate that stacking can slightly improve the accuracy and robustness of cognitive trait predictions. By comparing analysis from the Human Connectome Project and UK Biobank datasets, we show that stacking is relatively effective at improving prediction accuracy and robustness when there are enough subjects, and that the effectiveness of combining predictions from static and dynamic functional connectivity approaches depends on the length of scan per subject. We also show that the effectiveness of stacking predictions is driven by the accuracy and diversity in the underlying model estimations.

Enhancing speech emotion detection with Windowed Long-Term Average Spectrum and Logistic-Rectified Linear Unit

Speech emotion detection can help monitor people's mental health by analyzing their speech patterns and recognizing emotional indicators. The raw data is initially augmented using various techniques such as shifting, pitching, noise injection, and stretching. Some feature extraction methods may only produce a limited set of features, potentially missing out on important information present in the data. This can result in reduced accuracy and the inability to capture complex patterns and variations in speech signals. The planned work consists of two phases: attribute extraction and classification. In the first phase, the WLTAS (Windowed Long-Term Average Spectrum) feature extraction extracts spectral information from speech signals, providing a compact and representative feature representation. It captures the frequency content and energy distribution of speech over time, enabling the analysis of important spectral characteristics. In the second phase, the Logistic-Rectified Linear Unit (LoRLU) introduces sparsity. It enhances the discriminative power of the classifier by allowing only positive values to pass through while discarding negative values. The Gibbs Restricted Boltzmann Machine (GRBM) is a generative probabilistic model that studies a joint probability delivery of the input data. It is particularly effective in capturing dependencies and patterns in high-dimensional data. The performance analysis of WLTAS-L-GRBM (Windowed Long-Term Average Spectrum - Logistic-Rectified Linear Unit - Gibbs Restricted Boltzmann Machine) has been conducted using evaluation metrics such as accuracy (99%), precision (0.97), recall (1), F1 score (0.98), MSE (0.01), and AUC-ROC (0.97) and also evaluated against various speech emotion datasets.

Synthetic lidar point cloud generation using deep generative models for improved driving scene object recognition

The imbalanced distribution of different object categories poses a challenge for training accurate object recognition models in driving scenes. Supervised machine learning models trained on imbalanced data are biased and easily overfit the majority classes, such as vehicles and pedestrians, which appear more frequently in driving scenes. We propose a novel data augmentation approach for object recognition in lidar point cloud of driving scenes, which leverages probabilistic generative models to produce synthetic point clouds for the minority classes and complement the original imbalanced dataset. We evaluate five generative models based on different statistical principles, including Gaussian mixture model, variational autoencoder, generative adversarial network, adversarial autoencoder and the diffusion model. Experiments with a real-world autonomous driving dataset show that the synthetic point clouds generated for the minority classes by the Latent Generative Adversarial Network result in significant improvement of object recognition performance for both minority and majority classes. The codes are available at https://github.com/AAAALEX-XIANG/Synthetic-Lidar-Generation.

Crystal Composition Transformer: Self-Learning Neural Language Model for Generative and Tinkering Design of Materials.

Self-supervised neural language models have recently achieved unprecedented success from natural language processing to learning the languages of biological sequences and organic molecules. These models have demonstrated superior performance in the generation, structure classification, and functional predictions for proteins and molecules with learned representations. However, most of the masking-based pre-trained language models are not designed for generative design, and their black-box nature makes it difficult to interpret their design logic. Here a Blank-filling Language Model for Materials (BLMM) Crystal Transformer is proposed, a neural network-based probabilistic generative model for generative and tinkering design of inorganic materials. The model is built on the blank-filling language model for text generation and has demonstrated unique advantages in learning the "materials grammars" together with high-quality generation, interpretability, and data efficiency. It can generate chemically valid materials compositions with as high as 89.7% charge neutrality and 84.8% balanced electronegativity, which are more than four and eight times higher compared to a pseudo-random sampling baseline. The probabilistic generation process of BLMM allows it to recommend materials tinkering operations based on learned materials chemistry, which makes it useful for materials doping. The model is applied to discover a set of new materials as validated using the Density Functional Theory (DFT) calculations. This work thus brings the unsupervised transformer language models based generative artificial intelligence to inorganic materials. A user-friendly web app for tinkering materials design has been developed and can be accessed freely at www.materialsatlas.org/blmtinker.

Collective predictive coding hypothesis: symbol emergence as decentralized Bayesian inference.

Understanding the emergence of symbol systems, especially language, requires the construction of a computational model that reproduces both the developmental learning process in everyday life and the evolutionary dynamics of symbol emergence throughout history. This study introduces the collective predictive coding (CPC) hypothesis, which emphasizes and models the interdependence between forming internal representations through physical interactions with the environment and sharing and utilizing meanings through social semiotic interactions within a symbol emergence system. The total system dynamics is theorized from the perspective of predictive coding. The hypothesis draws inspiration from computational studies grounded in probabilistic generative models and language games, including the Metropolis-Hastings naming game. Thus, playing such games among agents in a distributed manner can be interpreted as a decentralized Bayesian inference of representations shared by a multi-agent system. Moreover, this study explores the potential link between the CPC hypothesis and the free-energy principle, positing that symbol emergence adheres to the society-wide free-energy principle. Furthermore, this paper provides a new explanation for why large language models appear to possess knowledge about the world based on experience, even though they have neither sensory organs nor bodies. This paper reviews past approaches to symbol emergence systems, offers a comprehensive survey of related prior studies, and presents a discussion on CPC-based generalizations. Future challenges and potential cross-disciplinary research avenues are highlighted.

Assessing the Accuracy and Efficiency of Free Energy Differences Obtained from Reweighted Flow-Based Probabilistic Generative Models.

Computing free energy differences between metastable states characterized by nonoverlapping Boltzmann distributions is often a computationally intensive endeavor, usually requiring chains of intermediate states to connect them. Targeted free energy perturbation (TFEP) can significantly lower the computational cost of FEP calculations by choosing a set of invertible maps used to directly connect the distributions of interest, achieving the necessary statistically significant overlaps without sampling any intermediate states. Probabilistic generative models (PGMs) based on normalizing flow architectures can make it much easier via machine learning to train invertible maps needed for TFEP. However, the accuracy and applicability of approaches based on empirically learned maps depend crucially on the choice of reweighting method adopted to estimate the free energy differences. In this work, we assess the accuracy, rate of convergence, and data efficiency of different free energy estimators, including exponential averaging, Bennett acceptance ratio (BAR), and multistate Bennett acceptance ratio (MBAR), in reweighting PGMs trained by maximum likelihood on limited amounts of molecular dynamics data sampled only from end-states of interest. We carry out the comparisons on a set of simple but representative case studies, including conformational ensembles of alanine dipeptide and ibuprofen. Our results indicate that BAR and MBAR are both data efficient and robust, even in the presence of significant model overfitting in the generation of invertible maps. This analysis can serve as a stepping stone for the deployment of efficient and quantitatively accurate ML-based free energy calculation methods in complex systems.

Zero-shot denoising of microscopy images recorded at high-resolution limits.

Conventional and electron microscopy visualize structures in the micrometer to nanometer range, and such visualizations contribute decisively to our understanding of biological processes. Due to different factors in recording processes, microscopy images are subject to noise. Especially at their respective resolution limits, a high degree of noise can negatively effect both image interpretation by experts and further automated processing. However, the deteriorating effects of strong noise can be alleviated to a large extend by image enhancement algorithms. Because of the inherent high noise, a requirement for such algorithms is their applicability directly to noisy images or, in the extreme case, to just a single noisy image without a priori noise level information (referred to as blind zero-shot setting). This work investigates blind zero-shot algorithms for microscopy image denoising. The strategies for denoising applied by the investigated approaches include: filtering methods, recent feed-forward neural networks which were amended to be trainable on noisy images, and recent probabilistic generative models. As datasets we consider transmission electron microscopy images including images of SARS-CoV-2 viruses and fluorescence microscopy images. A natural goal of denoising algorithms is to simultaneously reduce noise while preserving the original image features, e.g., the sharpness of structures. However, in practice, a tradeoff between both aspects often has to be found. Our performance evaluations, therefore, focus not only on noise removal but set noise removal in relation to a metric which is instructive about sharpness. For all considered approaches, we numerically investigate their performance, report their denoising/sharpness tradeoff on different images, and discuss future developments. We observe that, depending on the data, the different algorithms can provide significant advantages or disadvantages in terms of their noise removal vs. sharpness preservation capabilities, which may be very relevant for different virological applications, e.g., virological analysis or image segmentation.

Deep graph clustering by integrating community structure with neighborhood information

Deep graph clustering approaches employ deep graph neural networks to encode node embeddings and subsequently partition nodes based on these representations. Recent one-step methods have demonstrated progress; however, they often encounter issues during the representation learning phase. A primary challenge is that the encoding networks fail to capture the high-order structural information of the graph for effective clustering. We believe that combining the two distributions comprehensively characterizes the real node distribution in terms of neighborhood proximity and community aggregation to achieve better clustering. Additionally, the prevalent representation learning paradigms based on adjacency matrix reconstruction are computationally intensive and increase optimization challenges. This paper presents a novel deep graph clustering model, termed Deep Integration of Community structure and Neighborhood information (DICN). The model incorporates a graph self-attention mechanism to aggregate neighborhood-level node information, and utilizes a probabilistic generative model for edge community detection to captain community-level node information. These two-level embeddings are employed to compute explicit node membership distributions for the clustering loss. This ensures that the clustering objective effectively integrates both low-order and high-order node characteristics. The comprehensive learning process of DICN optimizes not only the node embeddings but also the clustering results by integrating the two-level losses from representation learning and the clustering loss with a weak guidance loss for clustering. Experiments on some benchmark datasets illustrate that DICN model has superior performance over existing state-of-the-art methods.

Towards parsimonious generative modeling of RNA families.

Generative probabilistic models emerge as a new paradigm in data-driven, evolution-informed design of biomolecular sequences. This paper introduces a novel approach, called Edge Activation Direct Coupling Analysis (eaDCA), tailored to the characteristics of RNA sequences, with a strong emphasis on simplicity, efficiency, and interpretability. eaDCA explicitly constructs sparse coevolutionary models for RNA families, achieving performance levels comparable to more complex methods while utilizing a significantly lower number of parameters. Our approach demonstrates efficiency in generating artificial RNA sequences that closely resemble their natural counterparts in both statistical analyses and SHAPE-MaP experiments, and in predicting the effect of mutations. Notably, eaDCA provides a unique feature: estimating the number of potential functional sequences within a given RNA family. For example, in the case of cyclic di-AMP riboswitches (RF00379), our analysis suggests the existence of approximately 1039 functional nucleotide sequences. While huge compared to the known <4000 natural sequences, this number represents only a tiny fraction of the vast pool of nearly 1082 possible nucleotide sequences of the same length (136 nucleotides). These results underscore the promise of sparse and interpretable generative models, such as eaDCA, in enhancing our understanding of the expansive RNA sequence space.

Abstract NG08: Dissecting and quantifying pancreatic cancer plasticity using single-cell multiomics, lineage tracing and functional genomics reveals novel mediators of therapy resistance

Abstract NG08: Dissecting and quantifying pancreatic cancer plasticity using single-cell multiomics, lineage tracing and functional genomics reveals novel mediators of therapy resistance

Temporal Dual-Attributed Network Generation Oriented Community Detection Model

Community detection is a crucial task on the research field of network analysis. However, this task recently has become challenging due to the explosion of network in terms of the scale and the side information, e.g., temporal information and attribute information. Here we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PGMTAN</i> —a probabilistic generative model for overlapping community detection on temporal dual-attributed networks. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PGMTAN</i> aims to characterize four generation processes: 1) generation of occurrence of the links, 2) generation of node-community memberships via assortative attributes, 3) generation of generative attributes, and 4) generation of evolutionary dynamics of community structure. Particularly, we adopt a hidden Markov chain model to capture the network's dynamics on the evolution of community structure over time. Moreover, we seek to optimize a lower-bound of likelihood function to accelerate the model's parameter estimation. We carry out extensive experiments on several real-world and synthetic networks to test <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PGMTAN</i> 's performance and the results substantiate that it can outperform multiple baselines and give us promising performance in terms of detection accuracy and convergence.

Diffusion model-based inverse design for thermal transparency

Generative models in the field of artificial intelligence and their applications and deployment have demonstrated their great strength in the past few years. Of the vast spectrum of generative models, diffusion probabilistic models have proven to be particularly powerful and productive, transforming notions such as text-to-image and text-to-video generation from ideas into practical applications. In our previous works, we proposed a thermal metamaterial-based periodic interparticle interaction mechanism for heat management, with a specific application in thermal transparency. To address the challenging problems associated with the inverse design of thermal metamaterial structures, we employed an autoencoder-based machine learning approach and a reinforcement learning-based approach successfully. In this work, we demonstrate that our particular problems with the inverse design of thermal metamaterial-based periodic lattices for the realization of thermal transparency can also be reframed and efficiently solved by training a generative diffusion probabilistic model that can generate the design parameters corresponding to the desired response. Furthermore, we show that for a specific response, multiple sets of design parameters can be obtained by simply performing multiple inferences with the generative diffusion probabilistic model, enabling us to select the ones that can be more economical to fabricate and implement. Our work is among the first to use a diffusion model for the inverse design of thermal metamaterial-based structures and demonstrates the effectiveness of generating low-dimensional design parameters through a diffusion model.