Probabilistic Generative Model Research Articles

Towards parsimonious generative modeling of RNA families.

Generative probabilistic models emerge as a new paradigm in data-driven, evolution-informed design of biomolecular sequences. This paper introduces a novel approach, called Edge Activation Direct Coupling Analysis (eaDCA), tailored to the characteristics of RNA sequences, with a strong emphasis on simplicity, efficiency, and interpretability. eaDCA explicitly constructs sparse coevolutionary models for RNA families, achieving performance levels comparable to more complex methods while utilizing a significantly lower number of parameters. Our approach demonstrates efficiency in generating artificial RNA sequences that closely resemble their natural counterparts in both statistical analyses and SHAPE-MaP experiments, and in predicting the effect of mutations. Notably, eaDCA provides a unique feature: estimating the number of potential functional sequences within a given RNA family. For example, in the case of cyclic di-AMP riboswitches (RF00379), our analysis suggests the existence of approximately 1039 functional nucleotide sequences. While huge compared to the known <4000 natural sequences, this number represents only a tiny fraction of the vast pool of nearly 1082 possible nucleotide sequences of the same length (136 nucleotides). These results underscore the promise of sparse and interpretable generative models, such as eaDCA, in enhancing our understanding of the expansive RNA sequence space.

Abstract NG08: Dissecting and quantifying pancreatic cancer plasticity using single-cell multiomics, lineage tracing and functional genomics reveals novel mediators of therapy resistance

Abstract NG08: Dissecting and quantifying pancreatic cancer plasticity using single-cell multiomics, lineage tracing and functional genomics reveals novel mediators of therapy resistance

Diffusion model-based inverse design for thermal transparency

Generative models in the field of artificial intelligence and their applications and deployment have demonstrated their great strength in the past few years. Of the vast spectrum of generative models, diffusion probabilistic models have proven to be particularly powerful and productive, transforming notions such as text-to-image and text-to-video generation from ideas into practical applications. In our previous works, we proposed a thermal metamaterial-based periodic interparticle interaction mechanism for heat management, with a specific application in thermal transparency. To address the challenging problems associated with the inverse design of thermal metamaterial structures, we employed an autoencoder-based machine learning approach and a reinforcement learning-based approach successfully. In this work, we demonstrate that our particular problems with the inverse design of thermal metamaterial-based periodic lattices for the realization of thermal transparency can also be reframed and efficiently solved by training a generative diffusion probabilistic model that can generate the design parameters corresponding to the desired response. Furthermore, we show that for a specific response, multiple sets of design parameters can be obtained by simply performing multiple inferences with the generative diffusion probabilistic model, enabling us to select the ones that can be more economical to fabricate and implement. Our work is among the first to use a diffusion model for the inverse design of thermal metamaterial-based structures and demonstrates the effectiveness of generating low-dimensional design parameters through a diffusion model.

MorphVAE: Advancing Morphological Design of Voxel-Based Soft Robots with Variational Autoencoders

Soft robot design is an intricate field with unique challenges due to its complex and vast search space. In the past literature, evolutionary computation algorithms, including novel probabilistic generative models (PGMs), have shown potential in this realm. However, these methods are sample inefficient and predominantly focus on rigid robots in locomotion tasks, which limit their performance and application in robot design automation. In this work, we propose MorphVAE, an innovative PGM that incorporates a multi-task training scheme and a meticulously crafted sampling technique termed ``continuous natural selection'', aimed at bolstering sample efficiency. This method empowers us to gain insights from assessed samples across diverse tasks and temporal evolutionary stages, while simultaneously maintaining a delicate balance between optimization efficiency and biodiversity. Through extensive experiments in various locomotion and manipulation tasks, we substantiate the efficiency of MorphVAE in generating high-performing and diverse designs, surpassing the performance of competitive baselines.

A motif-based probabilistic approach for community detection in complex networks

AbstractCommunity detection in complex networks is an important task for discovering hidden information in network analysis. Neighborhood density between nodes is one of the fundamental indicators of community presence in the network. A community with a high edge density will have correlations between nodes that extend beyond their immediate neighbors, denoted by motifs. Motifs are repetitive patterns of edges observed with high frequency in the network. We proposed the PCDMS method (Probabilistic Community Detection with Motif Structure) that detects communities by estimating the triangular motif in the network. This study employs structural density between nodes, a key concept in graph analysis. The proposed model has the advantage of using a probabilistic generative model that calculates the latent parameters of the probabilistic model and determines the community based on the likelihood of triangular motifs. The relationship between observing two pairs of nodes in multiple communities leads to an increasing likelihood estimation of the existence of a motif structure between them. The output of the proposed model is the intensity of each node in the communities. The efficiency and validity of the proposed method are evaluated through experimental work on both synthetic and real-world networks; the findings will show that the community identified by the proposed method is more accurate and dense than other algorithms with modularity, NMI, and F1score evaluation metrics.

A multi-demand operating system underlying diverse cognitive tasks

The existence of a multiple-demand cortical system with an adaptive, domain-general, role in cognition has been proposed, but the underlying dynamic mechanisms and their links to cognitive control abilities are poorly understood. Here we use a probabilistic generative Bayesian model of brain circuit dynamics to determine dynamic brain states across multiple cognitive domains, independent datasets, and participant groups, including task fMRI data from Human Connectome Project, Dual Mechanisms of Cognitive Control study and a neurodevelopment study. We discover a shared brain state across seven distinct cognitive tasks and found that the dynamics of this shared brain state predicted cognitive control abilities in each task. Our findings reveal the flexible engagement of dynamic brain processes across multiple cognitive domains and participant groups, and uncover the generative mechanisms underlying the functioning of a domain-general cognitive operating system. Our computational framework opens promising avenues for probing neurocognitive function and dysfunction.

Use of noisy labels as weak learners to identify incompletely ascertainable outcomes: A Feasibility study with opioid-induced respiratory depression

ObjectiveAssigning outcome labels to large observational data sets in a timely and accurate manner, particularly when outcomes are rare or not directly ascertainable, remains a significant challenge within biomedical informatics. We examined whether noisy labels generated from subject matter experts’ heuristics using heterogenous data types within a data programming paradigm could provide outcomes labels to a large, observational data set. We chose the clinical condition of opioid-induced respiratory depression for our use case because it is rare, has no administrative codes to easily identify the condition, and typically requires at least some unstructured text to ascertain its presence. Materials and methodsUsing de-identified electronic health records of 52,861 post-operative encounters, we applied a data programming paradigm (implemented in the Snorkel software) for the development of a machine learning classifier for opioid-induced respiratory depression. Our approach included subject matter experts creating 14 labeling functions that served as noisy labels for developing a probabilistic Generative model. We used probabilistic labels from the Generative model as outcome labels for training a Discriminative model on the source data. We evaluated performance of the Discriminative model with a hold-out test set of 599 independently-reviewed patient records. ResultsThe final Discriminative classification model achieved an accuracy of 0.977, an F1 score of 0.417, a sensitivity of 1.0, and an AUC of 0.988 in the hold-out test set with a prevalence of 0.83% (5/599). DiscussionAll of the confirmed Cases were identified by the classifier. For rare outcomes, this finding is encouraging because it reduces the number of manual reviews needed by excluding visits/patients with low probabilities. ConclusionApplication of a data programming paradigm with expert-informed labeling functions might have utility for phenotyping clinical phenomena that are not easily ascertainable from highly-structured data.

Unveiling Insights: A Knowledge Discovery Approach to Comparing Topic Modeling Techniques in Digital Health Research

This paper introduces a knowledge discovery approach focused on comparing topic modeling techniques within the realm of digital health research. Knowledge discovery has been applied in massive data repositories (databases) and also in various field studies, which use these techniques for finding patterns in the data, determining which models and parameters might be suitable, and looking for patterns of interest in a specific representational. Unfortunately, the investigation delves into the utilization of Latent Dirichlet Allocation (LDA) and Pachinko Allocation Models (PAM) as generative probabilistic models in knowledge discovery, which is still limited. The study's findings position PAM as the superior technique, showcasing the greatest number of distinctive tokens per topic and the fastest processing time. Notably, PAM identifies 87 unique tokens across 10 topics, surpassing LDA Gensim's identification of only 27 unique tokens. Furthermore, PAM demonstrates remarkable efficiency by swiftly processing 404 documents within an incredibly short span of 0.000118970870 seconds, in contrast to LDA Gensim's considerably longer processing time of 0.368770837783 seconds. Ultimately, PAM emerges as the optimum method for digital health research's topic modeling, boasting unmatched efficiency in analyzing extensive digital health text data.

A probabilistic generative model for tracking multi-knowledge concept mastery probability

A probabilistic generative model for tracking multi-knowledge concept mastery probability

Modelling dataset bias in machine-learned theories of economic decision-making

Normative and descriptive models have long vied to explain and predict human risky choices, such as those between goods or gambles. A recent study reported the discovery of a new, more accurate model of human decision-making by training neural networks on a new online large-scale dataset, choices13k. Here we systematically analyse the relationships between several models and datasets using machine-learning methods and find evidence for dataset bias. Because participants’ choices in stochastically dominated gambles were consistently skewed towards equipreference in the choices13k dataset, we hypothesized that this reflected increased decision noise. Indeed, a probabilistic generative model adding structured decision noise to a neural network trained on data from a laboratory study transferred best, that is, outperformed all models apart from those trained on choices13k. We conclude that a careful combination of theory and data analysis is still required to understand the complex interactions of machine-learning models and data of human risky choices.

Biomedical Image Segmentation Using Denoising Diffusion Probabilistic Models: A Comprehensive Review and Analysis

Biomedical image segmentation plays a pivotal role in medical imaging, facilitating precise identification and delineation of anatomical structures and abnormalities. This review explores the application of the Denoising Diffusion Probabilistic Model (DDPM) in the realm of biomedical image segmentation. DDPM, a probabilistic generative model, has demonstrated promise in capturing complex data distributions and reducing noise in various domains. In this context, the review provides an in-depth examination of the present status, obstacles, and future prospects in the application of biomedical image segmentation techniques. It addresses challenges associated with the uncertainty and variability in imaging data analyzing commonalities based on probabilistic methods. The paper concludes with insights into the potential impact of DDPM on advancing medical imaging techniques and fostering reliable segmentation results in clinical applications. This comprehensive review aims to provide researchers, practitioners, and healthcare professionals with a nuanced understanding of the current state, challenges, and future prospects of utilizing DDPM in the context of biomedical image segmentation.

Latent Dirichlet Allocation (LDA) topic models for Space Syntax studies on spatial experience

Spatial experience has been extensively researched in various fields, with Space Syntax being one of the most widely used methodologies. Multiple Space Syntax techniques have been developed and used to quantitively examine the relationship between spatial configuration and human experience. However, due to the heterogeneity of syntactic measures and experiential issues in the built environment, a systematic review of socio-spatial topics has yet to be developed for Space Syntax research. In response to this knowledge gap, this article employs an ‘intelligent’ method to classify and systematically review topics in Space Syntax studies on spatial experience. Specifically, after identifying 66 articles using the ‘Preferred Reporting Items for Systematic reviews and Meta-Analyses’ (PRISMA) framework, this research develops generative probabilistic topic models to classify the articles using the Latent Dirichlet Allocation (LDA) method. As a result, this research automatically generates three architectural topics from the collected literature data (A1. Wayfinding behaviour, A2. Interactive accessibility, and A3. Healthcare design) and three urban topics (U1. Pedestrian movement, U2. Park accessibility, and U3. Cognitive city). Thereafter it qualitatively examines the implications of the data and its LDA classification. This article concludes with an examination of the limitations of both the methods and the results. Along with demonstrating a methodological innovation (combining PRISMA with LDA), this research identifies critical socio-spatial concepts and examines the complexity of Space Syntax applications. In this way, this research contributes to future Space Syntax research that empirically investigates the relationships between syntactic and experiential variables in architectural and urban spaces. The findings support a detailed discussion about research gaps in the literature and future research directions.

Probabilistic Keyphrase Generation From Copy and Generating Spaces.

Keyphrase generation is one of the most fundamental tasks in natural language processing (NLP). Most existing works on keyphrase generation mainly focus on using holistic distribution to optimize the negative log-likelihood loss, but they do not directly manipulate the copy and generating spaces, which may reduce the generability of the decoder. Additionally, existing keyphrase models are either unable to determine the dynamic numbers of keyphrases or produce the number of keyphrases implicitly. In this article, we propose a probabilistic keyphrase generation model from copy and generating spaces. The proposed model is built upon the vanilla variational encoder-decoder (VED) framework. On top of VED, two separate latent variables are adopted to model the distribution of data within the latent copy and generating spaces, respectively. Specifically, we adopt a von Mises-Fisher (vMF) distribution to obtain a condensed variable for modifying the generating probability distribution over the predefined vocabulary. Meanwhile, we utilize a clustering module, which is designed to promote Gaussian Mixture learning and subsequently extract a latent variable for the copy probability distribution. Moreover, we utilize a natural property of the Gaussian mixture network and use the number of filtered components to determine the number of keyphrases. The approach is trained based on latent variable probabilistic modeling, neural variational inference, and self-supervised learning. Experiments on social media and scientific article datasets outperform the state-of-the-art baselines in generating accurate predictions and controllable keyphrase numbers.

Temporal Dual-Attributed Network Generation Oriented Community Detection Model

Community detection is a crucial task on the research field of network analysis. However, this task recently has become challenging due to the explosion of network in terms of the scale and the side information, e.g., temporal information and attribute information. Here we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PGMTAN</i> —a probabilistic generative model for overlapping community detection on temporal dual-attributed networks. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PGMTAN</i> aims to characterize four generation processes: 1) generation of occurrence of the links, 2) generation of node-community memberships via assortative attributes, 3) generation of generative attributes, and 4) generation of evolutionary dynamics of community structure. Particularly, we adopt a hidden Markov chain model to capture the network's dynamics on the evolution of community structure over time. Moreover, we seek to optimize a lower-bound of likelihood function to accelerate the model's parameter estimation. We carry out extensive experiments on several real-world and synthetic networks to test <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">PGMTAN</i> 's performance and the results substantiate that it can outperform multiple baselines and give us promising performance in terms of detection accuracy and convergence.

QARV: Quantization-Aware ResNet VAE for Lossy Image Compression.

This paper addresses the problem of lossy image compression, a fundamental problem in image processing and information theory that is involved in many real-world applications. We start by reviewing the framework of variational autoencoders (VAEs), a powerful class of generative probabilistic models that has a deep connection to lossy compression. Based on VAEs, we develop a new scheme for lossy image compression, which we name quantization-aware ResNet VAE (QARV). Our method incorporates a hierarchical VAE architecture integrated with test-time quantization and quantization-aware training, without which efficient entropy coding would not be possible. In addition, we design the neural network architecture of QARV specifically for fast decoding and propose an adaptive normalization operation for variable-rate compression. Extensive experiments are conducted, and results show that QARV achieves variable-rate compression, high-speed decoding, and better rate-distortion performance than existing baseline methods.

BiVAE를 활용한 MBTI 기반 OTT 서비스 개인화 추천 시스템

The OTT platform, which provides various media contents in the modern society, seeks to improve the user experience by utilizing a personalized recommendation system. However, current recommendation systems are having difficulty in making accurate recommendations because they do not fully understand user's tendencies. This study proposes the use of a personalized recommendation algorithm using BiVAE (bilateral variational autoencoder) to solve this problem, and considers group-specific applications through MBTI. BiVAE is a probabilistic generation model that compresses and generates data considering of bidirectional information flow, providing richer expression and reconstruction capabilities by encoding and decoding in both directions. As a result of applying four types of data to the model, when applying BiVAE for each group using MBTI, the accuracy of OTT genre recommendation was shown to be superior to that of other methods. In addition, when considering MBTI, a personal recommendation using BiVAE showed higher accuracy compared to the existing recommendation system. This result means that the personalized recommendation system considering the user's personal characteristics and personality is important.

Digital polycrystalline microstructure generation using diffusion probabilistic models

Accurate micromechanical simulation of polycrystalline materials requires a realistic digital representation of the grain scale microstructure. This work demonstrates the use of a generative diffusion probabilistic model for synthesizing single phase polycrystalline realizations. The model performs well and is capable of producing realistic microstructures consisting of not just simple equiaxed structures but also structures exhibiting more complex spatial arrangements. Masked microstructure generation reveals that the model is context aware of morphological descriptors which may be encoded in the latent space. Training on more diverse data sets, with scaled up architectures, may enable development of future models capable of synthesizing even more complex microstructural features.

Graph meets probabilistic generation model: A new perspective for graph disentanglement

Different from the existing graph disentanglement neural networks, we interpret the graph entanglement under a probabilistic generation framework in this paper. With this foundation, a Mixed Probabilistic Generation Model induced Graph Disentanglement Network (MPGD) is proposed. Considering the disentangled components corresponding to different factors as obeying specific distributions, a generalized probabilistic aggregation scheme among components is deduced theoretically. As a key part of the mixed probabilistic generative model, we provide a solution for estimating the mixture probabilities using self-attention and an in-depth analysis of its close connection with the classical EM parameter estimation method. Meanwhile, a way of probabilistic aggregation is formulated to obtain the node representation in embedding space. In addition, the prior mixture probabilities are formulated as an auxiliary factor-aware representation to facilitate the twin-branch prediction. A variety of experiments show that MPGD achieves more competitive performance than some existing state-of-the-art methods while having ideal disentangling effects. The code implementation is available in https://github.com/GiorgioPeng/MPGD.

A probabilistic generative model to discover the treatments of coexisting diseases with missing data

Background and ObjectiveComorbidities, defined as the presence of co-existing diseases, progress through complex temporal patterns among patients. Learning such dynamics from electronic health records is crucial for understanding the coevolution of diseases. In general, medical records are represented through temporal sequences of clinical variables together with their diagnosis. However, we consider the specific problem where most of the diagnoses are missing. We present a novel probabilistic generative model with a three-fold objective: (i) identify and segment the medical history of patients into treatments associated with comorbidities; (ii) learn the model associated with each identified disease treatment; and (iii) discover subtypes of patients with similar coevolution of comorbidities. MethodsTo this end, the model considers a latent structure for the sequences, where patients are modeled by a latent class defined by the evolution of their comorbidities, and each observed medical event of their clinical history is associated with a latent disease. The learning process is performed using an Expectation-Maximization algorithm that considers the exponential number of configurations of the latent variables and is efficiently solved with dynamic programming. ResultsThe evaluation of the method is carried out both on synthetic and real world data: the experiments on synthetic data show that the learning procedure allows the generative model underlying the data to be recovered; the experiments on real medical data show accurate results in the segmentation of sequences into different treatments, subtyping of patients and diagnosis imputation. ConclusionWe present an interpretable generative model that handles the incompleteness of EHRs and describes the different joint evolution of coexisting diseases depending on the active comorbidities of the patient at each moment.

Recursive Metropolis-Hastings naming game: symbol emergence in a multi-agent system based on probabilistic generative models.

In the studies on symbol emergence and emergent communication in a population of agents, a computational model was employed in which agents participate in various language games. Among these, the Metropolis-Hastings naming game (MHNG) possesses a notable mathematical property: symbol emergence through MHNG is proven to be a decentralized Bayesian inference of representations shared by the agents. However, the previously proposed MHNG is limited to a two-agent scenario. This paper extends MHNG to an N-agent scenario. The main contributions of this paper are twofold: (1) we propose the recursive Metropolis-Hastings naming game (RMHNG) as an N-agent version of MHNG and demonstrate that RMHNG is an approximate Bayesian inference method for the posterior distribution over a latent variable shared by agents, similar to MHNG; and (2) we empirically evaluate the performance of RMHNG on synthetic and real image data, i.e., YCB object dataset, enabling multiple agents to develop and share a symbol system. Furthermore, we introduce two types of approximations-one-sample and limited-length-to reduce computational complexity while maintaining the ability to explain communication in a population of agents. The experimental findings showcased the efficacy of RMHNG as a decentralized Bayesian inference for approximating the posterior distribution concerning latent variables, which are jointly shared among agents, akin to MHNG, although the improvement in ARI and κ coefficient is smaller in the real image dataset condition. Moreover, the utilization of RMHNG elucidated the agents' capacity to exchange symbols. Furthermore, the study discovered that even the computationally simplified version of RMHNG could enable symbols to emerge among the agents.