Probabilistic Generative Model Research Articles

MorphVAE: Advancing Morphological Design of Voxel-Based Soft Robots with Variational Autoencoders

Soft robot design is an intricate field with unique challenges due to its complex and vast search space. In the past literature, evolutionary computation algorithms, including novel probabilistic generative models (PGMs), have shown potential in this realm. However, these methods are sample inefficient and predominantly focus on rigid robots in locomotion tasks, which limit their performance and application in robot design automation. In this work, we propose MorphVAE, an innovative PGM that incorporates a multi-task training scheme and a meticulously crafted sampling technique termed ``continuous natural selection'', aimed at bolstering sample efficiency. This method empowers us to gain insights from assessed samples across diverse tasks and temporal evolutionary stages, while simultaneously maintaining a delicate balance between optimization efficiency and biodiversity. Through extensive experiments in various locomotion and manipulation tasks, we substantiate the efficiency of MorphVAE in generating high-performing and diverse designs, surpassing the performance of competitive baselines.

A motif-based probabilistic approach for community detection in complex networks

Community detection in complex networks is an important task for discovering hidden information in network analysis. Neighborhood density between nodes is one of the fundamental indicators of community presence in the network. A community with a high edge density will have correlations between nodes that extend beyond their immediate neighbors, denoted by motifs. Motifs are repetitive patterns of edges observed with high frequency in the network. We proposed the PCDMS method (Probabilistic Community Detection with Motif Structure) that detects communities by estimating the triangular motif in the network. This study employs structural density between nodes, a key concept in graph analysis. The proposed model has the advantage of using a probabilistic generative model that calculates the latent parameters of the probabilistic model and determines the community based on the likelihood of triangular motifs. The relationship between observing two pairs of nodes in multiple communities leads to an increasing likelihood estimation of the existence of a motif structure between them. The output of the proposed model is the intensity of each node in the communities. The efficiency and validity of the proposed method are evaluated through experimental work on both synthetic and real-world networks; the findings will show that the community identified by the proposed method is more accurate and dense than other algorithms with modularity, NMI, and F1score evaluation metrics.

A multi-demand operating system underlying diverse cognitive tasks

The existence of a multiple-demand cortical system with an adaptive, domain-general, role in cognition has been proposed, but the underlying dynamic mechanisms and their links to cognitive control abilities are poorly understood. Here we use a probabilistic generative Bayesian model of brain circuit dynamics to determine dynamic brain states across multiple cognitive domains, independent datasets, and participant groups, including task fMRI data from Human Connectome Project, Dual Mechanisms of Cognitive Control study and a neurodevelopment study. We discover a shared brain state across seven distinct cognitive tasks and found that the dynamics of this shared brain state predicted cognitive control abilities in each task. Our findings reveal the flexible engagement of dynamic brain processes across multiple cognitive domains and participant groups, and uncover the generative mechanisms underlying the functioning of a domain-general cognitive operating system. Our computational framework opens promising avenues for probing neurocognitive function and dysfunction.

Use of noisy labels as weak learners to identify incompletely ascertainable outcomes: A Feasibility study with opioid-induced respiratory depression

ObjectiveAssigning outcome labels to large observational data sets in a timely and accurate manner, particularly when outcomes are rare or not directly ascertainable, remains a significant challenge within biomedical informatics. We examined whether noisy labels generated from subject matter experts’ heuristics using heterogenous data types within a data programming paradigm could provide outcomes labels to a large, observational data set. We chose the clinical condition of opioid-induced respiratory depression for our use case because it is rare, has no administrative codes to easily identify the condition, and typically requires at least some unstructured text to ascertain its presence. Materials and methodsUsing de-identified electronic health records of 52,861 post-operative encounters, we applied a data programming paradigm (implemented in the Snorkel software) for the development of a machine learning classifier for opioid-induced respiratory depression. Our approach included subject matter experts creating 14 labeling functions that served as noisy labels for developing a probabilistic Generative model. We used probabilistic labels from the Generative model as outcome labels for training a Discriminative model on the source data. We evaluated performance of the Discriminative model with a hold-out test set of 599 independently-reviewed patient records. ResultsThe final Discriminative classification model achieved an accuracy of 0.977, an F1 score of 0.417, a sensitivity of 1.0, and an AUC of 0.988 in the hold-out test set with a prevalence of 0.83% (5/599). DiscussionAll of the confirmed Cases were identified by the classifier. For rare outcomes, this finding is encouraging because it reduces the number of manual reviews needed by excluding visits/patients with low probabilities. ConclusionApplication of a data programming paradigm with expert-informed labeling functions might have utility for phenotyping clinical phenomena that are not easily ascertainable from highly-structured data.

Unveiling Insights: A Knowledge Discovery Approach to Comparing Topic Modeling Techniques in Digital Health Research

This paper introduces a knowledge discovery approach focused on comparing topic modeling techniques within the realm of digital health research. Knowledge discovery has been applied in massive data repositories (databases) and also in various field studies, which use these techniques for finding patterns in the data, determining which models and parameters might be suitable, and looking for patterns of interest in a specific representational. Unfortunately, the investigation delves into the utilization of Latent Dirichlet Allocation (LDA) and Pachinko Allocation Models (PAM) as generative probabilistic models in knowledge discovery, which is still limited. The study's findings position PAM as the superior technique, showcasing the greatest number of distinctive tokens per topic and the fastest processing time. Notably, PAM identifies 87 unique tokens across 10 topics, surpassing LDA Gensim's identification of only 27 unique tokens. Furthermore, PAM demonstrates remarkable efficiency by swiftly processing 404 documents within an incredibly short span of 0.000118970870 seconds, in contrast to LDA Gensim's considerably longer processing time of 0.368770837783 seconds. Ultimately, PAM emerges as the optimum method for digital health research's topic modeling, boasting unmatched efficiency in analyzing extensive digital health text data.

A probabilistic generative model for tracking multi-knowledge concept mastery probability

A probabilistic generative model for tracking multi-knowledge concept mastery probability

Modelling dataset bias in machine-learned theories of economic decision-making

Normative and descriptive models have long vied to explain and predict human risky choices, such as those between goods or gambles. A recent study reported the discovery of a new, more accurate model of human decision-making by training neural networks on a new online large-scale dataset, choices13k. Here we systematically analyse the relationships between several models and datasets using machine-learning methods and find evidence for dataset bias. Because participants’ choices in stochastically dominated gambles were consistently skewed towards equipreference in the choices13k dataset, we hypothesized that this reflected increased decision noise. Indeed, a probabilistic generative model adding structured decision noise to a neural network trained on data from a laboratory study transferred best, that is, outperformed all models apart from those trained on choices13k. We conclude that a careful combination of theory and data analysis is still required to understand the complex interactions of machine-learning models and data of human risky choices.

Biomedical Image Segmentation Using Denoising Diffusion Probabilistic Models: A Comprehensive Review and Analysis

Biomedical image segmentation plays a pivotal role in medical imaging, facilitating precise identification and delineation of anatomical structures and abnormalities. This review explores the application of the Denoising Diffusion Probabilistic Model (DDPM) in the realm of biomedical image segmentation. DDPM, a probabilistic generative model, has demonstrated promise in capturing complex data distributions and reducing noise in various domains. In this context, the review provides an in-depth examination of the present status, obstacles, and future prospects in the application of biomedical image segmentation techniques. It addresses challenges associated with the uncertainty and variability in imaging data analyzing commonalities based on probabilistic methods. The paper concludes with insights into the potential impact of DDPM on advancing medical imaging techniques and fostering reliable segmentation results in clinical applications. This comprehensive review aims to provide researchers, practitioners, and healthcare professionals with a nuanced understanding of the current state, challenges, and future prospects of utilizing DDPM in the context of biomedical image segmentation.

Latent Dirichlet Allocation (LDA) topic models for Space Syntax studies on spatial experience

Spatial experience has been extensively researched in various fields, with Space Syntax being one of the most widely used methodologies. Multiple Space Syntax techniques have been developed and used to quantitively examine the relationship between spatial configuration and human experience. However, due to the heterogeneity of syntactic measures and experiential issues in the built environment, a systematic review of socio-spatial topics has yet to be developed for Space Syntax research. In response to this knowledge gap, this article employs an ‘intelligent’ method to classify and systematically review topics in Space Syntax studies on spatial experience. Specifically, after identifying 66 articles using the ‘Preferred Reporting Items for Systematic reviews and Meta-Analyses’ (PRISMA) framework, this research develops generative probabilistic topic models to classify the articles using the Latent Dirichlet Allocation (LDA) method. As a result, this research automatically generates three architectural topics from the collected literature data (A1. Wayfinding behaviour, A2. Interactive accessibility, and A3. Healthcare design) and three urban topics (U1. Pedestrian movement, U2. Park accessibility, and U3. Cognitive city). Thereafter it qualitatively examines the implications of the data and its LDA classification. This article concludes with an examination of the limitations of both the methods and the results. Along with demonstrating a methodological innovation (combining PRISMA with LDA), this research identifies critical socio-spatial concepts and examines the complexity of Space Syntax applications. In this way, this research contributes to future Space Syntax research that empirically investigates the relationships between syntactic and experiential variables in architectural and urban spaces. The findings support a detailed discussion about research gaps in the literature and future research directions.

Probabilistic Keyphrase Generation From Copy and Generating Spaces.

Keyphrase generation is one of the most fundamental tasks in natural language processing (NLP). Most existing works on keyphrase generation mainly focus on using holistic distribution to optimize the negative log-likelihood loss, but they do not directly manipulate the copy and generating spaces, which may reduce the generability of the decoder. Additionally, existing keyphrase models are either unable to determine the dynamic numbers of keyphrases or produce the number of keyphrases implicitly. In this article, we propose a probabilistic keyphrase generation model from copy and generating spaces. The proposed model is built upon the vanilla variational encoder-decoder (VED) framework. On top of VED, two separate latent variables are adopted to model the distribution of data within the latent copy and generating spaces, respectively. Specifically, we adopt a von Mises-Fisher (vMF) distribution to obtain a condensed variable for modifying the generating probability distribution over the predefined vocabulary. Meanwhile, we utilize a clustering module, which is designed to promote Gaussian Mixture learning and subsequently extract a latent variable for the copy probability distribution. Moreover, we utilize a natural property of the Gaussian mixture network and use the number of filtered components to determine the number of keyphrases. The approach is trained based on latent variable probabilistic modeling, neural variational inference, and self-supervised learning. Experiments on social media and scientific article datasets outperform the state-of-the-art baselines in generating accurate predictions and controllable keyphrase numbers.

Changen2: Multi-Temporal Remote Sensing Generative Change Foundation Model.

Our understanding of the temporal dynamics of the Earth's surface has been significantly advanced by deep vision models, which often require a massive amount of labeled multi-temporal images for training. However, collecting, preprocessing, and annotating multi-temporal remote sensing images at scale is non-trivial since it is expensive and knowledge-intensive. In this paper, we present scalable multi-temporal change data generators based on generative models, which are cheap and automatic, alleviating these data problems. Our main idea is to simulate a stochastic change process over time. We describe the stochastic change process as a probabilistic graphical model, namely the generative probabilistic change model (GPCM), which factorizes the complex simulation problem into two more tractable sub-problems, i.e., condition-level change event simulation and image-level semantic change synthesis. To solve these two problems, we present Changen2, a GPCM implemented with a resolution-scalable diffusion transformer which can generate time series of remote sensing images and corresponding semantic and change labels from labeled and even unlabeled single-temporal images. Changen2 is a "generative change foundation model" that can be trained at scale via self-supervision, and is capable of producing change supervisory signals from unlabeled single-temporal images. Unlike existing "foundation models", our generative change foundation model synthesizes change data to train task-specific foundation models for change detection. The resulting model possesses inherent zero-shot change detection capabilities and excellent transferability. Comprehensive experiments suggest Changen2 has superior spatiotemporal scalability in data generation, e.g., Changen2 model trained on 256 2 pixel single-temporal images can yield time series of any length and resolutions of 1,024 2 pixels. Changen2 pre-trained models exhibit superior zero-shot performance (narrowing the performance gap to 3% on LEVIR-CD and approximately 10% on both S2Looking and SECOND, compared to fully supervised counterpart) and transferability across multiple types of change tasks, including ordinary and off-nadir building change, land-use/land-cover change, and disaster assessment. The model and datasets are available at https://github.com/Z-Zheng/pytorch-change-models.

Hierarchical path planning from speech instructions with spatial concept-based topometric semantic mapping.

Assisting individuals in their daily activities through autonomous mobile robots is a significant concern, especially for users without specialized knowledge. Specifically, the capability of a robot to navigate to destinations based on human speech instructions is crucial. Although robots can take different paths toward the same objective, the shortest path is not always the most suitable. A preferred approach would be to accommodate waypoint specifications flexibly for planning an improved alternative path even with detours. Furthermore, robots require real-time inference capabilities. In this sense, spatial representations include semantic, topological, and metric-level representations, each capturing different aspects of the environment. This study aimed to realize a hierarchical spatial representation using a topometric semantic map and path planning with speech instructions by including waypoints. Thus, we present a hierarchical path planning method called spatial concept-based topometric semantic mapping for hierarchical path planning (SpCoTMHP), which integrates place connectivity. This approach provides a novel integrated probabilistic generative model and fast approximate inferences with interactions among the hierarchy levels. A formulation based on "control as probabilistic inference" theoretically supports the proposed path planning algorithm. We conducted experiments in a home environment using the Toyota human support robot on the SIGVerse simulator and in a lab-office environment with the real robot Albert. Here, the user issues speech commands that specify the waypoint and goal, such as "Go to the bedroom via the corridor." Navigation experiments were performed using speech instructions with a waypoint to demonstrate the performance improvement of the SpCoTMHP over the baseline hierarchical path planning method with heuristic path costs (HPP-I) in terms of the weighted success rate at which the robot reaches the closest target (0.590) and passes the correct waypoints. The computation time was significantly improved by 7.14 s with the SpCoTMHP than the baseline HPP-I in advanced tasks. Thus, hierarchical spatial representations provide mutually understandable instruction forms for both humans and robots, thus enabling language-based navigation.

QARV: Quantization-Aware ResNet VAE for Lossy Image Compression.

This paper addresses the problem of lossy image compression, a fundamental problem in image processing and information theory that is involved in many real-world applications. We start by reviewing the framework of variational autoencoders (VAEs), a powerful class of generative probabilistic models that has a deep connection to lossy compression. Based on VAEs, we develop a new scheme for lossy image compression, which we name quantization-aware ResNet VAE (QARV). Our method incorporates a hierarchical VAE architecture integrated with test-time quantization and quantization-aware training, without which efficient entropy coding would not be possible. In addition, we design the neural network architecture of QARV specifically for fast decoding and propose an adaptive normalization operation for variable-rate compression. Extensive experiments are conducted, and results show that QARV achieves variable-rate compression, high-speed decoding, and better rate-distortion performance than existing baseline methods.

BiVAE를 활용한 MBTI 기반 OTT 서비스 개인화 추천 시스템

The OTT platform, which provides various media contents in the modern society, seeks to improve the user experience by utilizing a personalized recommendation system. However, current recommendation systems are having difficulty in making accurate recommendations because they do not fully understand user's tendencies. This study proposes the use of a personalized recommendation algorithm using BiVAE (bilateral variational autoencoder) to solve this problem, and considers group-specific applications through MBTI. BiVAE is a probabilistic generation model that compresses and generates data considering of bidirectional information flow, providing richer expression and reconstruction capabilities by encoding and decoding in both directions. As a result of applying four types of data to the model, when applying BiVAE for each group using MBTI, the accuracy of OTT genre recommendation was shown to be superior to that of other methods. In addition, when considering MBTI, a personal recommendation using BiVAE showed higher accuracy compared to the existing recommendation system. This result means that the personalized recommendation system considering the user's personal characteristics and personality is important.

Digital polycrystalline microstructure generation using diffusion probabilistic models

Accurate micromechanical simulation of polycrystalline materials requires a realistic digital representation of the grain scale microstructure. This work demonstrates the use of a generative diffusion probabilistic model for synthesizing single phase polycrystalline realizations. The model performs well and is capable of producing realistic microstructures consisting of not just simple equiaxed structures but also structures exhibiting more complex spatial arrangements. Masked microstructure generation reveals that the model is context aware of morphological descriptors which may be encoded in the latent space. Training on more diverse data sets, with scaled up architectures, may enable development of future models capable of synthesizing even more complex microstructural features.

Graph meets probabilistic generation model: A new perspective for graph disentanglement

Different from the existing graph disentanglement neural networks, we interpret the graph entanglement under a probabilistic generation framework in this paper. With this foundation, a Mixed Probabilistic Generation Model induced Graph Disentanglement Network (MPGD) is proposed. Considering the disentangled components corresponding to different factors as obeying specific distributions, a generalized probabilistic aggregation scheme among components is deduced theoretically. As a key part of the mixed probabilistic generative model, we provide a solution for estimating the mixture probabilities using self-attention and an in-depth analysis of its close connection with the classical EM parameter estimation method. Meanwhile, a way of probabilistic aggregation is formulated to obtain the node representation in embedding space. In addition, the prior mixture probabilities are formulated as an auxiliary factor-aware representation to facilitate the twin-branch prediction. A variety of experiments show that MPGD achieves more competitive performance than some existing state-of-the-art methods while having ideal disentangling effects. The code implementation is available in https://github.com/GiorgioPeng/MPGD.

A probabilistic generative model to discover the treatments of coexisting diseases with missing data

Background and ObjectiveComorbidities, defined as the presence of co-existing diseases, progress through complex temporal patterns among patients. Learning such dynamics from electronic health records is crucial for understanding the coevolution of diseases. In general, medical records are represented through temporal sequences of clinical variables together with their diagnosis. However, we consider the specific problem where most of the diagnoses are missing. We present a novel probabilistic generative model with a three-fold objective: (i) identify and segment the medical history of patients into treatments associated with comorbidities; (ii) learn the model associated with each identified disease treatment; and (iii) discover subtypes of patients with similar coevolution of comorbidities. MethodsTo this end, the model considers a latent structure for the sequences, where patients are modeled by a latent class defined by the evolution of their comorbidities, and each observed medical event of their clinical history is associated with a latent disease. The learning process is performed using an Expectation-Maximization algorithm that considers the exponential number of configurations of the latent variables and is efficiently solved with dynamic programming. ResultsThe evaluation of the method is carried out both on synthetic and real world data: the experiments on synthetic data show that the learning procedure allows the generative model underlying the data to be recovered; the experiments on real medical data show accurate results in the segmentation of sequences into different treatments, subtyping of patients and diagnosis imputation. ConclusionWe present an interpretable generative model that handles the incompleteness of EHRs and describes the different joint evolution of coexisting diseases depending on the active comorbidities of the patient at each moment.

Beyond tree-shaped credal probabilistic circuits

Probabilistic circuits are a class of probabilistic generative models that allow us to compute different types of probabilistic queries in polynomial time. Unlike many of the mainstream approaches for generative modeling, they can compute exact likelihoods, marginals, and expectations. Yet, assessing the reliability of their inferences is not straightforward. Credal probabilistic circuits are the imprecise counterpart of probabilistic circuits allowing, among other queries, computations of cautious inferences and sensitivity analyses. In this work, we propose an efficient algorithm to compute the lower and upper expectations for factorizing functions using a credal probabilistic circuit. We discuss under what structural assumptions and types of factorizing functions the algorithm works. We prove that such algorithm has polynomial time complexity in the input size. In the general case, we prove that computing cautious inferences using credal probabilistic circuits is an NP-hard problem, yet the proposed algorithm can be used as an approximation. Some experiments show how the approximation degrades with the complexity of the model structure.

Recursive Metropolis-Hastings naming game: symbol emergence in a multi-agent system based on probabilistic generative models.

In the studies on symbol emergence and emergent communication in a population of agents, a computational model was employed in which agents participate in various language games. Among these, the Metropolis-Hastings naming game (MHNG) possesses a notable mathematical property: symbol emergence through MHNG is proven to be a decentralized Bayesian inference of representations shared by the agents. However, the previously proposed MHNG is limited to a two-agent scenario. This paper extends MHNG to an N-agent scenario. The main contributions of this paper are twofold: (1) we propose the recursive Metropolis-Hastings naming game (RMHNG) as an N-agent version of MHNG and demonstrate that RMHNG is an approximate Bayesian inference method for the posterior distribution over a latent variable shared by agents, similar to MHNG; and (2) we empirically evaluate the performance of RMHNG on synthetic and real image data, i.e., YCB object dataset, enabling multiple agents to develop and share a symbol system. Furthermore, we introduce two types of approximations-one-sample and limited-length-to reduce computational complexity while maintaining the ability to explain communication in a population of agents. The experimental findings showcased the efficacy of RMHNG as a decentralized Bayesian inference for approximating the posterior distribution concerning latent variables, which are jointly shared among agents, akin to MHNG, although the improvement in ARI and κ coefficient is smaller in the real image dataset condition. Moreover, the utilization of RMHNG elucidated the agents' capacity to exchange symbols. Furthermore, the study discovered that even the computationally simplified version of RMHNG could enable symbols to emerge among the agents.

Probabilistic forecast of nonlinear dynamical systems with uncertainty quantification

Data-driven modeling is useful for reconstructing nonlinear dynamical systems when the underlying process is unknown or too expensive to compute. Having reliable uncertainty assessment of the forecast enables tools to be deployed to predict new scenarios unobserved before. In this work, we first extend parallel partial Gaussian processes for predicting the vector-valued transition function that links the observations between the current and next time points, and quantify the uncertainty of predictions by posterior sampling. Second, we show the equivalence between the dynamic mode decomposition and the maximum likelihood estimator of the linear mapping matrix in the linear state space model. The connection provides a probabilistic generative model of dynamic mode decomposition and thus, uncertainty of predictions can be obtained. Furthermore, we draw close connections between different data-driven models for approximating nonlinear dynamics, through a unified view of generative models. We study two numerical examples, where the inputs of the dynamics are assumed to be known in the first example and the inputs are unknown in the second example. The examples indicate that uncertainty of forecast can be properly quantified, whereas model or input misspecification can degrade the accuracy of uncertainty quantification.