Probabilistic Generative Model Research Articles

Use of noisy labels as weak learners to identify incompletely ascertainable outcomes: A Feasibility study with opioid-induced respiratory depression

ObjectiveAssigning outcome labels to large observational data sets in a timely and accurate manner, particularly when outcomes are rare or not directly ascertainable, remains a significant challenge within biomedical informatics. We examined whether noisy labels generated from subject matter experts’ heuristics using heterogenous data types within a data programming paradigm could provide outcomes labels to a large, observational data set. We chose the clinical condition of opioid-induced respiratory depression for our use case because it is rare, has no administrative codes to easily identify the condition, and typically requires at least some unstructured text to ascertain its presence. Materials and methodsUsing de-identified electronic health records of 52,861 post-operative encounters, we applied a data programming paradigm (implemented in the Snorkel software) for the development of a machine learning classifier for opioid-induced respiratory depression. Our approach included subject matter experts creating 14 labeling functions that served as noisy labels for developing a probabilistic Generative model. We used probabilistic labels from the Generative model as outcome labels for training a Discriminative model on the source data. We evaluated performance of the Discriminative model with a hold-out test set of 599 independently-reviewed patient records. ResultsThe final Discriminative classification model achieved an accuracy of 0.977, an F1 score of 0.417, a sensitivity of 1.0, and an AUC of 0.988 in the hold-out test set with a prevalence of 0.83% (5/599). DiscussionAll of the confirmed Cases were identified by the classifier. For rare outcomes, this finding is encouraging because it reduces the number of manual reviews needed by excluding visits/patients with low probabilities. ConclusionApplication of a data programming paradigm with expert-informed labeling functions might have utility for phenotyping clinical phenomena that are not easily ascertainable from highly-structured data.

Unveiling Insights: A Knowledge Discovery Approach to Comparing Topic Modeling Techniques in Digital Health Research

This paper introduces a knowledge discovery approach focused on comparing topic modeling techniques within the realm of digital health research. Knowledge discovery has been applied in massive data repositories (databases) and also in various field studies, which use these techniques for finding patterns in the data, determining which models and parameters might be suitable, and looking for patterns of interest in a specific representational. Unfortunately, the investigation delves into the utilization of Latent Dirichlet Allocation (LDA) and Pachinko Allocation Models (PAM) as generative probabilistic models in knowledge discovery, which is still limited. The study's findings position PAM as the superior technique, showcasing the greatest number of distinctive tokens per topic and the fastest processing time. Notably, PAM identifies 87 unique tokens across 10 topics, surpassing LDA Gensim's identification of only 27 unique tokens. Furthermore, PAM demonstrates remarkable efficiency by swiftly processing 404 documents within an incredibly short span of 0.000118970870 seconds, in contrast to LDA Gensim's considerably longer processing time of 0.368770837783 seconds. Ultimately, PAM emerges as the optimum method for digital health research's topic modeling, boasting unmatched efficiency in analyzing extensive digital health text data.

A probabilistic generative model for tracking multi-knowledge concept mastery probability

A probabilistic generative model for tracking multi-knowledge concept mastery probability

Modelling dataset bias in machine-learned theories of economic decision-making

Normative and descriptive models have long vied to explain and predict human risky choices, such as those between goods or gambles. A recent study reported the discovery of a new, more accurate model of human decision-making by training neural networks on a new online large-scale dataset, choices13k. Here we systematically analyse the relationships between several models and datasets using machine-learning methods and find evidence for dataset bias. Because participants’ choices in stochastically dominated gambles were consistently skewed towards equipreference in the choices13k dataset, we hypothesized that this reflected increased decision noise. Indeed, a probabilistic generative model adding structured decision noise to a neural network trained on data from a laboratory study transferred best, that is, outperformed all models apart from those trained on choices13k. We conclude that a careful combination of theory and data analysis is still required to understand the complex interactions of machine-learning models and data of human risky choices.

Biomedical Image Segmentation Using Denoising Diffusion Probabilistic Models: A Comprehensive Review and Analysis

Biomedical image segmentation plays a pivotal role in medical imaging, facilitating precise identification and delineation of anatomical structures and abnormalities. This review explores the application of the Denoising Diffusion Probabilistic Model (DDPM) in the realm of biomedical image segmentation. DDPM, a probabilistic generative model, has demonstrated promise in capturing complex data distributions and reducing noise in various domains. In this context, the review provides an in-depth examination of the present status, obstacles, and future prospects in the application of biomedical image segmentation techniques. It addresses challenges associated with the uncertainty and variability in imaging data analyzing commonalities based on probabilistic methods. The paper concludes with insights into the potential impact of DDPM on advancing medical imaging techniques and fostering reliable segmentation results in clinical applications. This comprehensive review aims to provide researchers, practitioners, and healthcare professionals with a nuanced understanding of the current state, challenges, and future prospects of utilizing DDPM in the context of biomedical image segmentation.

Latent Dirichlet Allocation (LDA) topic models for Space Syntax studies on spatial experience

Spatial experience has been extensively researched in various fields, with Space Syntax being one of the most widely used methodologies. Multiple Space Syntax techniques have been developed and used to quantitively examine the relationship between spatial configuration and human experience. However, due to the heterogeneity of syntactic measures and experiential issues in the built environment, a systematic review of socio-spatial topics has yet to be developed for Space Syntax research. In response to this knowledge gap, this article employs an ‘intelligent’ method to classify and systematically review topics in Space Syntax studies on spatial experience. Specifically, after identifying 66 articles using the ‘Preferred Reporting Items for Systematic reviews and Meta-Analyses’ (PRISMA) framework, this research develops generative probabilistic topic models to classify the articles using the Latent Dirichlet Allocation (LDA) method. As a result, this research automatically generates three architectural topics from the collected literature data (A1. Wayfinding behaviour, A2. Interactive accessibility, and A3. Healthcare design) and three urban topics (U1. Pedestrian movement, U2. Park accessibility, and U3. Cognitive city). Thereafter it qualitatively examines the implications of the data and its LDA classification. This article concludes with an examination of the limitations of both the methods and the results. Along with demonstrating a methodological innovation (combining PRISMA with LDA), this research identifies critical socio-spatial concepts and examines the complexity of Space Syntax applications. In this way, this research contributes to future Space Syntax research that empirically investigates the relationships between syntactic and experiential variables in architectural and urban spaces. The findings support a detailed discussion about research gaps in the literature and future research directions.

Probabilistic Keyphrase Generation From Copy and Generating Spaces.

Keyphrase generation is one of the most fundamental tasks in natural language processing (NLP). Most existing works on keyphrase generation mainly focus on using holistic distribution to optimize the negative log-likelihood loss, but they do not directly manipulate the copy and generating spaces, which may reduce the generability of the decoder. Additionally, existing keyphrase models are either unable to determine the dynamic numbers of keyphrases or produce the number of keyphrases implicitly. In this article, we propose a probabilistic keyphrase generation model from copy and generating spaces. The proposed model is built upon the vanilla variational encoder-decoder (VED) framework. On top of VED, two separate latent variables are adopted to model the distribution of data within the latent copy and generating spaces, respectively. Specifically, we adopt a von Mises-Fisher (vMF) distribution to obtain a condensed variable for modifying the generating probability distribution over the predefined vocabulary. Meanwhile, we utilize a clustering module, which is designed to promote Gaussian Mixture learning and subsequently extract a latent variable for the copy probability distribution. Moreover, we utilize a natural property of the Gaussian mixture network and use the number of filtered components to determine the number of keyphrases. The approach is trained based on latent variable probabilistic modeling, neural variational inference, and self-supervised learning. Experiments on social media and scientific article datasets outperform the state-of-the-art baselines in generating accurate predictions and controllable keyphrase numbers.

Changen2: Multi-Temporal Remote Sensing Generative Change Foundation Model.

Our understanding of the temporal dynamics of the Earth's surface has been significantly advanced by deep vision models, which often require a massive amount of labeled multi-temporal images for training. However, collecting, preprocessing, and annotating multi-temporal remote sensing images at scale is non-trivial since it is expensive and knowledge-intensive. In this paper, we present scalable multi-temporal change data generators based on generative models, which are cheap and automatic, alleviating these data problems. Our main idea is to simulate a stochastic change process over time. We describe the stochastic change process as a probabilistic graphical model, namely the generative probabilistic change model (GPCM), which factorizes the complex simulation problem into two more tractable sub-problems, i.e., condition-level change event simulation and image-level semantic change synthesis. To solve these two problems, we present Changen2, a GPCM implemented with a resolution-scalable diffusion transformer which can generate time series of remote sensing images and corresponding semantic and change labels from labeled and even unlabeled single-temporal images. Changen2 is a "generative change foundation model" that can be trained at scale via self-supervision, and is capable of producing change supervisory signals from unlabeled single-temporal images. Unlike existing "foundation models", our generative change foundation model synthesizes change data to train task-specific foundation models for change detection. The resulting model possesses inherent zero-shot change detection capabilities and excellent transferability. Comprehensive experiments suggest Changen2 has superior spatiotemporal scalability in data generation, e.g., Changen2 model trained on 256 2 pixel single-temporal images can yield time series of any length and resolutions of 1,024 2 pixels. Changen2 pre-trained models exhibit superior zero-shot performance (narrowing the performance gap to 3% on LEVIR-CD and approximately 10% on both S2Looking and SECOND, compared to fully supervised counterpart) and transferability across multiple types of change tasks, including ordinary and off-nadir building change, land-use/land-cover change, and disaster assessment. The model and datasets are available at https://github.com/Z-Zheng/pytorch-change-models.

Hierarchical path planning from speech instructions with spatial concept-based topometric semantic mapping.

Assisting individuals in their daily activities through autonomous mobile robots is a significant concern, especially for users without specialized knowledge. Specifically, the capability of a robot to navigate to destinations based on human speech instructions is crucial. Although robots can take different paths toward the same objective, the shortest path is not always the most suitable. A preferred approach would be to accommodate waypoint specifications flexibly for planning an improved alternative path even with detours. Furthermore, robots require real-time inference capabilities. In this sense, spatial representations include semantic, topological, and metric-level representations, each capturing different aspects of the environment. This study aimed to realize a hierarchical spatial representation using a topometric semantic map and path planning with speech instructions by including waypoints. Thus, we present a hierarchical path planning method called spatial concept-based topometric semantic mapping for hierarchical path planning (SpCoTMHP), which integrates place connectivity. This approach provides a novel integrated probabilistic generative model and fast approximate inferences with interactions among the hierarchy levels. A formulation based on "control as probabilistic inference" theoretically supports the proposed path planning algorithm. We conducted experiments in a home environment using the Toyota human support robot on the SIGVerse simulator and in a lab-office environment with the real robot Albert. Here, the user issues speech commands that specify the waypoint and goal, such as "Go to the bedroom via the corridor." Navigation experiments were performed using speech instructions with a waypoint to demonstrate the performance improvement of the SpCoTMHP over the baseline hierarchical path planning method with heuristic path costs (HPP-I) in terms of the weighted success rate at which the robot reaches the closest target (0.590) and passes the correct waypoints. The computation time was significantly improved by 7.14 s with the SpCoTMHP than the baseline HPP-I in advanced tasks. Thus, hierarchical spatial representations provide mutually understandable instruction forms for both humans and robots, thus enabling language-based navigation.

QARV: Quantization-Aware ResNet VAE for Lossy Image Compression.

This paper addresses the problem of lossy image compression, a fundamental problem in image processing and information theory that is involved in many real-world applications. We start by reviewing the framework of variational autoencoders (VAEs), a powerful class of generative probabilistic models that has a deep connection to lossy compression. Based on VAEs, we develop a new scheme for lossy image compression, which we name quantization-aware ResNet VAE (QARV). Our method incorporates a hierarchical VAE architecture integrated with test-time quantization and quantization-aware training, without which efficient entropy coding would not be possible. In addition, we design the neural network architecture of QARV specifically for fast decoding and propose an adaptive normalization operation for variable-rate compression. Extensive experiments are conducted, and results show that QARV achieves variable-rate compression, high-speed decoding, and better rate-distortion performance than existing baseline methods.

BiVAE를 활용한 MBTI 기반 OTT 서비스 개인화 추천 시스템

The OTT platform, which provides various media contents in the modern society, seeks to improve the user experience by utilizing a personalized recommendation system. However, current recommendation systems are having difficulty in making accurate recommendations because they do not fully understand user's tendencies. This study proposes the use of a personalized recommendation algorithm using BiVAE (bilateral variational autoencoder) to solve this problem, and considers group-specific applications through MBTI. BiVAE is a probabilistic generation model that compresses and generates data considering of bidirectional information flow, providing richer expression and reconstruction capabilities by encoding and decoding in both directions. As a result of applying four types of data to the model, when applying BiVAE for each group using MBTI, the accuracy of OTT genre recommendation was shown to be superior to that of other methods. In addition, when considering MBTI, a personal recommendation using BiVAE showed higher accuracy compared to the existing recommendation system. This result means that the personalized recommendation system considering the user's personal characteristics and personality is important.

Digital polycrystalline microstructure generation using diffusion probabilistic models

Accurate micromechanical simulation of polycrystalline materials requires a realistic digital representation of the grain scale microstructure. This work demonstrates the use of a generative diffusion probabilistic model for synthesizing single phase polycrystalline realizations. The model performs well and is capable of producing realistic microstructures consisting of not just simple equiaxed structures but also structures exhibiting more complex spatial arrangements. Masked microstructure generation reveals that the model is context aware of morphological descriptors which may be encoded in the latent space. Training on more diverse data sets, with scaled up architectures, may enable development of future models capable of synthesizing even more complex microstructural features.

Graph meets probabilistic generation model: A new perspective for graph disentanglement

Different from the existing graph disentanglement neural networks, we interpret the graph entanglement under a probabilistic generation framework in this paper. With this foundation, a Mixed Probabilistic Generation Model induced Graph Disentanglement Network (MPGD) is proposed. Considering the disentangled components corresponding to different factors as obeying specific distributions, a generalized probabilistic aggregation scheme among components is deduced theoretically. As a key part of the mixed probabilistic generative model, we provide a solution for estimating the mixture probabilities using self-attention and an in-depth analysis of its close connection with the classical EM parameter estimation method. Meanwhile, a way of probabilistic aggregation is formulated to obtain the node representation in embedding space. In addition, the prior mixture probabilities are formulated as an auxiliary factor-aware representation to facilitate the twin-branch prediction. A variety of experiments show that MPGD achieves more competitive performance than some existing state-of-the-art methods while having ideal disentangling effects. The code implementation is available in https://github.com/GiorgioPeng/MPGD.

A probabilistic generative model to discover the treatments of coexisting diseases with missing data

Background and ObjectiveComorbidities, defined as the presence of co-existing diseases, progress through complex temporal patterns among patients. Learning such dynamics from electronic health records is crucial for understanding the coevolution of diseases. In general, medical records are represented through temporal sequences of clinical variables together with their diagnosis. However, we consider the specific problem where most of the diagnoses are missing. We present a novel probabilistic generative model with a three-fold objective: (i) identify and segment the medical history of patients into treatments associated with comorbidities; (ii) learn the model associated with each identified disease treatment; and (iii) discover subtypes of patients with similar coevolution of comorbidities. MethodsTo this end, the model considers a latent structure for the sequences, where patients are modeled by a latent class defined by the evolution of their comorbidities, and each observed medical event of their clinical history is associated with a latent disease. The learning process is performed using an Expectation-Maximization algorithm that considers the exponential number of configurations of the latent variables and is efficiently solved with dynamic programming. ResultsThe evaluation of the method is carried out both on synthetic and real world data: the experiments on synthetic data show that the learning procedure allows the generative model underlying the data to be recovered; the experiments on real medical data show accurate results in the segmentation of sequences into different treatments, subtyping of patients and diagnosis imputation. ConclusionWe present an interpretable generative model that handles the incompleteness of EHRs and describes the different joint evolution of coexisting diseases depending on the active comorbidities of the patient at each moment.

Beyond tree-shaped credal probabilistic circuits

Probabilistic circuits are a class of probabilistic generative models that allow us to compute different types of probabilistic queries in polynomial time. Unlike many of the mainstream approaches for generative modeling, they can compute exact likelihoods, marginals, and expectations. Yet, assessing the reliability of their inferences is not straightforward. Credal probabilistic circuits are the imprecise counterpart of probabilistic circuits allowing, among other queries, computations of cautious inferences and sensitivity analyses. In this work, we propose an efficient algorithm to compute the lower and upper expectations for factorizing functions using a credal probabilistic circuit. We discuss under what structural assumptions and types of factorizing functions the algorithm works. We prove that such algorithm has polynomial time complexity in the input size. In the general case, we prove that computing cautious inferences using credal probabilistic circuits is an NP-hard problem, yet the proposed algorithm can be used as an approximation. Some experiments show how the approximation degrades with the complexity of the model structure.

Recursive Metropolis-Hastings naming game: symbol emergence in a multi-agent system based on probabilistic generative models.

In the studies on symbol emergence and emergent communication in a population of agents, a computational model was employed in which agents participate in various language games. Among these, the Metropolis-Hastings naming game (MHNG) possesses a notable mathematical property: symbol emergence through MHNG is proven to be a decentralized Bayesian inference of representations shared by the agents. However, the previously proposed MHNG is limited to a two-agent scenario. This paper extends MHNG to an N-agent scenario. The main contributions of this paper are twofold: (1) we propose the recursive Metropolis-Hastings naming game (RMHNG) as an N-agent version of MHNG and demonstrate that RMHNG is an approximate Bayesian inference method for the posterior distribution over a latent variable shared by agents, similar to MHNG; and (2) we empirically evaluate the performance of RMHNG on synthetic and real image data, i.e., YCB object dataset, enabling multiple agents to develop and share a symbol system. Furthermore, we introduce two types of approximations-one-sample and limited-length-to reduce computational complexity while maintaining the ability to explain communication in a population of agents. The experimental findings showcased the efficacy of RMHNG as a decentralized Bayesian inference for approximating the posterior distribution concerning latent variables, which are jointly shared among agents, akin to MHNG, although the improvement in ARI and κ coefficient is smaller in the real image dataset condition. Moreover, the utilization of RMHNG elucidated the agents' capacity to exchange symbols. Furthermore, the study discovered that even the computationally simplified version of RMHNG could enable symbols to emerge among the agents.

Probabilistic forecast of nonlinear dynamical systems with uncertainty quantification

Data-driven modeling is useful for reconstructing nonlinear dynamical systems when the underlying process is unknown or too expensive to compute. Having reliable uncertainty assessment of the forecast enables tools to be deployed to predict new scenarios unobserved before. In this work, we first extend parallel partial Gaussian processes for predicting the vector-valued transition function that links the observations between the current and next time points, and quantify the uncertainty of predictions by posterior sampling. Second, we show the equivalence between the dynamic mode decomposition and the maximum likelihood estimator of the linear mapping matrix in the linear state space model. The connection provides a probabilistic generative model of dynamic mode decomposition and thus, uncertainty of predictions can be obtained. Furthermore, we draw close connections between different data-driven models for approximating nonlinear dynamics, through a unified view of generative models. We study two numerical examples, where the inputs of the dynamics are assumed to be known in the first example and the inputs are unknown in the second example. The examples indicate that uncertainty of forecast can be properly quantified, whereas model or input misspecification can degrade the accuracy of uncertainty quantification.

Emergent communication through Metropolis-Hastings naming game with deep generative models

Constructive studies on symbol emergence systems seek to investigate computational models that can better explain human language evolution, the creation of symbol systems, and the construction of internal representations. Specifically, emergent communication aims to formulate a computational model that enables agents to build efficient sign systems and internal representations. This study provides a new model for emergent communication, which is based on a probabilistic generative model (PGM) instead of a discriminative model based on deep reinforcement learning. We define the Metropolis-Hastings (MH) naming game by generalizing previously proposed models. It is not a referential game with explicit feedback, as assumed by many emergent communication studies. Instead, it is a game based on joint attention without explicit feedback. Mathematically, the MH naming game is proved to be a type of MH algorithm for an integrative PGM that combines two agents that play the naming game. From this viewpoint, symbol emergence is regarded as decentralized Bayesian inference, and semiotic communication is regarded as inter-personal cross-modal inference. This notion leads to the collective predictive coding hypothesis regarding language evolution and, in general, the emergence of symbols. We also propose the inter-Gaussian mixture model (GMM)+ variational autoencoder (VAE), a deep generative model for emergent communication based on the MH naming game. In this model, two agents create internal representations and categories and share signs (i.e. names of objects) from raw visual images observed from different viewpoints. The model has been validated on MNIST and Fruits 360 datasets. Experimental findings demonstrate that categories are formed from real images observed by agents, and signs are correctly shared across agents by successfully utilizing both of the observations of agents via the MH naming game. Furthermore, scholars verified that visual images were recalled from signs uttered by agents. Notably, emergent communication without supervision and reward feedback improved the performance of the unsupervised representation learning of agents.

Probabilistic generative transformer language models for generative design of molecules

Self-supervised neural language models have recently found wide applications in the generative design of organic molecules and protein sequences as well as representation learning for downstream structure classification and functional prediction. However, most of the existing deep learning models for molecule design usually require a big dataset and have a black-box architecture, which makes it difficult to interpret their design logic. Here we propose the Generative Molecular Transformer (GMTransformer), a probabilistic neural network model for generative design of molecules. Our model is built on the blank filling language model originally developed for text processing, which has demonstrated unique advantages in learning the “molecules grammars” with high-quality generation, interpretability, and data efficiency. Benchmarked on the MOSES datasets, our models achieve high novelty and Scaf compared to other baselines. The probabilistic generation steps have the potential in tinkering with molecule design due to their capability of recommending how to modify existing molecules with explanation, guided by the learned implicit molecule chemistry. The source code and datasets can be accessed freely at https://github.com/usccolumbia/GMTransformer

Unsupervised phoneme and word acquisition from continuous speech based on a hierarchical probabilistic generative model

Humans can divide the perceived continuous speech signals, which exhibit double articulation structure, into phonemes and words without explicit boundary points or labels and thus learn a language. In constructive developmental studies, learning the double articulation structure of speech signals is important for realizing robots with human-like language learning abilities. In this study, we propose a novel probabilistic generative model called the Gaussian process-hidden semi-Markov model-based double articulation analyzer (GP-HSMM-DAA), which can learn phonemes and words from continuous speech signals by hierarchically connecting two probabilistic generative models (PGMs), namely, the Gaussian process-hidden semi-Markov model and hidden semi-Markov model. In the proposed model, the parameters of each PGM are mutually and complementarily updated and learned, enabling accurate learning of the phonemes and words. The experimental results reveal that GP-HSMM-DAA can segment continuous speech into phonemes and words with higher accuracy than the conventional method.