Deep Probabilistic Models Research Articles

Compositional Deep Probabilistic Models of DNA-Encoded Libraries.

DNA-encoded library (DEL) has proven to be a powerful tool that utilizes combinatorially constructed small molecules to facilitate highly efficient screening experiments. These selection experiments, involving multiple stages of washing, elution, and identification of potent binders via unique DNA barcodes, often generate complex data. This complexity can potentially mask the underlying signals, necessitating the application of computational tools, such as machine learning, to uncover valuable insights. We introduce a compositional deep probabilistic model of DEL data, DEL-Compose, which decomposes molecular representations into their monosynthon, disynthon, and trisynthon building blocks and capitalizes on the inherent hierarchical structure of these molecules by modeling latent reactions between embedded synthons. Additionally, we investigate methods to improve the observation models for DEL count data, such as integrating covariate factors to more effectively account for data noise. Across two popular public benchmark data sets (CA-IX and HRP), our model demonstrates strong performance compared to count baselines, enriches the correct pharmacophores, and offers valuable insights via its intrinsic interpretable structure, thereby providing a robust tool for the analysis of DEL data.

ZEBRA: a hierarchically integrated gene expression atlas of the murine and human brain at single-cell resolution.

The molecular causes and mechanisms of neurodegenerative diseases remain poorly understood. A growing number of single-cell studies have implicated various neural, glial, and immune cell subtypes to affect the mammalian central nervous system in many age-related disorders. Integrating this body of transcriptomic evidence into a comprehensive and reproducible framework poses several computational challenges. Here, we introduce ZEBRA, a large single-cell and single-nucleus RNA-seq database. ZEBRA integrates and normalizes gene expression and metadata from 33 studies, encompassing 4.2 million human and mouse brain cells sampled from 39 brain regions. It incorporates samples from patients with neurodegenerative diseases like Alzheimer's disease, Parkinson's disease, and Multiple sclerosis, as well as samples from relevant mouse models. We employed scVI, a deep probabilistic auto-encoder model, to integrate the samples and curated both cell and sample metadata for downstream analysis. ZEBRA allows for cell-type and disease-specific markers to be explored and compared between sample conditions and brain regions, a cell composition analysis, and gene-wise feature mappings. Our comprehensive molecular database facilitates the generation of data-driven hypotheses, enhancing our understanding of mammalian brain function during aging and disease. The data sets, along with an interactive database are freely available at https://www.ccb.uni-saarland.de/zebra.

The graph embedded topic model

Most of existing graph neural networks (GNNs) developed for the prevalent text-rich networks typically treat texts as node attributes. This kind of approach unavoidably results in the loss of important semantic structures and restricts the representational power of GNNs. In this work, we introduce a document similarity-based graph convolutional network (DS-GCN) encoder to combine graph topologies and document semantics for text-rich network representation. Then, a graph decoder, based on the latent position model, is used to reconstruct the graph while preserving the network topology. The document matrix is rebuilt by a document decoder, based on the embedded topic model, which considers both topic and word embeddings. By including a cluster membership variable for each node in the network, we thus develop an end-to-end clustering technique relying on a new deep probabilistic model called the graph embedded topic model (GETM). Numerical experiments on three simulated scenarios emphasize the ability of GETM in fusing the graph topology structure and the document embeddings, and highlight its node clustering performance. Moreover, an application on the Cora-enrich citation network is conducted to demonstrate the effectiveness and interest of GETM in practice.

Handling Ill-conditioned Omics Data with Deep Probabilistic Models.

The advent of high-throughput technologies has produced an increase in the dimensionality of omics datasets, which limits the application of machine learning methods due to the great unbalance between the number of observations and features. In this scenario, dimensionality reduction is essential to extract the relevant information within these datasets and project it in a low-dimensional space, and probabilistic latent space models are becoming popular given their capability to capture the underlying structure of the data as well as the uncertainty in the information. This article aims to provide a general classification and dimensionality reduction method based on deep latent space models that tackles two of the main problems that arise in omics datasets: the presence of missing data and the limited number of observations against the number of features. We propose a semi-supervised Bayesian latent space model that infers a low-dimensional embedding driven by the target label: the Deep Bayesian Logistic Regression (DBLR) model. During inference, the model also learns a global vector of weights that allows it to make predictions given the low-dimensional embedding of the observations. Since this kind of dataset is prone to overfitting, we introduce an additional probabilistic regularization method based on the semi-supervised nature of the model. We compared the performance of the DBLR against several state-of-the-art methods for dimensionality reduction, both in synthetic and real datasets with different data types. The proposed model provides more informative low-dimensional representations, outperforms the baseline methods in classification, and can naturally handle missing entries.

Deep probabilistic graphical modeling for robust multivariate time series anomaly detection with missing data

Multivariate time series anomaly detection with missing data is one of the most pending issues for industrial monitoring. Due to scarcity of labeled anomalies, most advanced data-driven anomaly detection approaches fall in the unsupervised learning paradigm. As a premise in the presence of missing data, one needs to improve the data quality through data imputation with a separate model. Our concern lies in the consistency between data imputation and unsupervised learning for robust anomaly detection, regarding accurately discovering the spatiotemporal dependence among multiple variables over time. However, the existing practice tends to overlook this consistency and decouple the training process for these two closely linked tasks. This article novelly proposes a probabilistic multivariate time series anomaly detection framework that unifies data imputation and unsupervised learning. A deep probabilistic graphical model abbreviated SCNF is first devised for unsupervised density estimation. A tailored expectation maximization-based optimization scheme is then developed to achieve the joint training of data imputation and unsupervised learning with missing data. The efficacy is experimentally corroborated in several industrial applications, including chemical process, water treatment and network traffic. Briefly, the joint training framework enhances the AUROC of SCNF by averagely 6.34% for three applications under 50% data missing rate.

Interpretable fake news detection with topic and deep variational models

The growing societal dependence on social media and user generated content for news and information has increased the influence of unreliable sources and fake content, which muddles public discourse and lessens trust in the media. Validating the credibility of such information is a difficult task that is susceptible to confirmation bias, leading to the development of algorithmic techniques to distinguish between fake and real news. However, most existing methods are challenging to interpret, making it difficult to establish trust in predictions, and make assumptions that are unrealistic in many real-world scenarios, e.g., the availability of audiovisual features or provenance. In this work, we focus on fake news detection of textual content using interpretable features and methods. In particular, we have developed a deep probabilistic model that integrates a dense representation of textual news using a variational autoencoder and bi-directional Long Short-Term Memory (LSTM) networks with semantic topic-related features inferred from a Bayesian admixture model. Extensive experimental studies with 3 real-world datasets demonstrate that our model achieves comparable performance to state-of-the-art competing models while facilitating model interpretability from the learned topics. Finally, we have conducted model ablation studies to justify the effectiveness and accuracy of integrating neural embeddings and topic features both quantitatively by evaluating performance and qualitatively through separability in lower dimensional embeddings.

Max-Margin Deep Diverse Latent Dirichlet Allocation With Continual Learning.

Deep probabilistic aspect models are widely utilized in document analysis to extract the semantic information and obtain descriptive topics. However, there are two problems that may affect their applications. One is that common words shared among all documents with low representational meaning may reduce the representation ability of learned topics. The other is introducing supervision information to hierarchical topic models to fully utilize the side information of documents that is difficult. To address these problems, in this article, we first propose deep diverse latent Dirichlet allocation (DDLDA), a deep hierarchical topic model that can yield more meaningful semantic topics with less common and meaningless words by introducing shared topics. Moreover, we develop a variational inference network for DDLDA, which helps us to further generalize DDLDA to a supervised deep topic model called max-margin DDLDA (mmDDLDA) by employing max-margin principle as the classification criterion. Compared to DDLDA, mmDDLDA can discover more discriminative topical representations. In addition, a continual hybrid method with stochastic-gradient MCMC and variational inference is put forward for deep latent Dirichlet allocation (DLDA)-based models to make them more practical in real-world applications. The experimental results demonstrate that DDLDA and mmDDLDA are more efficient than existing unsupervised and supervised topic models in discovering highly discriminative topic representations and achieving higher classification accuracy. Meanwhile, DLDA and our proposed models trained by the proposed continual learning approach cannot only show good performance on preventing catastrophic forgetting but also fit the evolving new tasks well.

Interpretable Deep Probabilistic Model for HRR Radar Signal and its Application to Target Recognition

With the advent of neural network, unprecedented advancements have been achieved in many tasks, including radar signal processing. However, a key disadvantage of the current methods is the black-box structure, which makes it hard to interpret or assess the hidden representations of data. In this paper, by combining the physical generative mechanism of the high range resolution (HRR) radar signal with neural network, we develop an interpretable deep probabilistic model to learn the latent features from HRR radar signals that can characterize the physical structure of targets. In detail, based on the radar target's scattering center model which describes the HRR radar signal as the summation of echoes from the scattering centers, a deep probabilistic model is constructed to depict the generative process from the scattering centers to observations, where the latent features comprise the locations and amplitudes of scattering centers. Considering that the locations of scattering centers are nearly invariable and the amplitudes of them are fluctuant for the signals within a small angular range, our model defines the shared locations and independent amplitudes of scattering centers for the signals in a small angular range, aiming to improve the robustness of interpretable feature extraction model. In addition, our model performs probabilistic inference on the posterior distribution of latent features with high preciseness. With the proposed model, we further design a recognition scheme based on the minimum reconstruction error criterion. Experiments on the measured HRR radar dataset validate the effectiveness of our model on learning interpretable features and superior recognition performance.

Deep probabilistic time series forecasting using augmented recurrent input for dynamic systems

The demand of probabilistic time series forecasting has been recently raised in various dynamic system scenarios, for example, system identification and prognostic and health management of machines. To this end, we combine the advances in both deep generative models and state space model (SSM) to come up with a novel, data-driven deep probabilistic sequence model. Specifically, we follow the popular encoder–decoder generative structure to build the recurrent neural networks (RNN) assisted variational sequence model on an augmented recurrent input space, which could induce rich stochastic sequence dependency. Besides, in order to alleviate the inconsistency issue of the posterior between training and predicting as well as improving the mining of dynamic patterns, we (i) propose using a lagged hybrid output as input for the posterior at next time step, which brings training and predicting into alignment; and (ii) further devise a generalized auto-regressive strategy that encodes all the historical dependencies for the posterior. Thereafter, we first investigate the methodological characteristics of the proposed deep probabilistic sequence model on toy cases, and then comprehensively demonstrate the superiority of our model against existing deep probabilistic SSM models through extensive numerical experiments on eight system identification benchmarks from various dynamic systems. Finally, we apply our sequence model to a real-world centrifugal compressor forecasting problem, and again verify its outstanding performance by quantifying the time series predictive distribution.

Deep Probabilistic Learning Model for Prediction of Ionic Liquids Toxicity.

Identification of ionic liquids with low toxicity is paramount for applications in various domains. Traditional approaches used for determining the toxicity of ionic liquids are often expensive, and can be labor intensive and time consuming. In order to mitigate these limitations, researchers have resorted to using computational models. This work presents a probabilistic model built from deep kernel learning with the aim of predicting the toxicity of ionic liquids in the leukemia rat cell line (IPC-81). Only open source tools, namely, RDKit and Mol2vec, are required to generate predictors for this model; as such, its predictions are solely based on chemical structure of the ionic liquids and no manual extraction of features is needed. The model recorded an RMSE of 0.228 and of 0.943. These results indicate that the model is both reliable and accurate. Furthermore, this model provides an accompanying uncertainty level for every prediction it makes. This is important because discrepancies in experimental measurements that generated the dataset used herein are inevitable, and ought to be modeled. A user-friendly web server was developed as well, enabling researchers and practitioners ti make predictions using this model.

Nessy: A Neuro-Symbolic System for Label Noise Reduction

Noisy labels represent one of the key issues in supervised machine learning. Existing work for label noise reduction mainly takes a probabilistic approach that infers true labels from data distributions in low-level feature spaces. Such an approach is not only limited by its capability to learn high-quality data representations, but also by the low predictive power of data distributions in inferring true classes. To address those problems, we introduce Nessy, a neuro-symbolic system that integrates deep probabilistic modeling and symbolic knowledge for label noise reduction. Our deep probabilistic model infers the true classes of data instances with noisy labels by exploiting data distributions in an underlying latent feature representation space. For data instances where inference is not reliable enough, Nessy extracts symbolic rules and ranks them according to several utility metrics. Top-ranking rules are injected into the deep probabilistic model via expectation regularization, i.e., via a posterior regularization term constraining the class distribution in the objective function. In a real deployment over multiple relation extraction tasks, we demonstrate that Nessy is able to significantly improve the state of the art, by 7% accuracy and 10.7% AUC on average.

A Deep Probabilistic Sensing and Learning Model for Brain Tumor Classification With Fusion-Net and HFCMIK Segmentation.

Goal: Implementation of an artificial intelli gence-based medical diagnosis tool for brain tumor classification, which is called the BTFSC-Net. Methods: Medical images are preprocessed using a hybrid probabilistic wiener filter (HPWF) The deep learning convolutional neural network (DLCNN) was utilized to fuse MRI and CT images with robust edge analysis (REA) properties, which are used to identify the slopes and edges of source images. Then, hybrid fuzzy c-means integrated k-means (HFCMIK) clustering is used to segment the disease affected region from the fused image. Further, hybrid features such as texture, colour, and low-level features are extracted from the fused image by using gray-level cooccurrence matrix (GLCM), redundant discrete wavelet transform (RDWT) descriptors. Finally, a deep learning based probabilistic neural network (DLPNN) is used to classify malignant and benign tumors. The BTFSC-Net attained 99.21% of segmentation accuracy and 99.46% of classification accuracy. Conclusions: The simulations showed that BTFSC-Net outperformed as compared to existing methods.

A probabilistic deep motion model for unsupervised cardiac shape anomaly assessment.

Automatic shape anomaly detection in large-scale imaging data can be useful for screening suboptimal segmentations and pathologies altering the cardiac morphology without intensive manual labour. We propose a deep probabilistic model for local anomaly detection in sequences of heart shapes, modelled as point sets, in a cardiac cycle. A deep recurrent encoder-decoder network captures the spatio-temporal dependencies to predict the next shape in the cycle and thus derive the outlier points that are attributed to excessive deviations from the network prediction. A predictive mixture distribution models the inlier and outlier classes via Gaussian and uniform distributions, respectively. A Gibbs sampling Expectation-Maximisation (EM) algorithm computes soft anomaly scores of the points via the posterior probabilities of each class in the E-step and estimates the parameters of the network and the predictive distribution in the M-step. We demonstrate the versatility of the method using two shape datasets derived from: (i) one million biventricular CMR images from 20,000 participants in the UK Biobank (UKB), and (ii) routine diagnostic imaging from Multi-Centre, Multi-Vendor, and Multi-Disease Cardiac Image (M&Ms). Experiments show that the detected shape anomalies in the UKB dataset are mostly associated with poor segmentation quality, and the predicted shape sequences show significant improvement over the input sequences. Furthermore, evaluations on U-Net based shapes from the M&Ms dataset reveals that the anomalies are attributable to the underlying pathologies that affect the ventricles. The proposed model can therefore be used as an effective mechanism to sift shape anomalies in large-scale cardiac imaging pipelines for further analysis.

Conditional sum-product networks: Modular probabilistic circuits via gate functions

While probabilistic graphical models are a central tool for reasoning under uncertainty in AI, they are in general not as expressive as deep neural models, and inference is notoriously hard and slow. In contrast, deep probabilistic models such as sum-product networks (SPNs) capture joint distributions and ensure tractable inference, but still lack the expressive power of intractable models based on deep neural networks. In this paper, we introduce conditional SPNs (CSPNs)—conditional density estimators for multivariate and potentially hybrid domains—and develop a structure-learning approach that derives both the structure and parameters of CSPNs from data. To harness the expressive power of deep neural networks (DNNs), we also show how to realize CSPNs by conditioning the parameters of vanilla SPNs on the input using DNNs as gate functions. In contrast to SPNs whose high-level structure can not be explicitly manipulated, CSPNs can naturally be used as tractable building blocks of deep probabilistic models whose modular structure maintains high-level interpretability. In experiments, we demonstrate that CSPNs are competitive with other probabilistic models and yield superior performance on structured prediction, conditional density estimation, auto-regressive image modeling, and multilabel image classification. In particular, we show that employing CSPNs as encoders and decoders within variational autoencoders can help to relax the commonly used mean field assumption and in turn improve performance.

Deep Bayesian baseline for segmenting diabetic retinopathy lesions: Advances and challenges.

Early diagnosis of retinopathy is essential for preventing retinal complications and visual impairment due to diabetes. For the detection of retinopathy lesions from retinal images, several automatic approaches based on deep neural networks have been developed in the recent years. Most of the proposed methods produce point estimates of pixels belonging to the lesion areas and give no or little information on the uncertainty of method predictions. However, the latter can be essential in the examination of the medical condition of the patient when the goal is early detection of abnormalities. This work extends the recent research with a Bayesian framework by considering the parameters of a convolutional neural network as random variables and utilizing stochastic variational dropout based approximation for uncertainty quantification. The framework includes an extended validation procedure and it allows analyzing lesion segmentation distributions, model calibration and prediction uncertainties. Also the challenges related to the deep probabilistic model and uncertainty quantification are presented. The proposed method achieves area under precision-recall curve of 0.84 for hard exudates, 0.641 for soft exudates, 0.593 for haemorrhages, and 0.484 for microaneurysms on IDRiD dataset.

Bidirectional recurrent gamma belief network for HRRP target recognition

To characterize the bidirectional temporal dependence across the range cell of high resolution range profile (HRRP), we propose a bidirectional deep Poisson gamma dynamical system (bi-DPGDS) to extract features for target recognition. The proposed bi-DPGDS is a bidirectional dynamical deep probabilistic generative model, which builds temporal deep structure with a hierarchy of gamma distributions. For scalable training and fast out-of-sample prediction, we generalize bi-DPGDS to bidirectional recurrent gamma belief network (bi-rGBN) by incorporating a bidirectional recurrent variational inference network, which incorporates the temporal correlations from both directions into latent representation, and introduce a hybrid Bayesian inference scheme combining stochastic gradient Markov chain Monte Carlo (SG-MCMC) and amoritized variational inference. Moreover, we further propose an attention bi-rGBN (attn-bi-rGBN) for supervised learning. Experimental results on measured HRRP data demonstrate the effectiveness and efficiency of our model in classification and generalization, and its robustness to HRRP shift and data size.

Supervised and semi-supervised deep probabilistic models for indoor positioning problems

WiFi fingerprint-based indoor localization has been a popular research topic recently. In this work, we propose two novel deep learning-based models, the convolutional mixture density recurrent neural network and the variational autoencoder-based semi-supervised learning model. The convolutional mixture density recurrent neural network is designed for indoor next location prediction, in which the advantages of convolutional neural networks, recurrent neural networks and mixture density networks are combined. Furthermore, since most of real-world WiFi fingerprint data are not labeled, we devise a variational autoencoder-based model to compute accurate user location in a semi-supervised learning manner. Finally, in order to evaluate the proposed models, we conduct the validation experiments on two real-world datasets. The final results are compared to other existing methods and verify the effectiveness of our approaches.

Efficient algorithms for robustness analysis of maximum a posteriori inference in selective sum-product networks

Sum-Product Networks (SPN) are deep probabilistic models with demonstrated excellent performance in several machine learning tasks. As with many other probabilistic models, performing Maximum-A-Posteriori inference in SPNs is NP-hard. Selective SPNs are a subclass of SPNs that allow for efficient Maximum-A-Posteriori inference and closed-form parameter learning. Due to the high number of parameters, SPNs learned from data can produce unreliable and overconfident inferences, especially for instances with low statistical support. This issue can be partially mitigated by performing a robustness analysis of inferences with respect to small changes in the parameters. In this work, we address the problem of assessing the robustness of Maximum-A-Posteriori inferences produced with Selective SPNs to global perturbations of the parameters. We consider such an inference robust if it remains the single maximizer under small perturbations of the model parameters. We present efficient algorithms and an empirical analysis with realistic problems involving missing data completion and multilabel classification. The experiments show that our criteria are informative with respect to the inference accuracy, suggesting that it indeed discriminate robust and non-robust instances.

Toward Discriminating and Synthesizing Motion Traces Using Deep Probabilistic Generative Models.

Mining knowledge from human mobility, such as discriminating motion traces left by different anonymous users, also known as the trajectory-user linking (TUL) problem, is an important task in many applications requiring location-based services (LBSs). However, it inevitably raises an issue that may be aggravated by TUL, i.e., how to defend against location attacks (e.g., deanonymization and location recovery). In this work, we present a Semisupervised Trajectory- User Linking model with Interpretable representation and Gaussian mixture prior (STULIG)-a novel deep probabilistic framework for jointly learning disentangled representation of user trajectories in a semisupervised manner and tackling the location recovery problem. STULIG characterizes multiple latent aspects of human trajectories and their labels into separate latent variables, which can be then used to interpret user check-in styles and improve the performance of trace classification. It can also generate synthetic yet plausible trajectories, thus protecting users' actual locations while preserving the meaningful mobility information for various machine learning tasks. We analyze and evaluate STULIG's ability to learn disentangled representations, discriminating human traces and generating realistic motions on several real-world mobility data sets. As demonstrated by extensive experimental evaluations, in addition to outperforming the state-of-the-art methods, our method provides intuitive explanations of the classification and generation and sheds lights on the interpretable mobility mining.

Calibration of deep probabilistic models with decoupled bayesian neural networks

Deep Neural Networks (DNNs) have achieved state-of-the-art accuracy performance in many tasks. However, recent works have pointed out that the outputs provided by these models are not well-calibrated, seriously limiting their use in critical decision scenarios. In this work, we propose to use a decoupled Bayesian stage, implemented with a Bayesian Neural Network (BNN), to map the uncalibrated probabilities provided by a DNN to calibrated ones, consistently improving calibration. Our results evidence that incorporating uncertainty provides more reliable probabilistic models, a critical condition for achieving good calibration. We report a generous collection of experimental results using high-accuracy DNNs in standardized image classification benchmarks, showing the good performance, flexibility and robust behaviour of our approach with respect to several state-of-the-art calibration methods. Code for reproducibility is provided.