Probabilistic Generative Model Research Articles

Regularized Bayesian calibration and scoring of the WD-FAB IRT model improves predictive performance over marginal maximum likelihood.

Item response theory (IRT) is the statistical paradigm underlying a dominant family of generative probabilistic models for test responses, used to quantify traits in individuals relative to target populations. The graded response model (GRM) is a particular IRT model that is used for ordered polytomous test responses. Both the development and the application of the GRM and other IRT models require statistical decisions. For formulating these models (calibration), one needs to decide on methodologies for item selection, inference, and regularization. For applying these models (test scoring), one needs to make similar decisions, often prioritizing computational tractability and/or interpretability. In many applications, such as in the Work Disability Functional Assessment Battery (WD-FAB), tractability implies approximating an individual’s score distribution using estimates of mean and variance, and obtaining that score conditional on only point estimates of the calibrated model. In this manuscript, we evaluate the calibration and scoring of models under this common use-case using Bayesian cross-validation. Applied to the WD-FAB responses collected for the National Institutes of Health, we assess the predictive power of implementations of the GRM based on their ability to yield, on validation sets of respondents, ability estimates that are most predictive of patterns of item responses. Our main finding indicates that regularized Bayesian calibration of the GRM outperforms the regularization-free empirical Bayesian procedure of marginal maximum likelihood. We also motivate the use of compactly supported priors in test scoring.

Fault Diagnosis of Machines Using Deep Convolutional Beta-Variational Autoencoder

Industries are using fault diagnosis methods to prevent any downtime, which eventually led them to make profits and take necessary steps beforehand to avoid any mishaps. In recent years, deep learning methods have shown extraordinary performance in massive data applications with advancement in computing power. In this article, a novel intelligent fault diagnosis scheme based on deep convolutional variable-beta variational autoencoder (VAE) is proposed to extract discriminative features. A new min–max algorithm for data points reduction and a random sampling technique to get 2-D data has been proposed. The proposed fault diagnosis combines all intermediate steps (from preprocessing to classification) in a single framework, and an end-to-end training has been performed. The proposed training method with variable beta uses VAE as a feature extractor and classifier rather than just being a probabilistic generative model, which further improved the performance of the overall model. The proposed scheme reduces the needs of domain/expertise knowledge on time-series data. The proposed method has also been validated in the presence of noise. The proposed approach is validated through two case studies by utilizing rotating machinery datasets: First, on the case western reserve university vibration dataset (VD), and second, on the air compressor acoustic dataset (AD). Highest accuracies obtained are 99.93% and 99.91% on case western reserve university VD and air compressor AD, respectively, using the proposed scheme. Finally, a comparative study has been presented.

Get out of the BAG! Silos in AI Ethics Education: Unsupervised Topic Modeling Analysis of Global AI Curricula

The domain of Artificial Intelligence (AI) ethics is not new, with discussions going back at least 40 years. Teaching the principles and requirements of ethical AI to students is considered an essential part of this domain, with an increasing number of technical AI courses taught at several higher-education institutions around the globe including content related to ethics. By using Latent Dirichlet Allocation (LDA), a generative probabilistic topic model, this study uncovers topics in teaching ethics in AI courses and their trends related to where the courses are taught, by whom, and at what level of cognitive complexity and specificity according to Bloom’s taxonomy. In this exploratory study based on unsupervised machine learning, we analyzed a total of 166 courses: 116 from North American universities, 11 from Asia, 36 from Europe, and 10 from other regions. Based on this analysis, we were able to synthesize a model of teaching approaches, which we call BAG (Build, Assess, and Govern), that combines specific cognitive levels, course content topics, and disciplines affiliated with the department(s) in charge of the course. We critically assess the implications of this teaching paradigm and provide suggestions about how to move away from these practices. We challenge teaching practitioners and program coordinators to reflect on their usual procedures so that they may expand their methodology beyond the confines of stereotypical thought and traditional biases regarding what disciplines should teach and how.
 This article appears in the AI & Society track.

Non-volume preserving-based fusion to group-level emotion recognition on crowd videos

Group-level emotion recognition (ER) is a growing research area as the demands for assessing crowds of all sizes are becoming an interest in both the security arena as well as social media. This work extends the earlier ER investigations, which focused on either group-level ER on single images or within a video, by fully investigating group-level expression recognition on crowd videos. In this paper, we propose an effective deep feature level fusion mechanism to model the spatial-temporal information in the crowd videos. In our approach, the fusing process is performed on the deep feature domain by a generative probabilistic model, Non-Volume Preserving Fusion (NVPF), that models spatial information relationships. Furthermore, we extend our proposed spatial NVPF approach to the spatial-temporal NVPF approach to learn the temporal information between frames. To demonstrate the robustness and effectiveness of each component in the proposed approach, three experiments were conducted: (i) evaluation on AffectNet database to benchmark the proposed EmoNet for recognizing facial expression; (ii) evaluation on EmotiW2018 to benchmark the proposed deep feature level fusion mechanism NVPF; and, (iii) examine the proposed TNVPF on an innovative Group-level Emotion on Crowd Videos (GECV) dataset composed of 627 videos collected from publicly available sources. GECV dataset is a collection of videos containing crowds of people. Each video is labeled with emotion categories at three levels: individual faces, group of people, and the entire video frame.

A whole brain probabilistic generative model: Toward realizing cognitive architectures for developmental robots

Building a human-like integrative artificial cognitive system, that is, an artificial general intelligence (AGI), is the holy grail of the artificial intelligence (AI) field. Furthermore, a computational model that enables an artificial system to achieve cognitive development will be an excellent reference for brain and cognitive science. This paper describes an approach to develop a cognitive architecture by integrating elemental cognitive modules to enable the training of the modules as a whole. This approach is based on two ideas: (1) brain-inspired AI, learning human brain architecture to build human-level intelligence, and (2) a probabilistic generative model (PGM)-based cognitive architecture to develop a cognitive system for developmental robots by integrating PGMs. The proposed development framework is called a whole brain PGM (WB-PGM), which differs fundamentally from existing cognitive architectures in that it can learn continuously through a system based on sensory-motor information.In this paper, we describe the rationale for WB-PGM, the current status of PGM-based elemental cognitive modules, their relationship with the human brain, the approach to the integration of the cognitive modules, and future challenges. Our findings can serve as a reference for brain studies. As PGMs describe explicit informational relationships between variables, WB-PGM provides interpretable guidance from computational sciences to brain science. By providing such information, researchers in neuroscience can provide feedback to researchers in AI and robotics on what the current models lack with reference to the brain. Further, it can facilitate collaboration among researchers in neuro-cognitive sciences as well as AI and robotics.

Neural networks with upper and lower bound constraints and its application on industrial soft sensing modeling with missing values

Soft sensors estimate quality indicators that are difficult to measure online so that they are important in industrial processes. The sensors may malfunction so that some data may be unavailable or contain abnormal values, which means the data contain missing values. To deal with missing values, missing values are filled by evaluated estimations or utilized by probabilistic generative models. Since filling the missing values with expectations may overestimate the prediction error of the model, and using a generative model to calculate the generating probability may be harmful to prediction performance, we propose a new neural network method with the upper and lower bound constraints of the estimated missing values. Our methods will not overestimate the prediction error. From the probabilistic point of view, our optimization goal can be regarded as maximizing the posterior probability under the upper and lower bound constraints. Meanwhile, the mask-specific local models are adopted to improve the prediction reliability of the upper and lower bounds. Finally, our methods are verified in the industrial processes.

The cultural evolution of love in literary history

Since the late nineteenth century, cultural historians have noted that the importance of love increased during the Medieval and Early Modern European period (a phenomenon that was once referred to as the emergence of 'courtly love'). However, more recent works have shown a similar increase in Chinese, Arabic, Persian, Indian and Japanese cultures. Why such a convergent evolution in very different cultures? Using qualitative and quantitative approaches, we leverage literary history and build a database of ancient literary fiction for 19 geographical areas and 77 historical periods covering 3,800 years, from the Middle Bronze Age to the Early Modern period. We first confirm that romantic elements have increased in Eurasian literary fiction over the past millennium, and that similar increases also occurred earlier, in Ancient Greece, Rome and Classical India. We then explore the ecological determinants of this increase. Consistent with hypotheses from cultural history and behavioural ecology, we show that a higher level of economic development is strongly associated with a greater incidence of love in narrative fiction (our proxy for the importance of love in a culture). To further test the causal role of economic development, we used a difference-in-difference method that exploits exogenous regional variations in economic development resulting from the adoption of the heavy plough in medieval Europe. Finally, we used probabilistic generative models to reconstruct the latent evolution of love and to assess the respective role of cultural diffusion and economic development.

AIDA: An Active Inference-Based Design Agent for Audio Processing Algorithms

In this paper we present Active Inference-Based Design Agent (AIDA), which is an active inference-based agent that iteratively designs a personalized audio processing algorithm through situated interactions with a human client. The target application of AIDA is to propose on-the-spot the most interesting alternative values for the tuning parameters of a hearing aid (HA) algorithm, whenever a HA client is not satisfied with their HA performance. AIDA interprets searching for the “most interesting alternative” as an issue of optimal (acoustic) context-aware Bayesian trial design. In computational terms, AIDA is realized as an active inference-based agent with an Expected Free Energy criterion for trial design. This type of architecture is inspired by neuro-economic models on efficient (Bayesian) trial design in brains and implies that AIDA comprises generative probabilistic models for acoustic signals and user responses. We propose a novel generative model for acoustic signals as a sum of time-varying auto-regressive filters and a user response model based on a Gaussian Process Classifier. The full AIDA agent has been implemented in a factor graph for the generative model and all tasks (parameter learning, acoustic context classification, trial design, etc.) are realized by variational message passing on the factor graph. All verification and validation experiments and demonstrations are freely accessible at our GitHub repository.

Resilient Cyber-Physical Energy Systems Using Prior Information Based on Gaussian Process

The power grid infrastructure is a large-scale, heterogeneous, and complex cyber-physical system, which forms the lifeline of modern societies. The trend of tight coupling of physics, communication, and computation in cyber-physical energy systems (CPESs) is evident by the inclusion of numerous measurement sensors. This contributes to enhancing the monitoring and control functionalities of CPESs. At the same time, the occurrence of adverse effects constitutes a vital dimension of CPES operation. Increasing the resilience of critical energy systems is of key importance for safeguarding the national economy and security. This article considers the problem of optimal estimation with sensing measurements subject to arbitrary corruption resulting from adverse effects. Such signals can cause false situation awareness and/or trigger a sequence of cascading effects leading to an ultimate system failure. We formulate the problem as a constrained optimization with additional prior information posed as a set inclusion constraint on the measurement vector. It is shown that if the prior set satisfies certain conditions, the resulting recovery error bound is improved. The approach demonstrates enhancement of the CPES resiliency by using the Gaussian process as the basis of a prior generative probabilistic regression model using historical data. The validation of the resiliency mechanism using prior information is performed using the New York Independent System Operator grid data, demonstrating 100% successful state recovery for up to 60% of CPES sensor failures.

Reciprocity, community detection, and link prediction in dynamic networks

Many complex systems change their structure over time, in these cases dynamic networks can provide a richer representation of such phenomena. As a consequence, many inference methods have been generalized to the dynamic case with the aim to model dynamic interactions. Particular interest has been devoted to extend the stochastic block model and its variant, to capture community structure as the network changes in time. While these models assume that edge formation depends only on the community memberships, recent work for static networks show the importance to include additional parameters capturing structural properties, as reciprocity for instance. Remarkably, these models are capable of generating more realistic network representations than those that only consider community membership. To this aim, we present a probabilistic generative model with hidden variables that integrates reciprocity and communities as structural information of networks that evolve in time. The model assumes a fundamental order in observing reciprocal data, that is an edge is observed, conditional on its reciprocated edge in the past. We deploy a Markovian approach to construct the network’s transition matrix between time steps and parameters’ inference is performed with an expectation-maximization algorithm that leads to high computational efficiency because it exploits the sparsity of the dataset. We test the performance of the model on synthetic dynamical networks, as well as on real networks of citations and email datasets. We show that our model captures the reciprocity of real networks better than standard models with only community structure, while performing well at link prediction tasks.

Map completion from partial observation using the global structure of multiple environmental maps

Using the spatial structure of various indoor environments as prior knowledge, the robot would construct the map more efficiently. Autonomous mobile robots generally apply simultaneous localization and mapping (SLAM) methods to understand the reachable area in newly visited environments. However, conventional mapping approaches are limited by only considering sensor observation and control signals to estimate the current environment map. This paper proposes a novel SLAM method, map completion network-based SLAM (MCN-SLAM), based on a probabilistic generative model incorporating deep neural networks for map completion. These map completion networks are primarily trained in the framework of generative adversarial networks (GANs) to extract the global structure of large amounts of existing map data. We show in experiments that the proposed method can estimate the environment map 1.3 times better than the previous SLAM methods in the situation of partial observation.

Multiagent multimodal categorization for symbol emergence: emergent communication via interpersonal cross-modal inference

This paper describes a computational model of multiagent multimodal categorization that realizes emergent communication. We clarify whether the computational model can reproduce the following functions in a symbol emergence system, comprising two agents with different sensory modalities playing a naming game. (1) Function for forming a shared lexical system that comprises perceptual categories and corresponding signs, formed by agents through individual learning and semiotic communication. (2) Function to improve the categorization accuracy in an agent via semiotic communication with another agent, even when some sensory modalities of each agent are missing. (3) Function that an agent infers unobserved sensory information based on a sign sampled from another agent in the same manner as cross-modal inference. We propose an interpersonal multimodal Dirichlet mixture (Inter-MDM), which is derived by dividing an integrative probabilistic generative model, which is obtained by integrating two Dirichlet mixtures (DMs). The Markov chain Monte Carlo algorithm realizes emergent communication. The experimental results demonstrated that Inter-MDM enables agents to form multimodal categories and appropriately share signs between agents. It is shown that emergent communication improves categorization accuracy, even when some sensory modalities are missing. Inter-MDM enables an agent to predict unobserved information based on a shared sign.

A 30 year topic analysis of Veterinary Medicine literature

In the present study Latent Dirichlet allocation (LDA) was used as a generative probabilistic model to extract major topics in interdecadalresearch for the Veterinary Medicine scientific literature. A total of 22 topics were extracted during the 1991-2000 period, 23 topics during 2001-2010 and 60 topics during 2011-2020. Three different algorithms were used to validate the model: perplexity, silhouette clustering and gradient boosted trees. All three validation metrics showed that LDA performed well in extracting topics. Each decade was characterized by unique topics as well as common topics which existed throughout periods. The most frequent topics were identified and trends were quantified with the use of indexes. A list of the 30 most frequent and most associated with the term Veterinary Medicine words is provided. A shift in scientific thinking probably occurred during the 30-year-period in the process of incorporating the fields related to Veterinary students, antimicrobial resistance and animals’ behavior.

Investigating the Efficient Use of Word Embedding with Neural-Topic Models for Interpretable Topics from Short Texts.

With the rapid proliferation of social networking sites (SNS), automatic topic extraction from various text messages posted on SNS are becoming an important source of information for understanding current social trends or needs. Latent Dirichlet Allocation (LDA), a probabilistic generative model, is one of the popular topic models in the area of Natural Language Processing (NLP) and has been widely used in information retrieval, topic extraction, and document analysis. Unlike long texts from formal documents, messages on SNS are generally short. Traditional topic models such as LDA or pLSA (probabilistic latent semantic analysis) suffer performance degradation for short-text analysis due to a lack of word co-occurrence information in each short text. To cope with this problem, various techniques are evolving for interpretable topic modeling for short texts, pretrained word embedding with an external corpus combined with topic models is one of them. Due to recent developments of deep neural networks (DNN) and deep generative models, neural-topic models (NTM) are emerging to achieve flexibility and high performance in topic modeling. However, there are very few research works on neural-topic models with pretrained word embedding for generating high-quality topics from short texts. In this work, in addition to pretrained word embedding, a fine-tuning stage with an original corpus is proposed for training neural-topic models in order to generate semantically coherent, corpus-specific topics. An extensive study with eight neural-topic models has been completed to check the effectiveness of additional fine-tuning and pretrained word embedding in generating interpretable topics by simulation experiments with several benchmark datasets. The extracted topics are evaluated by different metrics of topic coherence and topic diversity. We have also studied the performance of the models in classification and clustering tasks. Our study concludes that though auxiliary word embedding with a large external corpus improves the topic coherency of short texts, an additional fine-tuning stage is needed for generating more corpus-specific topics from short-text data.

Machine learning methods for generating high dimensional discrete datasets

AbstractThe development of platforms and techniques for emerging Big Data and Machine Learning applications requires the availability of real‐life datasets. A possible solution is to synthesize datasets that reflect patterns of real ones using a two‐step approach: first, a real dataset is analyzed to derive relevant patterns and, then, to use such patterns for reconstructing a new dataset that preserves the main characteristics of . This survey explores two possible approaches: (1) Constraint‐based generation and (2) probabilistic generative modeling. The former is devised using inverse mining () techniques, and consists of generating a dataset satisfying given support constraints on the itemsets of an input set, that are typically the frequent ones. By contrast, for the latter approach, recent developments in probabilistic generative modeling () are explored that model the generation as a sampling process from a parametric distribution, typically encoded as neural network. The two approaches are compared by providing an overview of their instantiations for the case of discrete data and discussing their pros and cons.This article is categorized under: Fundamental Concepts of Data and Knowledge > Big Data Mining Technologies > Machine Learning Algorithmic Development > Structure Discovery

Modeling and Detecting Communities in Node Attributed Networks

As a fundamental structure in real-world networks, in addition to graph topology, communities can also be reflected by abundant node attributes. In attributed community detection, probabilistic generative models (PGMs) have become the mainstream method due to their principled characterization and competitive performances. Here, we propose a novel PGM without imposing any distributional assumptions on attributes, which is superior to the existing PGMs that require attributes to be categorical or Gaussian distributed. Based on the block model of graph structure, our model incorporates the attribute by describing its effect on node popularity. To characterize the effect quantitatively, we analyze the community detectability for our model and then establish the requirements of the node popularity term. This leads to a new scheme for the crucial model selection problem in choosing and solving attributed community detection models. With the model determined, an efficient algorithm is developed to estimate the parameters and to infer the communities. The proposed method is validated from two aspects. First, the effectiveness of our algorithm is theoretically guaranteed by the detectability condition. Second, extensive experiments indicate that our method not only outperforms the competing approaches on the employed datasets, but also shows better applicability to networks with various node attributes.

Probabilistic Generative Model for Hyperspectral Unmixing Accounting for Endmember Variability

The complex nature of hyperspectral images makes the analysis of spectral signatures a challenging task in remote sensing. For quantitative analysis, spectral unmixing is a well-established and effective tool to analyze the spectra and spatial distribution of substances in the scene. The classical unmixing algorithms usually fail to tackle spectral variability caused by variations in environmental conditions. Many variants based on the linear mixing process have been proposed to tackle this problem; however, the spectral variability modeling capacity of these algorithms is usually insufficient. In this article, we present a probabilistic generative model to address endmember variability and provide more accurate abundance and endmember estimates. The proposed model simultaneously extracts the endmembers and estimates abundances in an unsupervised manner. In particular, it allows fitting arbitrary endmember distributions through the nonlinear modeling capability of neural networks compared to other methods that use parametric endmember variability models. The performance of the proposed approach is evaluated on both synthetic and real datasets. Experimental results show its superiority in comparison with other state-of-the-art methods. The code of this work is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/shuaikaishi/PGMSU</uri> for the sake of reproducibility.

Supervising the Decoder of Variational Autoencoders to Improve Scientific Utility.

Probabilistic generative models are attractive for scientific modeling because their inferred parameters can be used to generate hypotheses and design experiments. This requires that the learned model provides an accurate representation of the input data and yields a latent space that effectively predicts outcomes relevant to the scientific question. Supervised Variational Autoencoders (SVAEs) have previously been used for this purpose, as a carefully designed decoder can be used as an interpretable generative model of the data, while the supervised objective ensures a predictive latent representation. Unfortunately, the supervised objective forces the encoder to learn a biased approximation to the generative posterior distribution, which renders the generative parameters unreliable when used in scientific models. This issue has remained undetected as reconstruction losses commonly used to evaluate model performance do not detect bias in the encoder. We address this previously-unreported issue by developing a second-order supervision framework (SOS-VAE) that updates the decoder parameters, rather than the encoder, to induce a predictive latent representation. This ensures that the encoder maintains a reliable posterior approximation and the decoder parameters can be effectively interpreted. We extend this technique to allow the user to trade-off the bias in the generative parameters for improved predictive performance, acting as an intermediate option between SVAEs and our new SOS-VAE. We also use this methodology to address missing data issues that often arise when combining recordings from multiple scientific experiments. We demonstrate the effectiveness of these developments using synthetic data and electrophysiological recordings with an emphasis on how our learned representations can be used to design scientific experiments.

Dynamic Self-Supervised Teacher-Student Network Learning.

Lifelong learning (LLL) represents the ability of an artificial intelligence system to learn successively a sequence of different databases. In this paper we introduce the Dynamic Self-Supervised Teacher-Student Network (D-TS), representing a more general LLL framework, where the Teacher is implemented as a dynamically expanding mixture model which automatically increases its capacity to deal with a growing number of tasks. We propose the Knowledge Discrepancy Score (KDS) criterion for measuring the relevance of the incoming information characterizing a new task when compared to the existing knowledge accumulated by the Teacher module from its previous training. The KDS ensures a light Teacher architecture while also enabling to reuse the learned knowledge whenever appropriate, accelerating the learning of given tasks. The Student module is implemented as a lightweight probabilistic generative model. We introduce a novel self-supervised learning procedure for the Student that allows to capture cross-domain latent representations from the entire knowledge accumulated by the Teacher as well as from novel data. We perform several experiments which show that D-TS can achieve the state of the art results in LLL while requiring fewer parameters than other methods.

Generative Text Convolutional Neural Network for Hierarchical Document Representation Learning.

For document analysis, existing methods often resort to the document representation that either discards the word order information or projects each word into a low-dimensional dense embedding vector. However, confined by the data's sparsity and high-dimensionality, limited effort has been made to explore the semantic structures underlying the document representation that formulates each document as a sequence of one-hot vectors, especially in the probabilistic modeling literature. To construct a probabilistic generative model for this type of document representation, we first develop convolutional Poisson factor analysis (CPFA) that not only utilizes the sparse property of data but also enables model parallelism. Through interleaving probabilistic Dirichlet-gamma pooling layers with learnable parameters, we extend the shallow CPFA into a generative text convolutional neural network (GTCNN), which captures richer semantic information with multiple probabilistic convolutional layers and can be coupled with existing deep topic models to alleviate their loss of word order. For efficient and scalable model inference, we not only develop both a parallel upward-downward Gibbs sampler and SG-MCMC based algorithm for training GTCNN, but also construct a hierarchical Weibull convolutional inference network for fast out-of-sample prediction. Experimental results on document representation learning tasks demonstrate the effectiveness of the proposed methods.