Nonlinear Generative Model Research Articles

Bayesian Inference With Nonlinear Generative Models: Comments on Secure Learning

Bayesian Inference With Nonlinear Generative Models: Comments on Secure Learning

Gaussian Process Koopman Mode Decomposition.

We propose a nonlinear probabilistic generative model of Koopman mode decomposition based on an unsupervised gaussian process. Existing data-driven methods for Koopman mode decomposition have focused on estimating the quantities specified by Koopman mode decomposition: eigenvalues, eigenfunctions, and modes. Our model enables the simultaneous estimation of these quantities and latent variables governed by an unknown dynamical system. Furthermore, we introduce an efficient strategy to estimate the parameters of our model by low-rank approximations of covariance matrices. Applying the proposed model to both synthetic data and a real-world epidemiological data set, we show that various analyses are available using the estimated parameters.

Joint Separation and Localization of Moving Sound Sources Based on Neural Full-Rank Spatial Covariance Analysis

This paper presents an unsupervised multichannel method that can separate moving sound sources based on an amortized variational inference (AVI) of joint separation and localization. A recently proposed blind source separation (BSS) method called neural full-rank spatial covariance analysis (FCA) trains a neural separation model based on a nonlinear generative model of multichannel mixtures and can precisely separate unseen mixture signals. This method, however, assumes that the sound sources hardly move, and thus its performance is easily degraded by the source movements. In this paper, we solve this problem by introducing time-varying spatial covariance matrices and directions of arrival of sources into the nonlinear generative model of the neural FCA. This generative model is used for training a neural network to jointly separate and localize moving sources by using only multichannel mixture signals and array geometries. The training objective is derived as a lower bound on the log-marginal posterior probability in the framework of AVI. Experimental results obtained with mixture signals of moving sources show that our method outperformed an existing joint separation and localization method and standard BSS methods.

Predicting Lung Cancers Using Epidemiological Data: A Generative-Discriminative Framework

Predictive models for assessing the risk of developing lung cancers can help identify high-risk individuals with the aim of recommending further screening and early intervention. To facilitate pre-hospital self-assessments, some studies have exploited predictive models trained on non-clinical data (e.g., smoking status and family history). The performance of these models is limited due to not considering clinical data (e.g., blood test and medical imaging results). Deep learning has shown the potential in processing complex data that combine both clinical and non-clinical information. However, predicting lung cancers remains difficult due to the severe lack of positive samples among follow-ups. To tackle this problem, this paper presents a generative-discriminative framework for improving the ability of deep learning models to generalize. According to the proposed framework, two nonlinear generative models, one based on the generative adversarial network and another on the variational autoencoder, are used to synthesize auxiliary positive samples for the training set. Then, several discriminative models, including a deep neural network (DNN), are used to assess the lung cancer risk based on a comprehensive list of risk factors. The framework was evaluated on over 55 000 subjects questioned between January 2014 and December 2017, with 699 subjects being clinically diagnosed with lung cancer between January 2014 and August 2019. According to the results, the best performing predictive model built using the proposed framework was based on DNN. It achieved an average sensitivity of 76.54% and an area under the curve of 69.24% in distinguishing between the cases of lung cancer and normal cases on test sets.

Neural Full-Rank Spatial Covariance Analysis for Blind Source Separation

This paper describes aneural blind source separation (BSS) method based on amortized variational inference (AVI) of a non-linear generative model of mixture signals. A classical statistical approach to BSS is to fit a linear generative model that consists of spatial and source models representing the inter-channel covariances and power spectral densities of sources, respectively. Although the variational autoencoder (VAE) has successfully been used as a non-linear source model with latent features, it should be pretrained from a sufficient amount of isolated signals. Our method, in contrast, enables the VAE-based source model to be trained only from mixture signals. Specifically, we introduce a neural mixture-to-feature inference model that directly infers the latent features from the observed mixture and integrate it with a neural feature-to-mixture generative model consisting of a full-rank spatial model and a VAE-based source model. All the models are optimized jointly such that the likelihood for the training mixtures is maximized in the framework of AVI. Once the inference model is optimized, it can be used for estimating the latent features of sources included in unseen mixture signals. The experimental results show that the proposed method outperformed the state-of-the-art BSS methods based on linear generative models and was comparable to a method based on supervised learning of the VAE-based sourcemodel.

Unsupervised Hyperspectral and Multispectral Images Fusion Based on Nonlinear Variational Probabilistic Generative Model.

Due to hardware limitations, it is challenging for sensors to acquire images of high resolution in both spatial and spectral domains, which arouses a trend that utilizing a low-resolution hyperspectral image (LR-HSI) and a high-resolution multispectral image (HR-MSI) to fuse an HR-HSI in an unsupervised manner. Considering the fact that most existing methods are restricted by using linear spectral unmixing, we propose a nonlinear variational probabilistic generative model (NVPGM) for the unsupervised fusion task based on nonlinear unmixing. We model the joint full likelihood of the observed pixels in an LR-HSI and an HR-MSI, both of which are assumed to be generated from the corresponding latent representations, i.e., the abundance vectors. The sufficient statistics of the generative conditional distributions are nonlinear functions with respect to the latent variable, realized by neural networks, which results in a nonlinear spectral mixture model. For scalability and efficiency, we construct two recognition models to infer the latent representations, which are parameterized by neural networks as well. Simultaneously inferring the latent representations and optimizing the parameters are achieved using stochastic gradient variational inference, after which the target HR-HSI is retrieved via feedforward mapping. Though without supervised information about the HR-HSI, NVPGM still can be trained based on extra LR-HSI and HR-MSI data sets in advance unsupervisedly and processes the images at the test phase in real time. Three commonly used data sets are used to evaluate the effectiveness and efficiency of NVPGM, illustrating the outperformance of NVPGM in the unsupervised LR-HSI and HR-MSI fusion task.

Recurrent Variational Autoencoders for Learning Nonlinear Generative Models in the Presence of Outliers

This paper explores two useful modifications of the recent variational autoencoder (VAE), a popular deep generative modeling framework that dresses traditional autoencoders with probabilistic attire. The first involves a specially-tailored form of conditioning that allows us to simplify the VAE decoder structure while simultaneously introducing robustness to outliers. In a related vein, a second, complementary alteration is proposed to further build invariance to contaminated or dirty samples via a data augmentation process that amounts to recycling. In brief, to the extent that the VAE is legitimately a representative generative model, then each output from the decoder should closely resemble an authentic sample, which can then be resubmitted as a novel input ad infinitum. Moreover, this can be accomplished via special recurrent connections without the need for additional parameters to be trained. We evaluate these proposals on multiple practical outlier-removal and generative modeling tasks involving nonlinear low-dimensional manifolds, demonstrating considerable improvements over existing algorithms.

Cross-correlation conditional restricted Boltzmann machines for modeling motion style

Temporal dependency plays a fundamental role in nonlinear generative models for capturing temporal features. One such model, the conditional restricted Boltzmann machine (CRBM), learns these temporal dependencies by considering the visible variables in the previous time slice as additional fixed inputs so that static and temporal features can be captured simultaneously to generate new human motions. However, the temporal dependencies in the CRBM fail to describe various common equilibrium postures in human motion. In this paper, we present cross-correlation as a new representation for modeling temporal dependencies by introducing the Pearson correlation coefficient. We also propose an approach to enhance the discrimination of the CRBM for various human motions by incorporating cross-correlation and temporal dependency features. The experimental results on benchmark databases demonstrate that the proposed method not only retains all the merits of the CRBM, such as exact inference and efficient learning, but also greatly improves the model’s ability to blend motion styles and achieve smooth transitions between various motion segments.

Biophysically interpretable recurrent neural network for functional magnetic resonance imaging analysis and sparsity based causal architecture discovery.

Recent efforts use state-of-the-art Recurrent Neural Networks (RNN) to gain insight into neuroscience. A limitation of these works is that the used generic RNNs lack biophysical meaning, making the interpretation of the results in a neuroscience context difficult. In this paper, we propose a biophysically interpretable RNN built on the Dynamic Causal Modelling (DCM). DCM is an advanced nonlinear generative model typically used to test hypotheses of brain causal architectures and associated effective connectivities. We show that DCM can be cast faithfully as a special form of a new generalized RNN. In the resulting DCM-RNN, the hidden states are neural activity, blood flow, blood volume, and deoxyhemoglobin content and the parameters are biological quantities such as effective connectivity, oxygen extraction fraction and vessel wall elasticity. DCM-RNN is a versatile tool for neuroscience with great potential especially when combined with deep learning networks. In this study, we explore sparsity- based causal architecture discovery with DCM-RNN. In the experiments, we demonstrate that DCM-RNN equipped with $l_{1}$ connectivity regulation is more robust to noise and more powerful at discovering sparse architectures than classic DCM with $l_{2}$ connectivity regulation.

ANALYSIS AND CONTROL OF FACIAL EXPRESSIONS USING DECOMPOSABLE NONLINEAR GENERATIVE MODELS

Facial expressions convey personal characteristics and subtle emotional states. This paper presents a new framework for modeling subtle facial motions of different people with different types of expressions from high-resolution facial expression tracking data to synthesize new stylized subtle facial expressions. A conceptual facial motion manifold is used for a unified representation of facial motion dynamics from three-dimensional (3D) high-resolution facial motions as well as from two-dimensional (2D) low-resolution facial motions. Variant subtle facial motions in different people with different expressions are modeled by nonlinear mappings from the embedded conceptual manifold to input facial motions using empirical kernel maps. We represent facial expressions by a factorized nonlinear generative model, which decomposes expression style factors and expression type factors from different people with multiple expressions. We also provide a mechanism to control the high-resolution facial motion model from low-resolution facial video sequence tracking and analysis. Using the decomposable generative model with a common motion manifold embedding, we can estimate parameters to control 3D high resolution facial expressions from 2D tracking results, which allows performance-driven control of high-resolution facial expressions.

Learning Nonlinear Generative Models of Time Series With a Kalman Filter in RKHS

This paper presents a novel generative model for time series based on the Kalman filter algorithm in a reproducing kernel Hilbert space (RKHS) using the conditional embedding operator. The end result is a nonlinear model that quantifies the hidden state uncertainty and propagates its probability distribution forward as in the Kalman algorithm. The embedded dynamics can be described by the estimated conditional embedding operator constructed directly from the training measurement data. Using this operator as the counterpart of the state transition matrix, we reformulate the Kalman filter algorithm in RKHS. For the state model, the hidden states are the estimated embeddings of the measurement distribution, while the measurement model serves to connect the estimated measurement embeddings with the current mapped measurements in the RKHS. This novel algorithm is applied to noisy time-series estimation and prediction, and simulation results show that it outperforms other existing algorithms. In addition, improvements are proposed to reduce the size of the operator and reduce the computation complexity.

Tracking hand rotation and various grasping gestures from an IR camera using extended cylindrical manifold embedding

This paper presents a new approach for tracking hand rotation and various grasping gestures through an infrared camera. For the complexity and ambiguity of an observed hand shape, it is difficult to simultaneously estimate hand configuration and orientation from a silhouette image of a grasping hand gesture. This paper proposes a dynamic shape model for hand grasping gestures using cylindrical manifold embedding to analyze variations of hand shape in different hand configurations between two key hand poses and in simultaneous circular view change by hand rotation. An arbitrary hand shape between two key hand poses from any view can be generated using a cylindrical manifold embedding point after learning nonlinear generative models from the embedding space to the corresponding hand shape observed. The cylindrical manifold embedding model is extended to various grasping gestures by decomposing multiple cylindrical manifold embeddings through grasping style analysis. Grasping hand gestures with simultaneous hand rotation are tracked using particle filters on the manifold space with grasping style estimation. Experimental results for synthetic and real data indicate that the proposed model can accurately track various grasping gestures with hand rotation. The proposed approach may be applied to advanced user interfaces in dark environments by using images beyond the visible spectrum.

Two Distributed-State Models For Generating High-Dimensional Time Series

In this paper we develop a class of nonlinear generative models for high-dimensional time series. We first propose a model based on the restricted Boltzmann machine (RBM) that uses an undirected model with binary latent variables and real-valued variables. The latent and visible variables at each time step receive directed connections from the visible variables at the last few time-steps. This conditional RBM (CRBM) makes on-line inference efficient and allows us to use a simple approximate learning procedure. We demonstrate the power of our approach by synthesizing various sequences from a model trained on motion capture data and by performing on-line filling in of data lost during capture. We extend the CRBM in a way that preserves its most important computational properties and introduces multiplicative three-way interactions that allow the effective interaction weight between two variables to be modulated by the dynamic state of a third variable. We introduce a factoring of the implied three-way weight tensor to permit a more compact parameterization. The resulting model can capture diverse styles of motion with a single set of parameters, and the three-way interactions greatly improve its ability to blend motion styles or to transition smoothly among them. Videos and source code can be found at http://www.cs.nyu.edu/~gwtaylor/publications/jmlr2011.

DYNAMIC SHAPE STYLE ANALYSIS: BILINEAR AND MULTILINEAR HUMAN IDENTIFICATION WITH TEMPORAL NORMALIZATION

Modeling and analyzing the dynamic shape of human motion is a challenging task owing to temporal variations in the shape and multiple sources of observed shape variations such as viewpoint, motion speed, clothing, etc. We present a new framework for dynamic shape analysis based on temporal normalization and factorized shape style analysis. Using a nonlinear generative model with motion manifold embedding in a low-dimensional space, we detect cycles of periodic motion like gait in different views and synthesize temporally-aligned shape sequences from the same type of motion at different speeds. The bilinear analysis of temporally-aligned shape sequences decomposes dynamic motion into time-invariant shape style factors and time-dependent motion factors. We extend the bilinear model into a tensor shape model, a multilinear decomposition of dynamic shape sequences for view-invariant shape style representations. The shape style is a view-invariant, time-invariant, and speed-invariant shape signature and is used as a feature vector for human identification. The shape style can be adapted to new environmental conditions by iterative estimation of style and content factors to reflect new observation conditions. We present the experimental results of gait recognition using the CMU Mobo gait database and the USF gait challenging database.

BM³E : Discriminative Density Propagation for Visual Tracking

We introduce BM3 E, a Conditional Bayesian Mixture of Experts Markov Model, for consistent probabilistic estimates in discriminative visual tracking. The model applies to problems of temporal and uncertain inference and represents the unexplored bottom-up counterpart of pervasive generative models estimated with Kalman filtering or particle filtering. Instead of inverting a non-linear generative observation model at run-time, we learn to cooperatively predict complex state distributions directly from descriptors that encode image observations - typically bag-of-feature global image histograms or descriptors computed over regular spatial grids. These are integrated in a conditional graphical model in order to enforce temporal smoothness constraints and allow a principled management of uncertainty. The algorithms combine sparsity, mixture modeling, and non-linear dimensionality reduction for efficient computation in high-dimensional continuous state spaces. The combined system automatically self-initializes and recovers from failure. The research has three contributions: (1) We establish the density propagation rules for discriminative inference in continuous, temporal chain models; (2) We propose flexible supervised and unsupervised algorithms for learning feedforward, multivalued contextual mappings (multimodal state distributions) based on compact, conditional Bayesian mixture of experts models; (3) We validate the framework empirically for the reconstruction of 3d human motion in monocular video sequences. Our tests on both real and motion capture-based sequences show significant performance gains with respect to competing nearest-neighbor, regression, and structured prediction methods.

Recognizing handwritten digits using hierarchical products of experts

The product of experts learning procedure can discover a set of stochastic binary features that constitute a nonlinear generative model of handwritten images of digits. The quality of generative models learned in this way can be assessed by learning a separate model for each class of digit and then comparing the unnormalized probabilities of test images under the 10 different class-specific models. To improve discriminative performance, a hierarchy of separate models can be learned, for each digit class. Each model in the hierarchy learns a layer of binary feature detectors that model the probability distribution of vectors of activity of feature detectors in the layer below. The models in the hierarchy are trained sequentially and each model uses a layer of binary feature detectors to learn a generative model of the patterns of feature activities in the preceding layer. After training, each layer of feature detectors produces a separate, unnormalized log probability score. With three layers of feature detectors for each of the 10 digit classes, a test image produces 30 scores which can be used as inputs to a supervised, logistic classification network that is trained on separate data.

Probabilistic Detection and Tracking of Motion Boundaries

We propose a Bayesian framework for representing and recognizing local image motion in terms of two basic models: translational motion and motion boundaries. Motion boundaries are represented using a non-linear generative model that explicitly encodes the orientation of the boundary, the velocities on either side, the motion of the occluding edge over time, and the appearance/disappearance of pixels at the boundary. We represent the posterior probability distribution over the model parameters given the image data using discrete samples. This distribution is propagated over time using a particle filtering algorithm. To efficiently represent such a high-dimensional space we initialize samples using the responses of a low-level motion discontinuity detector. The formulation and computational model provide a general probabilistic framework for motion estimation with multiple, non-linear, models.