Multimodal Deep Belief Network Research Articles

Active Inference Through Energy Minimization in Multimodal Affective Human-Robot Interaction.

During communication, humans express their emotional states using various modalities (e.g., facial expressions and gestures), and they estimate the emotional states of others by paying attention to multimodal signals. To ensure that a communication robot with limited resources can pay attention to such multimodal signals, the main challenge involves selecting the most effective modalities among those expressed. In this study, we propose an active perception method that involves selecting the most informative modalities using a criterion based on energy minimization. This energy-based model can learn the probability of the network state using energy values, whereby a lower energy value represents a higher probability of the state. A multimodal deep belief network, which is an energy-based model, was employed to represent the relationships between the emotional states and multimodal sensory signals. Compared to other active perception methods, the proposed approach demonstrated improved accuracy using limited information in several contexts associated with affective human–robot interaction. We present the differences and advantages of our method compared to other methods through mathematical formulations using, for example, information gain as a criterion. Further, we evaluate performance of our method, as pertains to active inference, which is based on the free energy principle. Consequently, we establish that our method demonstrated superior performance in tasks associated with mutually correlated multimodal information.

DeepFrag-k: a fragment-based deep learning approach for protein fold recognition

BackgroundOne of the most essential problems in structural bioinformatics is protein fold recognition. In this paper, we design a novel deep learning architecture, so-called DeepFrag-k, which identifies fold discriminative features at fragment level to improve the accuracy of protein fold recognition. DeepFrag-k is composed of two stages: the first stage employs a multi-modal Deep Belief Network (DBN) to predict the potential structural fragments given a sequence, represented as a fragment vector, and then the second stage uses a deep convolutional neural network (CNN) to classify the fragment vector into the corresponding fold.ResultsOur results show that DeepFrag-k yields 92.98% accuracy in predicting the top-100 most popular fragments, which can be used to generate discriminative fragment feature vectors to improve protein fold recognition.ConclusionsThere is a set of fragments that can serve as structural “keywords” distinguishing between major protein folds. The deep learning architecture in DeepFrag-k is able to accurately identify these fragments as structure features to improve protein fold recognition.

Emotion recognition using multimodal deep learning in multiple psychophysiological signals and video

Emotion recognition has attracted great interest. Numerous emotion recognition approaches have been proposed, most of which focus on visual, acoustic or psychophysiological information individually. Although more recent research has considered multimodal approaches, individual modalities are often combined only by simple fusion or are directly fused with deep learning networks at the feature level. In this paper, we propose an approach to training several specialist networks that employs deep learning techniques to fuse the features of individual modalities. This approach includes a multimodal deep belief network (MDBN), which optimizes and fuses unified psychophysiological features derived from the features of multiple psychophysiological signals, a bimodal deep belief network (BDBN) that focuses on representative visual features among the features of a video stream, and another BDBN that focuses on the high multimodal features in the unified features obtained from two modalities. Experiments are conducted on the BioVid Emo DB database and 80.89% accuracy is achieved, which outperforms the state-of-the-art approaches. The results demonstrate that the proposed approach can solve the problems of feature redundancy and lack of key features caused by multimodal fusion.

Discovering hierarchical common brain networks via multimodal deep belief network.

Studying a common architecture reflecting both brain's structural and functional organizations across individuals and populations in a hierarchical way has been of significant interest in the brain mapping field. Recently, deep learning models exhibited ability in extracting meaningful hierarchical structures from brain imaging data, e.g., fMRI and DTI. However, deep learning models have been rarely used to explore the relation between brain structure and function yet. In this paper, we proposed a novel multimodal deep believe network (DBN) model to discover and quantitatively represent the hierarchical organizations of common and consistent brain networks from both fMRI and DTI data. A prominent characteristic of DBN is that it is capable of extracting meaningful features from complex neuroimaging data with a hierarchical manner. With our proposed DBN model, three hierarchical layers with hundreds of common and consistent brain networks across individual brains are successfully constructed through learning a large dimension of representative features from fMRI/DTI data.

Cross-modal motion regeneration using Multimodal Deep Belief Network

In this paper, we propose a Multimodal Deep Belief Network (MDBN) for learning a generative model of 2D and 3D skeletal data. The MDBM learns the cross-modal relationship between these data in the form of a joint probability distribution over the space of multimodal inputs. It can regenerate any missing modality by sampling from the conditional distribution over the given data modality. The skeletal sequences are converted into the motion images which help us utilize the impressive power of the generative deep networks in the image processing. Furthermore, we use the variation of information (VI) as the training criterion, instead of the conventional maximum likelihood. It is proven that VI is efficient in cross-modal learning where some data modalities are missing. Our experimental results have shown that the model has an outstanding performance on the over-complete MHAD and CMU Mocap datasets in data-driven motion regeneration on a full-body 2D and 3D skeleton structures.

A deep learning approach for predicting the quality of online health expert question-answering services

Recently, online health expert question-answering (HQA) services (systems) have attracted more and more health consumers to ask health-related questions everywhere at any time due to the convenience and effectiveness. However, the quality of answers in existing HQA systems varies in different situations. It is significant to provide effective tools to automatically determine the quality of the answers. Two main characteristics in HQA systems raise the difficulties of classification: (1) physicians’ answers in an HQA system are usually written in short text, which yields the data sparsity issue; (2) HQA systems apply the quality control mechanism, which refrains the wisdom of crowd. The important information, such as the best answer and the number of users’ votes, is missing. To tackle these issues, we prepare the first HQA research data set labeled by three medical experts in 90days and formulate the problem of predicting the quality of answers in the system as a classification task. We not only incorporate the standard textual feature of answers, but also introduce a set of unique non-textual features, i.e., the popular used surface linguistic features and the novel social features, from other modalities. A multimodal deep belief network (DBN)-based learning framework is then proposed to learn the high-level hidden semantic representations of answers from both textual features and non-textual features while the learned joint representation is fed into popular classifiers to determine the quality of answers. Finally, we conduct extensive experiments to demonstrate the effectiveness of including the non-textual features and the proposed multimodal deep learning framework.

Integrative Data Analysis of Multi-Platform Cancer Data with a Multimodal Deep Learning Approach.

Identification of cancer subtypes plays an important role in revealing useful insights into disease pathogenesis and advancing personalized therapy. The recent development of high-throughput sequencing technologies has enabled the rapid collection of multi-platform genomic data (e.g., gene expression, miRNA expression, and DNA methylation) for the same set of tumor samples. Although numerous integrative clustering approaches have been developed to analyze cancer data, few of them are particularly designed to exploit both deep intrinsic statistical properties of each input modality and complex cross-modality correlations among multi-platform input data. In this paper, we propose a new machine learning model, called multimodal deep belief network (DBN), to cluster cancer patients from multi-platform observation data. In our integrative clustering framework, relationships among inherent features of each single modality are first encoded into multiple layers of hidden variables, and then a joint latent model is employed to fuse common features derived from multiple input modalities. A practical learning algorithm, called contrastive divergence (CD), is applied to infer the parameters of our multimodal DBN model in an unsupervised manner. Tests on two available cancer datasets show that our integrative data analysis approach can effectively extract a unified representation of latent features to capture both intra- and cross-modality correlations, and identify meaningful disease subtypes from multi-platform cancer data. In addition, our approach can identify key genes and miRNAs that may play distinct roles in the pathogenesis of different cancer subtypes. Among those key miRNAs, we found that the expression level of miR-29a is highly correlated with survival time in ovarian cancer patients. These results indicate that our multimodal DBN based data analysis approach may have practical applications in cancer pathogenesis studies and provide useful guidelines for personalized cancer therapy.