Multimodal Feature Learning Research Articles

Self-Distillation Feature Learning Network for Optical and SAR Image Registration

Optical and SAR image registration is important for multi-modal remote sensing image information fusion. Recently, deep matching networks have shown better performances than traditional methods on image matching. However, due to significant differences between optical and SAR images, the performances of existing deep learning methods still need to be further improved. This paper proposes a self-distillation feature learning network (SDNet) for optical and SAR image registration, improving performance from network structure and network optimization. Firstly, we explore the impact of different weight-sharing strategies on optical and SAR image matching. Then, we design a partially unshared feature learning network for multi-modal image feature learning. It has fewer parameters than the fully unshared network and has more flexibility than the fully shared network. Additionally, the limited binary supervised information (matching or non-matching) is insufficient to train the deep matching networks for optical-SAR image registration. Thus, we propose a self-distillation feature learning method to exploit more similarity information for deep network optimization enhancing, such as the similarity ordering between a series of non-matching patch-pairs. The exploited rich similarity information will significantly enhance network training and improve matching accuracy. Finally, considering that existing deep learning methods brute-force constrain the features of the matching optical and SAR image patches are similar, which will be lost many discriminative information, degenerating matching performances. Thus, we build an auxiliary task reconstruction learning to optimize the feature learning network to keep more discriminative information. Extensive experiments demonstrate the effectiveness of our proposed method on multi-modal image registration.

A Multimodal Approach for Identifying Autism Spectrum Disorders in Children.

Identification of autism spectrum disorder (ASD) in children is challenging due to the complexity and heterogeneity of ASD. Currently, most existing methods mainly rely on a single modality with limited information and often cannot achieve satisfactory performance. To address this issue, this paper investigates from internal neurophysiological and external behavior perspectives simultaneously and proposes a new multimodal diagnosis framework for identifying ASD in children with fusion of electroencephalogram (EEG) and eye-tracking (ET) data. Specifically, we designed a two-step multimodal feature learning and fusion model based on a typical deep learning algorithm, stacked denoising autoencoder (SDAE). In the first step, two SDAE models are designed for feature learning for EEG and ET modality, respectively. Then, a third SDAE model in the second step is designed to perform multimodal fusion with learned EEG and ET features in a concatenated way. Our designed multimodal identification model can automatically capture correlations and complementarity from behavior modality and neurophysiological modality in a latent feature space, and generate informative feature representations with better discriminability and generalization for enhanced identification performance. We collected a multimodal dataset containing 40 ASD children and 50 typically developing (TD) children to evaluate our proposed method. Experimental results showed that our proposed method achieved superior performance compared with two unimodal methods and a simple feature-level fusion method, which has promising potential to provide an objective and accurate diagnosis to assist clinicians.

Learning Social Relationship From Videos via Pre-Trained Multimodal Transformer

As a crucial task for video analysis, social relation recognition from characters provides intelligent applications with great potential to better understand the behaviors or emotions of human beings. Most existing methods mainly focus on training models from a large amount of labeled data. However, labeling social relations in videos is time-consuming. To solve this problem, we propose a Pre-trained Multimodal Feature Learning (PMFL) framework for self-supervised learning from unlabeled video data, and then transfer the pre-trained PMFL to downstream social relationship recognition task. First, the space-time interaction between visual instances, and cross-modal interaction between visual and textual information provide important cues for social relation understanding. To incorporate these cues, we design a Multimodal Instance Interaction Transformer (MIIT), which consists of two Transformers to capture intra-modal and cross-modal information interaction, respectively. Second, to better endow PMFL with the capability of learning visual and textual semantic features, we pre-train it via two tasks: Masked Action Feature Regression (MAFR) and Masked Object Label Classification (MOLC). These tasks can help learn both intra-modal and cross-modal semantic information. After fine-tuning PMFL from pre-trained parameters, it achieves the state-of-the-art results on a public benchmark.

LinkNet: 2D-3D linked multi-modal network for online semantic segmentation of RGB-D videos

This paper proposes LinkNet, a 2D-3D linked multi-modal network served for online semantic segmentation of RGB-D videos, which is essential for real-time applications such as robot navigation. Existing methods for RGB-D semantic segmentation usually work in the regular image domain, which allows efficient processing using convolutional neural networks (CNNs). However, RGB-D videos are captured from a 3D scene, and different frames can contain useful information of the same local region from different views. Working solely in the image domain fails to utilize such crucial information. Our novel approach is based on joint 2D and 3D analysis. The online process is realized simultaneously with 3D scene reconstruction, from which we set up 2D-3D links between continuous RGB-D frames and 3D point cloud. We combine image color and view-insensitive geometric features generated from the 3D point cloud for multi-modal semantic feature learning. Our LinkNet further uses a recurrent neural network (RNN) module to dynamically maintain the hidden semantic states during 3D fusion, and refines the voxel-based labeling results. The experimental results on SceneNet [1] and ScanNet [2] demonstrate that the semantic segmentation results of our framework are stable and effective.

A deep multimodal feature learning network for RGB-D salient object detection

In this paper, we propose a deep multimodal feature learning (DMFL) network for RGB-D salient object detection. The color and depth features are firstly extracted from low level to high level feature using CNN. Then the features at the high layer are shared and concatenated to construct joint feature representation of multi-modalities. The fused features are embedded to a high dimension metric space to express the salient and non-salient parts. And also a new objective function, consisting of cross-entropy and metric loss, is proposed to optimize the model. Both pixel and attribute level discriminative features are learned for semantical grouping to detect the salient objects. Experimental results show that the proposed model achieves promising performance and has about 1% to 2% improvement to conventional methods.

An Effective Multimodal Image Fusion Method Using MRI and PET for Alzheimer's Disease Diagnosis.

Alzheimer's disease (AD) is an irreversible brain disease that severely damages human thinking and memory. Early diagnosis plays an important part in the prevention and treatment of AD. Neuroimaging-based computer-aided diagnosis (CAD) has shown that deep learning methods using multimodal images are beneficial to guide AD detection. In recent years, many methods based on multimodal feature learning have been proposed to extract and fuse latent representation information from different neuroimaging modalities including magnetic resonance imaging (MRI) and 18-fluorodeoxyglucose positron emission tomography (FDG-PET). However, these methods lack the interpretability required to clearly explain the specific meaning of the extracted information. To make the multimodal fusion process more persuasive, we propose an image fusion method to aid AD diagnosis. Specifically, we fuse the gray matter (GM) tissue area of brain MRI and FDG-PET images by registration and mask coding to obtain a new fused modality called “GM-PET.” The resulting single composite image emphasizes the GM area that is critical for AD diagnosis, while retaining both the contour and metabolic characteristics of the subject's brain tissue. In addition, we use the three-dimensional simple convolutional neural network (3D Simple CNN) and 3D Multi-Scale CNN to evaluate the effectiveness of our image fusion method in binary classification and multi-classification tasks. Experiments on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset indicate that the proposed image fusion method achieves better overall performance than unimodal and feature fusion methods, and that it outperforms state-of-the-art methods for AD diagnosis.

Joint Discriminative Sparse Coding for Robust Hand-Based Multimodal Recognition

Multimodal biometrics recognition has recently attracted much interest for its higher security and effectiveness compared with unimodal biometrics recognition. However, most of the conventional multimodal recognition approaches generally focus on extracting semantic information from different modalities independently, while ignoring the implicit correlations among inter-modality. In this paper, we propose a simple yet effective supervised multimodal feature learning method, called joint discriminative sparse coding (JDSC), which is applied for hand-based multimodal recognition including finger-vein and finger-knuckle-print fusion, palm-vein and palmprint fusion, as well as palm-vein and dorsal-hand-vein fusion. Considering that relevant samples from different modalities have semantic correlations, JDSC projects the raw data into a shared space in which the distance of the between-class is maximized and the distance of the within-class is minimized, at the same time, the correlation among the inter-modality of the within-class is maximized. Therefore, sparse binary codes quantified by the obtained projection matrix can have more discriminative power for multimodal recognition tasks. Thorough experiments on six commonly used multimodal datasets demonstrate the superiority of our proposed method over several state-of-the-art techniques.

Learning deep multimodal affective features for spontaneous speech emotion recognition

Recently, spontaneous speech emotion recognition has become an active and challenging research subject. This paper proposes a new method of spontaneous speech emotion recognition by using deep multimodal audio feature learning based on multiple deep convolutional neural networks (multi-CNNs). The proposed method initially generates three different audio inputs for multi-CNNs so as to learn deep multimodal segment-level features from the original 1D audio signal in three aspects: 1) a 1D CNN for 1D raw waveform modeling, 2) a 2D CNN for 2D time-frequency Mel-spectrogram modeling, and 3) a 3D CNN for temporal-spatial dynamic modeling. Then, an average-pooling is performed on the obtained segment-level classification results from 1D, 2D, and 3D CNN networks, to produce utterance-level classification results. Finally, a score-level fusion strategy is adopted as a multi-CNN fusion method to integrate different utterance-level classification results for final emotion classification. The learned deep multimodal audio features are shown to be complementary to each other so that they are combined in a multi-CNN fusion network to achieve significantly improved emotion classification performance. Experiments are conducted on two challenging spontaneous emotional speech datasets, i.e., the AFEW5.0 and BAUM-1 s databases, demonstrating the promising performance of our proposed method.

New Ideas and Trends in Deep Multimodal Content Understanding: A Review

The focus of this survey is on the analysis of two modalities of multimodal deep learning: image and text. Unlike classic reviews of deep learning where monomodal image classifiers such as VGG, ResNet and Inception module are central topics, this paper will examine recent multimodal deep models and structures, including auto-encoders, generative adversarial nets and their variants. These models go beyond the simple image classifiers in which they can do uni-directional (e.g. image captioning, image generation) and bi-directional (e.g. cross-modal retrieval, visual question answering) multimodal tasks. Besides, we analyze two aspects of the challenge in terms of better content understanding in deep multimodal applications. We then introduce current ideas and trends in deep multimodal feature learning, such as feature embedding approaches and objective function design, which are crucial in overcoming the aforementioned challenges. Finally, we include several promising directions for future research.

C-GCN: Correlation Based Graph Convolutional Network for Audio-Video Emotion Recognition

With the development of both hardware and deep neural network technologies, tremendous improvements have been achieved in the performance of automatic emotion recognition (AER) based on the video data. However, AER is still a challenging task due to subtle expression, abstract concept of emotion and the representation of multi-modal information. Most proposed approaches focus on the multi-modal feature learning and fusion strategy, which pay more attention to the characteristic of a single video and ignore the correlation among the videos. To explore this correlation, in this paper, we propose a novel correlation-based graph convolutional network (C-GCN) for AER, which can comprehensively consider the correlation of the intra-class and inter-class videos for feature learning and information fusion. More specifically, we introduce the graph model to represent the correlation among the videos. This correlated information can help to improve the discrimination of node features in the progress of graph convolutional network. Meanwhile, the multi-head attention mechanism is applied to predict the hidden relationship among the videos, which can strengthen the inter-class correlation to improve the performance of classifiers. The C-GCN is evaluated on the AFEW datasets and eNTERFACE 05 dataset. The final experimental results demonstrate the superiority of our proposed method over the state-of-the-art methods.

Private and common feature learning with adversarial network for RGBD object classification

A key issue in RGBD classification is how to fuse the RGB and depth modalities. A popular way is to extract the private features of unimodality and the common features between the two modalities. Most of them use low order algebraic metrics to find the common part of the RGB and depth signals. In this paper, adversarial network is used to learn the common features between the RGB and depth modalities. A modality discriminator is designed to compete with the feature encoder so as to generate the modality-invariant information, which can be regarded as the common features. With this concept, we present a Multi-Modal Feature Learning algorithm with Adversarial Network (MMFLAN) to decouple the RGBD signals and obtain the fused features. Comprehensive experiments based on three datasets are used to evaluate the effectiveness and robustness of our MMFLAN.

Weakly-supervised learning of multi-modal features for regularised iterative descent in 3D image registration.

Methods for deep learning based medical image registration have only recently approached the quality of classical model-based image alignment. The dual challenge of both a very large trainable parameter space and often insufficient availability of expert supervised correspondence annotations has led to slower progress compared to other domains such as image segmentation. Yet, image registration could also more directly benefit from an iterative solution than segmentation. We therefore believe that significant improvements, in particular for multi-modal registration, can be achieved by disentangling appearance-based feature learning and deformation estimation. In this work, we examine an end-to-end trainable, weakly-supervised deep learning-based feature extraction approach that is able to map the complex appearance to a common space. Our results on thoracoabdominal CT and MRI image registration show that the proposed method compares favourably well to state-of-the-art hand-crafted multi-modal features, Mutual Information-based approaches and fully-integrated CNN-based methods - and handles even the limitation of small and only weakly-labeled training data sets.

Web Objectionable Video Recognition Based on Deep Multi-Instance Learning With Representative Prototypes Selection

To protect underage people from accessing objectionable videos in the Internet, an effective objectionable video recognition algorithm is necessary for web filtering. Recently, the multi-instance learning has been introduced for objectionable video recognition and achieves impressive results. However, hand-crafted features as well as redundant and noisy frames in objectionable videos become an intractable problem that inevitably degrades the recognition performance. In this paper, we propose a novel representative prototype selection algorithm embedding deep multi-instance representation learning. In the proposed method, an improved convolutional neural network is designed for multimodal multi-instance feature learning and a self-expressive dictionary learning model based on sparse and low rank constraint is designed to select the representative prototypes from each subspace of instances. Then the bag-level feature is constructed via mapping the bag to the selected prototypes. Experiments on three objectionable video sets show the effectiveness of our method for objectionable video recognition.

Anomaly Feature Learning for Unsupervised Change Detection in Heterogeneous Images: A Deep Sparse Residual Model

In this article, we propose a novel and simple automatic model based on multimodal anomaly feature learning in a residual space, aiming at solving the binary classification problem of temporal change detection (CD) between pairs of heterogeneous remote sensing images. The model starts by learning from image pairs the normal existing patterns in the before and after images to come up with a suitable representation of the normal (nonchange) class. To achieve this, we employ a stacked sparse autoencoder trained on a large number of temporal image features (training data) in an unsupervised manner. To classify pixels of new unseen image-pairs, the built anomaly detection model reconstructs the input from its representation in the latent space. First, the probe (new) image (i.e., the bitemporal heterogeneous image pair as the input request) is encoded in this compact normal space from a stacked hidden representation. The reconstruction error is computed using the L2 norm in what we call the residual normal space. In which, the nonchange patterns are characterized by small reconstruction errors as a normal class while the change patterns are quantified by high reconstruction errors categorizing the abnormal class. The dichotomic (changed/unchanged) classification map is generated in the residual space by clustering the reconstructed errors using a Gaussian mixture model. Experimental results on different real heterogeneous images, reflecting a mixture of imaging and land surface CD conditions, confirm the robustness of the proposed anomaly detection model.

Multimodal feature learning and fusion on B-mode ultrasonography and sonoelastography using point-wise gated deep networks for prostate cancer diagnosis.

B-mode ultrasonography and sonoelastography are used in the clinical diagnosis of prostate cancer (PCa). A combination of the two ultrasound (US) modalities using computer aid may be helpful for improving the diagnostic performance. A technique for computer-aided diagnosis (CAD) of PCa is presented based on multimodal US. Firstly, quantitative features are extracted from both B-mode US images and sonoelastograms, including intensity statistics, regional percentile features, gray-level co-occurrence matrix (GLCM) texture features and binary texture features. Secondly, a deep network named PGBM-RBM2 is proposed to learn and fuse multimodal features, which is composed of the point-wise gated Boltzmann machine (PGBM) and two layers of the restricted Boltzmann machines (RBMs). Finally, the support vector machine (SVM) is used for prostatic disease classification. Experimental evaluation was conducted on 313 multimodal US images of the prostate from 103 patients with prostatic diseases (47 malignant and 56 benign). Under five-fold cross-validation, the classification sensitivity, specificity, accuracy, Youden's index and area under the receiver operating characteristic (ROC) curve with the PGBM-RBM2 were 87.0%, 88.8%, 87.9%, 75.8% and 0.851, respectively. The results demonstrate that multimodal feature learning and fusion using the PGBM-RBM2 can assist in the diagnosis of PCa. This deep network is expected to be useful in the clinical diagnosis of PCa.

Deep Robust Unsupervised Multi-Modal Network

In real-world applications, data are often with multiple modalities, and many multi-modal learning approaches are proposed for integrating the information from different sources. Most of the previous multi-modal methods utilize the modal consistency to reduce the complexity of the learning problem, therefore the modal completeness needs to be guaranteed. However, due to the data collection failures, self-deficiencies, and other various reasons, multi-modal instances are often incomplete in real applications, and have the inconsistent anomalies even in the complete instances, which jointly result in the inconsistent problem. These degenerate the multi-modal feature learning performance, and will finally affect the generalization abilities in different tasks. In this paper, we propose a novel Deep Robust Unsupervised Multi-modal Network structure (DRUMN) for solving this real problem within a unified framework. The proposed DRUMN can utilize the extrinsic heterogeneous information from unlabeled data against the insufficiency caused by the incompleteness. On the other hand, the inconsistent anomaly issue is solved with an adaptive weighted estimation, rather than adjusting the complex thresholds. As DRUMN can extract the discriminative feature representations for each modality, experiments on real-world multimodal datasets successfully validate the effectiveness of our proposed method.

User community detection via embedding of social network structure and temporal content

Identifying and extracting user communities is an important step towards understanding social network dynamics from a macro perspective. For this reason, the work in this paper explores various aspects related to the identification of user communities. To date, user community detection methods employ either explicit links between users (link analysis), or users’ topics of interest in posted content (content analysis), or in tandem. Little work has considered temporal evolution when identifying user communities in a way to group together those users who share not only similar topical interests but also similar temporal behavior towards their topics of interest. In this paper, we identify user communities through multimodal feature learning (embeddings). Our core contributions can be enumerated as (a) we propose a new method for learning neural embeddings for users based on their temporal content similarity; (b) we learn user embeddings based on their social network connections (links) through neural graph embeddings; (c) we systematically interpolate temporal content-based embeddings and social link-based embeddings to capture both social network connections and temporal content evolution for representing users, and (d) we systematically evaluate the quality of each embedding type in isolation and also when interpolated together and demonstrate their performance on a Twitter dataset under two different application scenarios, namely news recommendation and user prediction. We find that (1) content-based methods produce higher quality communities compared to link-based methods; (2) methods that consider temporal evolution of content, our proposed method in particular, show better performance compared to their non-temporal counter-parts; (3) communities that are produced when time is explicitly incorporated in user vector representations have higher quality than the ones produced when time is incorporated into a generative process, and finally (4) while link-based methods are weaker than content-based methods, their interpolation with content-based methods leads to improved quality of the identified communities.

Collaborative multimodal feature learning for RGB-D action recognition

The emergence of cost-effective depth sensors opens up a new dimension for RGB-D based human action recognition. In this paper, we propose a collaborative multimodal feature learning (CMFL) model for human action recognition from RGB-D sequences. Specifically, we propose a robust spatio-temporal pyramid feature (RSTPF) to capture dynamic local patterns around each human joint. The proposed CMFL model fuses multimodal data (skeleton, depth and RGB), and learns action classifiers using the fused features. The original low-level feature matrices are factorized to learn shared features and modality-specific features under a supervised fashion. The shared features describe the common structures among the three modalities while the modality-specific features capture intrinsic information of each modality. We formulate shared-specific features mining and action classifiers learning in a unified max-margin framework, and solve the formulation using an iterative optimization algorithm. Experimental results on four action datasets demonstrate the efficacy of the proposed method.

RGB-D Object Recognition Using Multi-Modal Deep Neural Network and DS Evidence Theory.

With the development of low-cost RGB-D (Red Green Blue-Depth) sensors, RGB-D object recognition has attracted more and more researchers’ attention in recent years. The deep learning technique has become popular in the field of image analysis and has achieved competitive results. To make full use of the effective identification information in the RGB and depth images, we propose a multi-modal deep neural network and a DS (Dempster Shafer) evidence theory based RGB-D object recognition method. First, the RGB and depth images are preprocessed and two convolutional neural networks are trained, respectively. Next, we perform multi-modal feature learning using the proposed quadruplet samples based objective function to fine-tune the network parameters. Then, two probability classification results are obtained using two sigmoid SVMs (Support Vector Machines) with the learned RGB and depth features. Finally, the DS evidence theory based decision fusion method is used for integrating the two classification results. Compared with other RGB-D object recognition methods, our proposed method adopts two fusion strategies: Multi-modal feature learning and DS decision fusion. Both the discriminative information of each modality and the correlation information between the two modalities are exploited. Extensive experimental results have validated the effectiveness of the proposed method.

An ensemble long short-term memory neural network for hourly PM2.5 concentration forecasting

To protect public health by providing an early warning, PM2.5 concentration forecasting is an essential and effective work. In this paper, an ensemble long short-term memory neural network (E-LSTM) is proposed for hourly PM2.5 concentration forecasting. The presented model is implemented using three steps: (1) ensemble empirical mode decomposition (EEMD) is firstly utilized for multi-modal feature extraction, (2) long short-term memory approach (LSTM) is then employed for multi-modal feature learning, and (3) inverse EEMD computation is finally used for multi-modal feature estimated integration. In each modeling process, the mode information of the PM2.5 and the corresponding meteorological variables in 1-h advance are utilized as inputs to forecast the next mode information of the PM2.5 concentration. To evaluate the performance of the E-LSTM model, two datasets collected from two environmental monitoring stations in Beijing, China, are investigated. It is demonstrated that the E-LSTM model inspired by ensemble learning, which constructs multiple LSTMs in different modes, obtained better forecasting performance than that using the single LSTM and feed forward neural network in terms of mean absolute percentage error (19.604% and 16.929%), root mean square error (12.077 μg m−3 and 13.983 μg m−3), and correlation coefficient criteria (0.994 and 0.991) respectively.