Representation Learning Ability Research Articles

A Bridge Neural Network-Based Optical-SAR Image Joint Intelligent Interpretation Framework

The current interpretation technology of remote sensing images is mainly focused on single-modal data, which cannot fully utilize the complementary and correlated information of multimodal data with heterogeneous characteristics, especially for synthetic aperture radar (SAR) data and optical imagery. To solve this problem, we propose a bridge neural network- (BNN-) based optical-SAR image joint intelligent interpretation framework, optimizing the feature correlation between optical and SAR images through optical-SAR matching tasks. It adopts BNN to effectively improve the capability of common feature extraction of optical and SAR images and thus improving the accuracy and application scenarios of specific intelligent interpretation tasks for optical-SAR/SAR/optical images. Specifically, BNN projects optical and SAR images into a common feature space and mines their correlation through pair matching. Further, to deeply exploit the correlation between optical and SAR images and ensure the great representation learning ability of BNN, we build the QXS-SAROPT dataset containing 20,000 pairs of perfectly aligned optical-SAR image patches with diverse scenes of high resolutions. Experimental results on optical-to-SAR crossmodal object detection demonstrate the effectiveness and superiority of our framework. In particular, based on the QXS-SAROPT dataset, our framework can achieve up to 96% high accuracy on four benchmark SAR ship detection datasets.

Redundancy Removal Adversarial Active Learning Based on Norm Online Uncertainty Indicator.

Active learning aims to select the most valuable unlabelled samples for annotation. In this paper, we propose a redundancy removal adversarial active learning (RRAAL) method based on norm online uncertainty indicator, which selects samples based on their distribution, uncertainty, and redundancy. RRAAL includes a representation generator, state discriminator, and redundancy removal module (RRM). The purpose of the representation generator is to learn the feature representation of a sample, and the state discriminator predicts the state of the feature vector after concatenation. We added a sample discriminator to the representation generator to improve the representation learning ability of the generator and designed a norm online uncertainty indicator (Norm-OUI) to provide a more accurate uncertainty score for the state discriminator. In addition, we designed an RRM based on a greedy algorithm to reduce the number of redundant samples in the labelled pool. The experimental results on four datasets show that the state discriminator, Norm-OUI, and RRM can improve the performance of RRAAL, and RRAAL outperforms the previous state-of-the-art active learning methods.

Structural Deep Nonnegative Matrix Factorization for community detection

Due to the important role in analyzing the topological structure of complex networks, community detection has attracted increasing attention recently. The network embedding methods have shown promising performances in community detection, aiming to learn low-dimensional representations of nodes in networks. Among them, Nonnegative Matrix Factorization (NMF) is proved to be an efficient approach. However, considering that the mapping between the original network and the community membership space contains rather complex hierarchical information, classic shallow-NMF approaches often fail to capture the complex underlying network structure, resulting in sub-optimal network representations. Therefore, how to find a method that is able to effectively capture the complex underlying network structure and preserve the global and local structure is an open and important topic. Inspired by the excellent ability of representation learning in deep autoencoder, we propose a Structural Deep Nonnegative Matrix Factorization model, named SDNMF, for community detection. Similar to deep autoencoder, the proposed multi-layer NMF-based model consists of a decoder module and an encoder module. Further, we propose to exploit the first-order similarity and second-order similarity jointly to preserve the structural information. The first-order similarity characterizes the local network information. Meanwhile, the global network information can be captured and the sparsity problem is alleviated by the second-order similarity. An efficient learning algorithm is developed to optimize the proposed DANMF model, which simultaneously optimizes the first-order and second-order similarity. The effectiveness of the proposed SDNMF is verified by comparing it with several state-of-the-art approaches for community detection on six real-word networks. The comparison is based on three performance metrics. Experimental results demonstrate that the proposed SDNMF can obtain a more accurate community membership matrix compared with the baselines. The convergence analysis is also performed to verify the efficiency of our approach.

Video person re-identification with global statistic pooling and self-attention distillation

Most existing methods for video person re-identification apply spatial-temporal global average or attention pooling to aggregate frame-level feature and get video-level feature. The obtained video-level feature models only the first-order statistics of the appearance feature from holistic video, resulting in limited representation capability of the feature network. In this paper, we propose a novel Global Statistic Pooling network (GSPnet) which takes full advantage of the second-order information for enhancing modeling capability. Firstly, a novel global statistic pooling module is proposed to summarize both the first- and second-order statistics across frame-level feature, and then transfer them into a compact and robust video-level feature embedding. Secondly, a statistic-based attention block is incorporated into multiple stages of convolutional networks to fully explore the second-order representations from low- to high-level features. To enhance the representation learning ability and further boost re-identification (re-ID) performance, we also propose a multi-level self-attention distillation training scheme, which squeezes the knowledge learned in the deeper portion of the networks into the shallow ones. Extensive experimental results have demonstrated the effectiveness and superiority of our approach on four popular video person re-ID datasets.

Data representation for CNN based internet traffic classification: a comparative study

It has been well established that the Internet of Things will bring an expansion in traffic volume and types. This will bring new challenges in terms of Quality of Service (QoS) and security, requiring innovative traffic management techniques. Traffic classification is a main network function that helps in managing both QoS and security. Different machine learning based methods have been applied for this aim. However, traditional machine learning methods rely on hand crafted features, limiting the model ability to learn. Deep Learning (DL), a branch of machine learning, is characterized by its representation learning ability. In this paper, we analyse two methods of data representation for DL-based classification: a raw packet-based representation and a quasi-raw flow-based representation. Different tests are performed to evaluate the robustness of these data representation methods. The tests include features’ importance, model robustness, and anonymization tests. The results show that raw data representation suffers from traffic anonymization and the fact that many packet fields are data-dependent. On the other hand, the flow-based representation is sensitive to the number of packets used for classification and to traffic obfuscation.

SAR target classification using multi-aspect multi-feature collaborative representation

ABSTRACT In this paper, a novel approach is proposed for Synthetic Aperture Radar (SAR) target classification based on multi-aspect multi-feature collaborative representation. Firstly, principal component analysis (PCA), wavelet and 2-dimensional slice Zernike moments (2DSZM) features are extracted from SAR images. Next, based on the strong correlation among the adjacent aspect SAR target images, we extend the basic collaborative representation classification (CRC) model to a neighbourhood multi-aspect CRC model. For each feature of the current test sample, neighbourhood multi-aspect test samples are regarded as the input to the model, then the temporary label is obtained for the current test sample under this feature. Finally, the temporary label is fused using the voting method to get the final classification result. The novelty of the proposed method is to improve the performance of target classification by integrating representation learning ability of different features and exploiting neighbourhood multi-aspect correlation. Experiments are investigated on the Moving and Stationary Target Acquisition and Recognition (MSTAR) dataset. The results show that the proposed algorithm can achieve a 98.52% overall accuracy and is superior to state-of-the-art methods for SAR target classification.

Application of deep learning methods in biological networks.

The increase in biological data and the formation of various biomolecule interaction databases enable us to obtain diverse biological networks. These biological networks provide a wealth of raw materials for further understanding of biological systems, the discovery of complex diseases and the search for therapeutic drugs. However, the increase in data also increases the difficulty of biological networks analysis. Therefore, algorithms that can handle large, heterogeneous and complex data are needed to better analyze the data of these network structures and mine their useful information. Deep learning is a branch of machine learning that extracts more abstract features from a larger set of training data. Through the establishment of an artificial neural network with a network hierarchy structure, deep learning can extract and screen the input information layer by layer and has representation learning ability. The improved deep learning algorithm can be used to process complex and heterogeneous graph data structures and is increasingly being applied to the mining of network data information. In this paper, we first introduce the used network data deep learning models. After words, we summarize the application of deep learning on biological networks. Finally, we discuss the future development prospects of this field.

An intelligent personalized web blog searching technique using fuzzy-based feedback recurrent neural network

In web information retrieval, every single user has a unique contextual objective when searching for information on web blogs. Based on the given query, the task of the web search engine is to fetch the most related information from the collection of web blogs. In order to enhance the searching ability, efficient semantic matching models, richer training, and evaluation resources are required. The conventional keyword-based search algorithms have a minimum efficiency in knowing users’ intentions compared to machine learning algorithms. Recently, neural networks are well-recognized in information retrieval due to the ability of vector representation learning. This paper proposes an adaptive fuzzy feedback recurrent neural network-based web blog searching technique which follows inverse filtering (IF) algorithm using Word2Vec representation. Initially, the user query is pre-processed, and then given to the IF for accelerating the search process. Inside the IF, the web blog content is labelled according to their blog information and stored in the hash table, and then relevant contents are extracted by TF–IDF; this result is given to the similarity estimation to obtain the similarity score. Finally, the proposed technique considers the feedback got from users and performs re-ranking by fuzzy-based RNN to achieve rich user intention satisfaction. The implementation is performed in Python, and statistical measures such as accuracy, precision, and recall are utilized for performance evaluation. The results demonstrate that the proposed technique has enhanced accuracy (94%) compared to conventional techniques, namely deep auto-encoder, deep neural networks, and artificial neural networks.

Review on Deep Learning Based Fault Diagnosis

The massive high-dimensional measurements accumulated by distributed control systems bring great computational and modeling complexity to the traditional fault diagnosis algorithms, which fail to take advantage of the higher-order for online estimation. In view of its powerful ability of representation learning, deep learning based fault diagnosis is extensively studied, both in academia and in industry, making intelligent process control more automated and effective. In this paper, deep learning based fault diagnosis is reviewed and summarized as four parts, i.e., stacked auto-encoder based fault diagnosis, deep belief network based fault diagnosis, convolutional neural network based fault diagnosis, and recurrent neural network based fault diagnosis. Furthermore, some necessity and potential trends, integrated innovation, + knowledge and information fusion, are discussed from the view of data preprocessing, network design and decision.

Exemplar-Supported Representation for Effective Class-Incremental Learning

Catastrophic forgetting is a key challenge for class-incremental learning with deep neural networks, where the performance decreases considerably while dealing with long sequences of new classes. To tackle this issue, in this paper, we propose a new exemplar-supported representation for incremental learning (ESRIL) approach that consists of three components. First, we use memory aware synapses (MAS) pre-trained on the ImageNet to retain the ability of robust representation learning and classification for old classes from the perspective of the model. Second, exemplar-based subspace clustering (ESC) is utilized to construct the exemplar set, which can keep the performance from various views of the data. Third, the nearest class multiple centroids (NCMC) is used as the classifier to save the training cost of the fully connected layer of MAS when the criterion is met. Intensive experiments and analyses are presented to show the influence of various backbone structures and the effectiveness of different components in our model. Experiments on several general-purpose and fine-grained image recognition datasets have fully demonstrated the efficacy of the proposed methodology.

Improved Deep Forest Mode for Detection of Fraudulent Online Transaction

As the rapid development of online transactions, transaction frauds have also emerged seriously. The fraud strategies are characterized by specialization, industrialization, concealment and scenes. Anti-fraud technologies face many challenges under the trend of new situations. In this paper, aiming at sample imbalance and strong concealment of online transactions, we enhance the original deep forest framework to propose a deep forest-based online transaction fraud detection model. Based on the BaggingBalance method we propose, we establish a global sample imbalance processing mechanism to deal with the problem of sample imbalance. In addition, the autoencoder model is introduced into the detection model to enhance the representation learning ability. Via the three-month real online transactions data of a China's bank, the experimental results show that, evaluating by the metric of precision and recall rate, the proposed model has a beyond 10 % improvement compared to the random forest model, and a beyond 5 % improvement compared to the original deep forest model.

Deep Gabor convolution network for person re-identification

Person re-identification is an import problem in computer vision fields and more and more deep neural network models have been developed for representation learning in this task due to their good performance. However, compared with hand-crafted feature representations, deep learned features cannot not be interpreted easily. To meet this demand, motivated by the Gabor filters’ good interpretability and the deep neural network models’ reliable learning ability, we propose a new convolution module for deep neural networks based on Gabor function (Gabor convolution). Compared with classical convolution module, every parameter in the proposed Gabor convolution kernel has a specific meaning while classical one has not. The Gabor convolution module has a good texture representation ability and is effective when it is embedded in the low layers of a network. Besides, in order to make the proposed Gabor module meaningful, a new loss function designed for this module is proposed as a regularizer of total loss function. By embedding the Gabor convolution module to the Resnet-50 network, we show that it has a good representation learning ability for person re-identification. And experiments on three widely used person re-identification datasets show favorable results compared with the state-of-the-arts.

Deep Neural Networks in the View of Ecological Holism

In the view point of ecological holism, deep neural networks can be view as a machine cognitive agent with overall multi-level nested structure, the representation learning and problem-solving ability of machine cognitive agent with multi-modal perception function is also the results from the perception and judgment of ambient situation environment. In the paper, a machine cognitive method based on ecological holism was deconstructed. Then, two core questions to characterize learnability of AlphaGo were answered, one is how to explain the characteristics of the hierarchical gradient and update parameters between different layers of networks? Two is how to understand the effective ability to choose the drop sampling at the strongest level of game of Go. These core controlling structural and functional roles proposed in this study may be further improved to a new type of artificial intelligence law of machine cognitive system.

Deep separable convolutional network for remaining useful life prediction of machinery

Abstract Deep learning is gaining attention in data-driven remaining useful life (RUL) prediction of machinery because of its powerful representation learning ability. With the help of deep learning techniques, the machine degradation information can be mined more sufficiently and some promising RUL prediction results have been achieved in several studies recently. These deep learning-based prognostics approaches, however, have the following weaknesses: 1) Their prediction performance largely depends on the hand-crafted feature design. 2) The correlations of different sensor data are not explicitly considered in representation learning. To overcome the above weaknesses, a new deep prognostics network named deep separable convolutional network (DSCN) is proposed in this paper for RUL prediction of machinery. In the proposed DSCN, the monitoring data acquired by different sensors are directly used as the inputs of the prognostics network. Then, a separable convolutional building block with a residual connection is built based on separable convolutions and squeeze and excitation operations. Through stacking multiple separable convolutional building blocks, the high-level representations are automatically learned from the input data. Finally, the RUL is estimated by feeding the learned representations into the fully-connected output layer. The proposed DSCN is validated using the vibration data from accelerated degradation tests of rolling element bearings and the public degradation simulation data of turbine engines, respectively. The experimental results show that the proposed DSCN is able to provide accurate RUL prediction results based on the raw multi-sensor data and is superior to some existing data-driven prognostics approaches.

Dynamic Deep Forest: An Ensemble Classification Method for Network Intrusion Detection

Network Intrusion Detection System (NIDS) is one of the key technologies to prevent network attacks and data leakage. In combination with machine learning, intrusion detection has achieved great progress in recent years. However, due to the diversity of intrusion types, the representation learning ability of the existing models is still deficient, which limits the further improvement of the detection performance. Meanwhile, with the increasing of model complexity, the training time becomes longer and longer. In this paper, we propose a Dynamic Deep Forest method for network intrusion detection. It uses cascade tree structure to strengthen the representation learning ability. At the same time, the training process is accelerated due to small-scale parameter fitting and dynamic level-growing strategy. The proposed Dynamic Deep Forest is a tree-based ensemble approach and consists of two parts. The first part, Multi-Grained Traversing, uses selectors to pick up features as complete as possible. The selectors are constructed dynamically so that the training process will stop as soon as the optimal feature combination is found. The second part, Cascade Forest, introduces level-by-level tree structures. It has fewer hyper-parameters and follows a dynamic level-growing strategy to reduce model complexity. In experiments, we evaluate our model on network intrusion dataset KDD’99. The results show that the Dynamic Deep Forest method obtains higher recall and precision through a short time of model training. Moreover, the Dynamic Deep Forest method has lower risk of misclassification, which is more stable and reliable in a real network environment.

Protein Function Prediction: From Traditional Classifier to Deep Learning.

Deep learning demonstrates greater competence over traditional machine learning techniques for many tasks. In last several years, deep learning has been applied to protein function prediction and a series of good achievements has been obtained. These findings extensively advanced our understanding of protein function. However, the accuracy of protein function prediction based upon deep learning still has yet to be improved. In article number 1900019, Issue 12, Zhang etal. construct DeepFunc, a deep learning framework using derived feature information of protein sequence and protein interactions network. They find that implementing DeepFunc for protein function prediction is more accurate than using DeepGO, a similar method reported previously. Meanwhile, they find that the method of combining multiple derived feature information in DeepFunc is much better than the method of using only single derived feature information. Due to its fully exploiting feature representation learning ability, deep learning with more derived feature information will enable it to be a promising method for solving more complicated protein function prediction problems and other bioinformatics challenges. Recent researches have provided some major insights into the value for using deep learning to protein function prediction problem.

An Improved Approach for Text Sentiment Classification Based on a Deep Neural Network via a Sentiment Attention Mechanism

Text sentiment analysis is an important but challenging task. Remarkable success has been achieved along with the wide application of deep learning methods, but deep learning methods dealing with text sentiment classification tasks cannot fully exploit sentiment linguistic knowledge, which hinders the development of text sentiment analysis. In this paper, we propose a sentiment-feature-enhanced deep neural network (SDNN) to address the problem by integrating sentiment linguistic knowledge into a deep neural network via a sentiment attention mechanism. Specifically, first we introduce a novel sentiment attention mechanism to help select the crucial sentiment-word-relevant context words by leveraging the sentiment lexicon in an attention mechanism, which bridges the gap between traditional sentiment linguistic knowledge and current popular deep learning methods. Second, we develop an improved deep neural network to extract sequential correlation information and text local features by combining bidirectional gated recurrent units with a convolutional neural network, which further enhances the ability of comprehensive text representation learning. With this design, the SDNN model can generate a powerful semantic representation of text to improve the performance of text sentiment classification tasks. Extensive experiments were conducted to evaluate the effectiveness of the proposed SDNN model on two real-world datasets with a binary-sentiment-label and a multi-sentiment-label. The experimental results demonstrated that the SDNN achieved substantially better performance than the strong competitors for text sentiment classification tasks.

Change Detection Based on Multi-Grained Cascade Forest and Multi-Scale Fusion for SAR Images

In this paper, a novel change detection approach based on multi-grained cascade forest (gcForest) and multi-scale fusion for synthetic aperture radar (SAR) images is proposed. It detects the changed and unchanged areas of the images by using the well-trained gcForest. Most existing change detection methods need to select the appropriate size of the image block. However, the single size image block only provides a part of the local information, and gcForest cannot achieve a good effect on the image representation learning ability. Therefore, the proposed approach chooses different sizes of image blocks as the input of gcForest, which can learn more image characteristics and reduce the influence of the local information of the image on the classification result as well. In addition, in order to improve the detection accuracy of those pixels whose gray value changes abruptly, the proposed approach combines gradient information of the difference image with the probability map obtained from the well-trained gcForest. Therefore, the image edge information can be enhanced and the accuracy of edge detection can be improved by extracting the image gradient information. Experiments on four data sets indicate that the proposed approach outperforms other state-of-the-art algorithms.

A Multi-Level-Denoising Autoencoder Approach for Wind Turbine Fault Detection

The effective fault detection of wind turbines (WTs) can greatly help to improve their availability and reduce their operation and maintenance costs. In this context, data-driven fault detection approaches have attracted a lot of interests due to the availability of a large amount of monitoring sensor data containing rich information related to health conditions of WTs. However, sensor data collected from WTs are naturally multivariate and highly nonlinear correlated with redundant information and significantly contaminated measurement noise, which makes the WT fault detection more challenging. To this end, this paper develops a multivariate data-driven fault detection (MDFD) framework based on a recently emerged neural network algorithm named denoising autoencoder (DAE). Instead of using a single fixed noise level in the traditional DAE, a novel multi-level-denoising autoencoder (MLD-AE) method is proposed to enhance the representation learning ability by designing different multi-level noise adding schemes. The proposed MLD-AE could better discover useful patterns at multiple corrupted scales and capture nonlinear dependencies from noisy multivariate sensor data, therefore robustly reconstruct the original signal with the preserved largest information. The proposed framework and method are evaluated on both simulated data from a generic 5 MW WT benchmark and SCADA data from a real wind farm. The results demonstrate that our proposed MLD-AE-based fault detection approach significantly outperforms traditional DAE, AE, and linear PCA approaches, which has great potentials for practical applications in the wind industry.

Adversarial Examples for Hamming Space Search.

Due to its strong representation learning ability and its facilitation of joint learning for representation and hash codes, deep learning-to-hash has achieved promising results and is becoming increasingly popular for the large-scale approximate nearest neighbor search. However, recent studies highlight the vulnerability of deep image classifiers to adversarial examples; this also introduces profound security concerns for deep retrieval systems. Accordingly, in order to study the robustness of modern deep hashing models to adversarial perturbations, we propose hash adversary generation (HAG), a novel method of crafting adversarial examples for Hamming space search. The main goal of HAG is to generate imperceptibly perturbed examples as queries, whose nearest neighbors from a targeted hashing model are semantically irrelevant to the original queries. Extensive experiments prove that HAG can successfully craft adversarial examples with small perturbations to mislead targeted hashing models. The transferability of these perturbations under a variety of settings is also verified. Moreover, by combining heterogeneous perturbations, we further provide a simple yet effective method of constructing adversarial examples for black-box attacks.