Extract High-level Representations Research Articles

Construction of a network intrusion detection system based on a convolutional neural network and a bidirectional gated recurrent unit with attention mechanism

The object of this study is the process of recognizing intrusions in computer networks. Network intrusion detection systems (NIDS) have become an important area of research as they are used to protect computer systems from hacker attacks. Deep learning is becoming increasingly popular for detecting and classifying malicious network traffic, including for building NIDS. In this paper, we propose a network intrusion detection model CNN-BiGRU-Attention based on a time-based approach to deep learning using the attention mechanism. The main goal of the study is to build an effective combined deep learning model that can detect various network cyber threats. A 1D convolutional neural network is implemented to extract high-level representations of intrusion information features. A bidirectional gated recurrent unit (BiGRU) with an attention mechanism for traffic data classification has been designed. The attention mechanism plays a key role in the constructed model as it allows the system to focus only on important aspects of network traffic and allows the model to adapt to new types of threats. The results of the study show that using a combination of CNN and BiGRU with the attention mechanism speeds up and improves the process of classifying network attacks. On the NSL-KDD and UNSW-NB15 training datasets, the model shows an accuracy of 99.81 % and 97.80 %. On the NSL-KDD and UNSW-NB15 test datasets, the model demonstrates 82.16 % and 97.72 % accuracy. The proposed NIDS model will be considered for implementation in a real-time corporate network security system. In general, the results of the study provide a new perspective on improving the performance of NIDS and are quite relevant in terms of using attention mechanisms to classify network traffic

Improving the accuracy of diagnostic predictions for power transformers by employing a hybrid approach combining SMOTE and DNN

In power transformers, accurate diagnosis prediction plays a crucial role in ensuring the reliability and efficiency of electrical systems. This paper presents an enhanced diagnostic prediction approach for power transformers by combining the Synthetic Minority Over-sampling Technique (SMOTE) and Deep Neural Networks (DNN). Our approach addresses the challenge of imbalanced data in power transformer diagnosis prediction by leveraging SMOTE to generate synthetic samples and balance the dataset. By effectively handling rare events through the augmentation of minority class instances, our predictive model becomes more robust. Additionally, using deep neural networks enables the exploration of complex patterns and relationships within the transformer dataset. The DNN architecture integrates multiple layers of learnable features, enabling the prediction model to extract high-level representations and achieve improved accuracy and performance. To evaluate the effectiveness of our proposed approach, we conducted extensive experiments using real-world power transformer datasets. Comparative analysis of the results demonstrates that the combination of SMOTE and DNN outperforms traditional classification techniques applied to imbalanced datasets. The proposed model achieves a higher training accuracy of 98.22 % and a testing accuracy of 94.6 % and mitigates the risks associated with misdiagnosis, contributing to power systems' overall reliability and maintenance efficiency.

Generalizing Graph Neural Networks on Out-of-Distribution Graphs.

Graph Neural Networks (GNNs) are proposed without considering the agnostic distribution shifts between training graphs and testing graphs, inducing the degeneration of the generalization ability of GNNs in Out-Of-Distribution (OOD) settings. The fundamental reason for such degeneration is that most GNNs are developed based on the I.I.D hypothesis. In such a setting, GNNs tend to exploit subtle statistical correlations existing in the training set for predictions, even though it is a spurious correlation. This learning mechanism inherits from the common characteristics of machine learning approaches. However, such spurious correlations may change in the wild testing environments, leading to the failure of GNNs. Therefore, eliminating the impact of spurious correlations is crucial for stable GNN models. To this end, in this paper, we argue that the spurious correlation exists among subgraph-level units and analyze the degeneration of GNN in causal view. Based on the causal view analysis, we propose a general causal representation framework for stable GNN, called StableGNN. The main idea of this framework is to extract high-level representations from raw graph data first and resort to the distinguishing ability of causal inference to help the model get rid of spurious correlations. Particularly, to extract meaningful high-level representations, we exploit a differentiable graph pooling layer to extract subgraph-based representations by an end-to-end manner. Furthermore, inspired by the confounder balancing techniques from causal inference, based on the learned high-level representations, we propose a causal variable distinguishing regularizer to correct the biased training distribution by learning a set of sample weights. Hence, GNNs would concentrate more on the true connection between discriminative substructures and labels. Extensive experiments are conducted on both synthetic datasets with various distribution shift degrees and eight real-world OOD graph datasets. The results well verify that the proposed model StableGNN not only outperforms the state-of-the-arts but also provides a flexible framework to enhance existing GNNs. In addition, the interpretability experiments validate that StableGNN could leverage causal structures for predictions.

Hybrid data augmentation and deep attention-based dilated convolutional-recurrent neural networks for speech emotion recognition

Recently, speech emotion recognition (SER) has become an active research area in speech processing, particularly with the advent of deep learning (DL). Numerous DL-based methods have been proposed for SER. However, most of the existing DL-based models are complex and require a large amounts of data to achieve a good performance. In this study, a new framework of deep attention-based dilated convolutional-recurrent neural networks coupled with a hybrid data augmentation method was proposed for addressing SER tasks. The hybrid data augmentation method constitutes an upsampling technique for generating more speech data samples based on the traditional and generative adversarial network approaches. By leveraging both convolutional and recurrent neural networks in a dilated form along with an attention mechanism, the proposed DL framework can extract high-level representations from three-dimensional log Mel spectrogram features. Dilated convolutional neural networks acquire larger receptive fields, whereas dilated recurrent neural networks overcome complex dependencies as well as the vanishing and exploding gradient issues. Furthermore, the loss functions are reconfigured by combining the SoftMax loss and the center-based losses to classify various emotional states. The proposed framework was implemented using the Python programming language and the TensorFlow deep learning library. To validate the proposed framework, the EmoDB and ERC benchmark datasets, which are imbalanced and/or small datasets, were employed. The experimental results indicate that the proposed framework outperforms other related state-of-the-art methods, yielding the highest unweighted recall rates of 88.03 ± 1.39 (%) and 66.56 ± 0.67 (%) for the EmoDB and ERC datasets, respectively.

A survey of field programmable gate array (FPGA)-based graph convolutional neural network accelerators: challenges and opportunities.

Graph convolutional networks (GCNs) based on convolutional operations have been developed recently to extract high-level representations from graph data. They have shown advantages in many critical applications, such as recommendation system, natural language processing, and prediction of chemical reactivity. The problem for the GCN is that its target applications generally pose stringent constraints on latency and energy efficiency. Several studies have demonstrated that field programmable gate array (FPGA)-based GCNs accelerators, which balance high performance and low power consumption, can continue to achieve orders-of-magnitude improvements in the inference of GCNs models. However, there still are many challenges in customizing FPGA-based accelerators for GCNs. It is necessary to sort out the current solutions to these challenges for further research. For this purpose, we first summarize the four challenges in FPGA-based GCNs accelerators. Then we introduce the process of the typical GNN algorithm and several examples of representative GCNs. Next, we review the FPGA-based GCNs accelerators in recent years and introduce their design details according to different challenges. Moreover, we compare the key metrics of these accelerators, including resource utilization, performance, and power consumption. Finally, we anticipate the future challenges and directions for FPGA-based GCNs accelerators: algorithm and hardware co-design, efficient task scheduling, higher generality, and faster development.

Applying Graph Convolution Networks to Recommender Systems based on graph topology

The recommender systems are widely used in online applications to suggest products to the potential users. The main aim of recommender system is to produce meaningful recommendation to a potential user by monitoring user’s purchasing habits, history, and useful information. Recently, graph representation learning methods based on node embedding have drawn attention in Recommender systems such as Graph Convolutional Networks (GCNs) that is powerful method for collaborative filtering. The GCN performs neighborhood aggregation mechanism to extract high level representation for both user and items. In this paper, we propose a recommendation algorithm based on node similarity convolutional matrices with topological property in GCNs where the linkage measure is illustrated as a bipartite graph. The experiments indicate the necessity of capturing user–item graph structure in recommendation. The experimental results show that node similarity-based convolution matrices and GCN-based embeddings significantly improve the prediction accuracy in recommender systems compared to state-of-art approaches.

Sensing the Sounds of Silence: A Pilot Study on the Detection of Model Mice of Autism Spectrum Disorder from Ultrasonic Vocalisations.

Studying the animal models of human neuropsychiatric disorders can facilitate the understanding of mechanisms of symptoms both physiologically and genetically. Previous studies have shown that ultrasonic vocalisations (USVs) of mice might be efficient markers to distinguish the wild type group and the model of autism spectrum disorder (mASD). Nevertheless, in-depth analysis of these 'silence' sounds by leveraging the power of advanced computer audition technologies (e. g., deep learning) is limited. To this end, we propose a pilot study on using a large-scale pre-trained audio neural network to extract high-level representations from the USVs of mice for the task on detection of mASD. Experiments have shown a best result reaching an unweighted average recall of 79.2 % for the binary classification task in a rigorous subject-independent scenario. To the best of our knowledge, this is the first time to analyse the sounds that cannot be heard by human beings for the detection of mASD mice. The novel findings can be significant to motivate future works with according means on studying animal models of human patients.

A parallel attention-augmented bilinear network for early magnetic resonance imaging-based diagnosis of Alzheimer's disease.

Structural magnetic resonance imaging (sMRI) can capture the spatial patterns of brain atrophy in Alzheimer's disease (AD) and incipient dementia. Recently, many sMRI‐based deep learning methods have been developed for AD diagnosis. Some of these methods utilize neural networks to extract high‐level representations on the basis of handcrafted features, while others attempt to learn useful features from brain regions proposed by a separate module. However, these methods require considerable manual engineering. Their stepwise training procedures would introduce cascading errors. Here, we propose the parallel attention‐augmented bilinear network, a novel deep learning framework for AD diagnosis. Based on a 3D convolutional neural network, the framework directly learns both global and local features from sMRI scans without any prior knowledge. The framework is lightweight and suitable for end‐to‐end training. We evaluate the framework on two public datasets (ADNI‐1 and ADNI‐2) containing 1,340 subjects. On both the AD classification and mild cognitive impairment conversion prediction tasks, our framework achieves competitive results. Furthermore, we generate heat maps that highlight discriminative areas for visual interpretation. Experiments demonstrate the effectiveness of the proposed framework when medical priors are unavailable or the computing resources are limited. The proposed framework is general for 3D medical image analysis with both efficiency and interpretability.

Towards multilingual end‐to‐end speech recognition for air traffic control

In this work, an end-to-end framework is proposed to achieve multilingual automatic speech recognition (ASR) in air traffic control (ATC) systems. Considering the standard ATC procedure, a recurrent neural network (RNN) based framework is selected to mine the temporal dependencies among speech frames. Facing the distributed feature space caused by the radio transmission, a hybrid feature embedding block is designed to extract high-level representations, in which multiple convolutional neural networks are designed to accommodate different frequency and temporal resolutions. The residual mechanism is performed on the RNN layers to improve the trainability and the convergence. To integrate the multilingual ASR into a single model and relieve the class imbalance, a special vocabulary is designed to unify the pronunciation of the vocabulary in Chinese and English, i.e., pronunciation-oriented vocabulary. The proposed model is optimized by the connectionist temporal classification loss and is validated on a real-world speech corpus (ATCSpeech). A character error rate of 4.4% and 5.9% is achieved for Chinese and English speech, respectively, which outperforms other popular approaches. Most importantly, the proposed approach achieves the multilingual ASR task in an end-to-end manner with considerable high performance.

Analyzing Multimodal Sentiment Via Acoustic- and Visual-LSTM With Channel-Aware Temporal Convolution Network

The emotion of human is always expressed in a multimodal perspective. Analyzing multimodal human sentiment remains challenging due to the difficulties of the interpretation in inter-modality dynamics. Mainstream multimodal learning architectures tend to design various fusion strategies to learn inter-modality interactions, which barely consider the fact that the language modality is far more important than the acoustic and visual modalities. In contrast, we learn inter-modality dynamics in a different perspective via acoustic- and visual-LSTMs where language features play dominant role. Specifically, inside each LSTM variant, a well-designed gating mechanism is introduced to enhance the language representation via the corresponding auxiliary modality. Furthermore, in the unimodal representation learning stage, instead of using RNNs, we introduce `channel-aware' temporal convolution network to extract high-level representations for each modality to explore both temporal and channel-wise interdependencies. Extensive experiments demonstrate that our approach achieves very competitive performance compared to the state-of-the-art methods on three widely-used benchmarks for multimodal sentiment analysis and emotion recognition.

Multimodal Deep Learning Framework for Image Popularity Prediction on Social Media

Billions of photos are uploaded to the Web daily through various types of social networks. Some of these images receive millions of views and become popular, whereas others remain completely unnoticed. This raises the problem of predicting image popularity on social media. The popularity of an image can be affected by several factors, such as visual content, aesthetic quality, user, post metadata, and time. Thus, considering all these factors is essential for accurately predicting image popularity. In addition, the efficiency of the predictive model also plays a crucial role. In this study, motivated by multimodal learning, which uses information from various modalities, and the current success of convolutional neural networks (CNNs) in various fields, we propose a deep learning model, called visual-social CNN (VSCNN), which predicts the popularity of a posted image by incorporating various types of visual and social features into a unified network model. VSCNN first learns to extract high-level representations from the input visual and social features by utilizing two individual CNNs. The outputs of these two networks are then fused into a joint network to estimate the popularity score in the output layer. We assess the performance of the proposed method by conducting extensive experiments on a data set of approximately 432K images posted on Flickr. The simulation results demonstrate that the proposed VSCNN model significantly outperforms state-of-the-art models, with a relative improvement of greater than 2.33%, 7.59%, and 14.16% in terms of Spearman's Rho, mean absolute error, and mean-squared error, respectively.

Offline mobile diagnosis system for citrus pests and diseases using deep compression neural network

This study presents an offline mobile diagnosis system for citrus pests and diseases by compression convolutional neural network. Recently, with the growth of labelled data, the deep neural network incites the revolutionary change with a quantum leap in various fields. Benefiting from the backpropagation method, the proper network structure can automatically extract high-level representations and find corresponding labels. The authors made use of the advantages of the deep neural network to design an android application, which can be installed in any stand-alone devices to instantaneously identify the citrus pests and diseases. The proposed diagnosis system has three characteristics: low cost, low latency and high accuracy. These characteristics contribute to make the professional offline prediction for avoiding further economic loss caused by disease spreading. In order to validate the proposed system, the authors conducted thorough evaluations on two data sets, `citrus pests and diseases', CIFAR, which show the superiority of the proposed approach in terms of the accuracy and the number of model parameters.

A novel gas/liquid two-phase flow imaging method through electrical resistance tomography with DDELM-AE sparse dictionary

PurposeThe purpose of this paper is to explore gas/liquid two-phase flow is widely existed in industrial fields, especially in chemical engineering. Electrical resistance tomography (ERT) is considered to be one of the most promising techniques to monitor the transient flow process because of its advantages such as fast respond speed and cross-section imaging. However, maintaining high resolution in space together with low cost is still challenging for two-phase flow imaging because of the ill-conditioning of ERT inverse problem.Design/methodology/approachIn this paper, a sparse reconstruction (SR) method based on the learned dictionary has been proposed for ERT, to accurately monitor the transient flow process of gas/liquid two-phase flow in a pipeline. The high-level representation of the conductivity distributions for typical flow regimes can be extracted based on denoising the deep extreme learning machine (DDELM) model, which is used as prior information for dictionary learning.FindingsThe results from simulation and dynamic experiments indicate that the proposed algorithm efficiently improves the quality of reconstructed images as compared to some typical algorithms such as Landweber and SR-discrete fourier transformation/discrete cosine transformation. Furthermore, the SR-DDELM has also used to estimate the important parameters of the chemical process, a case in point is the volume flow rate. Therefore, the SR-DDELM is considered an ideal candidate for online monitor the gas/liquid two-phase flow.Originality/valueThis paper fulfills a novel approach to effectively monitor the gas/liquid two-phase flow in pipelines. One deep learning model and one adaptive dictionary are trained via the same prior conductivity, respectively. The model is used to extract high-level representation. The dictionary is used to represent the features of the flow process. SR and extraction of high-level representation are performed iteratively. The new method can obviously improve the monitoring accuracy and save calculation time.

Network Embedding-Aware Point-of-Interest Recommendation in Location-Based Social Networks

As one of the important techniques to explore unknown places for users, the methods that are proposed for point-of-interest (POI) recommendation have been widely studied in recent years. Compared with traditional recommendation problems, POI recommendations are suffering from more challenges, such as the cold-start and one-class collaborative filtering problems. Many existing studies have focused on how to overcome these challenges by exploiting different types of contexts (e.g., social and geographical information). However, most of these methods only model these contexts as regularization terms, and the deep information hidden in the network structure has not been fully exploited. On the other hand, neural network-based embedding methods have shown its power in many recommendation tasks with its ability to extract high-level representations from raw data. According to the above observations, to well utilize the network information, a neural network-based embedding method (node2vec) is first exploited to learn the user and POI representations from a social network and a predefined location network, respectively. To deal with the implicit feedback, a pair-wise ranking-based method is then introduced. Finally, by regarding the pretrained network representations as the priors of the latent feature factors, an embedding-based POI recommendation method is proposed. As this method consists of an embedding model and a collaborative filtering model, when the training data are absent, the predictions will mainly be generated by the extracted embeddings. In other cases, this method will learn the user and POI factors from these two components. Experiments on two real-world datasets demonstrate the importance of the network embeddings and the effectiveness of our proposed method.

A weighted auto regressive LSTM based approach for chemical processes modeling

Data-driven methods have been regarded as effective methods for modeling in chemical processes. However, with the increasing complexity of chemical processes in spatial domain and time domain, how to extract meaningful features and build corresponding models are keys for accurate modeling tasks. To retain temporal features of original inputs, a Recurrent Denosing Autoencoder (RDAE) is built to extract meaningful features and reduce input dimension in the spatial domain, and Cumulative Percent Variance (CPV) is introduced to decide the number of extracted features. Considering correction and prior knowledge effects of real history outputs, a Weighted Auto Regressive Long Short Term Memory (WAR-LSTM) structure is proposed as the basic cell, then multiple WAR-LSTMs are stacked as deep WAR-LSTM to extract high level representations from multi variables. Hence, spatial and temporal information are sufficiently used both in extracting features and building model. The benchmarked Tennessee Eastman process data and real process data from a Fluid Catalytic Cracking (FCC) unit verify the effectiveness of our work.

Multi-input 1-dimensional deep belief network: action and activity recognition as case study

This paper develops a new variation of deep belief networks which is evaluated on the basis of supervised classification of human actions and activities. The proposed multi-input 1-dimensional deep belief network (M1DBN) can work based on three inputs which contain different information structures. The multi input features helps M1DBN automatically search the solution space more accurately and extract high-level representations more efficiently. M1DBN utilizes three inputs to provide spatial, short-term and long-term information for the action and activity recognition. Spatial information can distinguish between human movements which have a high inter-class variation. However, regarding similar inter-class variations, a temporal description is used. Short-term and long-term inputs learn actions or activities for short and long video intervals, respectively. Experimental results show the superiority of this approach over state-of-the-art methods on KTH (97.04%), HMDB51 (67.19%), UCI-HAD (97.16%) and Skoda (93.28%) datasets. Also, a detailed explanation of learning, training and test procedures used in M1DBN are available.

Exploiting Pre-Trained Network Embeddings for Recommendations in Social Networks

Recommender systems as one of the most efficient information filtering techniques have been widely studied in recent years. However, traditional recommender systems only utilize user-item rating matrix for recommendations, and the social connections and item sequential patterns are ignored. But in our real life, we always turn to our friends for recommendations, and often select the items that have similar sequential patterns. In order to overcome these challenges, many studies have taken social connections and sequential information into account to enhance recommender systems. Although these existing studies have achieved good results, most of them regard social influence and sequential information as regularization terms, and the deep structure hidden in social networks and rating patterns has not been fully explored. On the other hand, neural network based embedding methods have shown their power in many recommendation tasks with their ability to extract high-level representations from raw data. Motivated by the above observations, we take the advantage of network embedding techniques and propose an embedding-based recommendation method, which is composed of the embedding model and the collaborative filtering model. Specifically, to exploit the deep structure hidden in social networks and rating patterns, a neural network based embedding model is first pre-trained, where the external user and item representations are extracted. Then, we incorporate these extracted factors into a collaborative filtering model by fusing them with latent factors linearly, where our method not only can leverage the external information to enhance recommendation, but also can exploit the advantage of collaborative filtering techniques. Experimental results on two real-world datasets demonstrate the effectiveness of our proposed method and the importance of these external extracted factors.

Contractive Slab and Spike Convolutional Deep Boltzmann Machine

Abstract Deep unsupervised learning for robust and effective feature extractions from high-resolution images still keeps greatly challenging. Although Deep Boltzmann Machine (DBM) has demonstrated the impressive capacity of feature extractions, there is still a lot of potential for improvement in scaling such model to full-sized images, the robustness, and the quality of the learned features. In this paper, we propose a Contractive Slab and Spike Convolutional Deep Boltzmann Machine in order to settle these issues. First, the proposed model extends convolution operation to the DBM in order to deal with real-size images. Second, we induce element-wise multiplication between real-valued slab hidden units and binary spike hidden units in order to enhance the quality of feature extraction in the receptive field. Then, we add the frobenius norm of the jacobian of the features as a regularization term to the maximum likelihood function in order to enhance the robustness of the features during training. The proposed regularization term results in a localized space contraction, which in turn obtains robust features on the hidden layer. Last, we use a new block-Contractive Slab and Spike Convolutional Restricted Boltzmann Machine in order to pretrain the proposed model. The proposed deep model shows the better capacity to extract high-level representations. The results on various visual tasks demonstrate the proposed model can achieve the improved performance over several state-of-the-art methods.

Compact Multiview Representation of Documents Based on the Total Variability Space

Mapping text documents in an LDA-based topic-space is a classical way to extract high-level representation of text documents. Unfortunately, LDA is highly sensitive to hyper-parameters related to the number of classes, or word and topic distribution, and there is no systematic way to pre-estimate optimal configurations. Moreover, various hyper-parameter configurations offer complementary views on the document. In this paper, we propose a method based on a two-step process that, first, expands the representation space by using a set of topic spaces and, second, compacts the representation space by removing poorly relevant dimensions. These two steps are based respectively on multi-view LDA-based representation spaces and factor-analysis models. This model provides a view-independent representation of documents while extracting complementary information from a massive multi-view representation. Experiments are conducted on the DECODA conversation corpus and the Reuters-21578 textual dataset. Results show the efficiency of the proposed multiview compact representation paradigm. The proposed categorization system reaches an accuracy of 86.5% with automatic transcriptions of conversations from DECODA corpus and a Macro-F1 of 80% during a classification task of the well-known Reuters-21578 corpus, with a significant gain compared to the baseline (best single topic space configuration), as well as methods and document representations previously studied.