Efficient Deep Neural Network Research Articles

Intelligent approach to build a Deep Neural Network based IDS for cloud environment using combination of machine learning algorithms

The appealing features of Cloud Computing continue to fuel its adoption and its integration in many sectors such industry, governments, education and entertainment. Nevertheless, uploading sensitive data to public cloud storage services poses security risks such as integrity, availability and confidentiality to organizations. Moreover, the open and distributed (decentralized) structure of the cloud has resulted this class of computing, prone to cyber attackers and intruders. Thereby, it is imperative to develop an anomaly network intrusion system to detect and prevent both inside and outside assaults in cloud environment with high detection precision and low false warnings. In this work, we propose an intelligent approach to build automatically an efficient and effective Deep Neural Network (DNN) based anomaly Network IDS using a hybrid optimization framework (IGASAA) based on Improved Genetic Algorithm (IGA) and Simulated Annealing Algorithm (SAA). The IDS resulted is called “MLIDS” (Machine Learning based Intrusion Detection System). Genetic Algorithm (GA) is improved through optimization strategies, namely Parallel Processing and Fitness Value Hashing, which reduce execution time, convergence time and save processing power. Moreover, SAA was incorporated to IGA with the aim to optimize its heuristic search. Our approach consists of using IGASAA in order to search the optimal or near-optimal combination of most relevant values of the parameters included in construction of DNN based IDS or impacting its performance, like feature selection, data normalization, architecture of DNN, activation function, learning rate and Momentum term, which ensure high detection rate, high accuracy and low false alarm rate. For simulation and validation of the proposed method, CloudSim 4.0 simulator platform and three benchmark IDS datasets were used, namely CICIDS2017, NSL-KDD version 2015 and CIDDS-001. The implementation results of our model demonstrate its ability to detect intrusions with high detection accuracy and low false alarm rate, and indicate its superiority in comparison with state-of-the-art methods.

StressedNets: Efficient feature representations via stress-induced evolutionary synthesis of deep neural networks

The computational complexity of leveraging deep neural networks for extracting deep feature representations is a significant barrier to its widespread adoption. This is particularly a bottleneck for use in embedded devices and application such as self-driving cars. One promising strategy to addressing the complexity issue is the notion of evolutionary synthesis of deep neural networks. It was demonstrated that it successfully produces highly efficient deep neural networks while retaining modeling performance. Here, we further extend upon the evolutionary synthesis strategy for achieving efficient feature extraction. A stress-induced evolutionary synthesis framework is proposed where the stress signals are imposed upon the synapses of a deep neural network during training step. This process induces stress and steers the synthesis process towards the production of more efficient deep neural networks over successive generations. As a result, it improves model fidelity at a greater efficiency. Applying stress during the training phase helps a network to adopt itself for the changes which would happen at the evolution step. The proposed stress-induced evolutionary synthesis approach is evaluated on a variety of different deep neural network architectures (LeNet5, AlexNet, and YOLOv2), different tasks (object classification and object detection) to synthesize efficient StressedNets over multiple generations. Experimental results demonstrate the efficacy of the proposed framework to synthesize StressedNets with significant improvement in network architecture efficiency (e.g., 40 × for AlexNet and 33 × for YOLOv2). It is also shown the speed improvements by the synthesized networks (e.g., 5.5 × inference speed-up for YOLOv2 on an Nvidia Tegra X1 mobile processor).

A DEEP CONVOLUTIONAL AUTO-ENCODER WITH POOLING – UNPOOLING LAYERS IN CAFFE

This paper presents the development of several models of a deep convolutional auto-encoder in the Caffe deep learning framework and their experimental evaluation on the example of MNIST dataset. We have created five models of a convolutional auto-encoder which differ architecturally by the presence or absence of pooling and unpooling layers in the auto-encoder’s encoder and decoder parts. Our results show that the developed models provide very good results in dimensionality reduction and unsupervised clustering tasks, and small classification errors when we used the learned internal code as an input of a supervised linear classifier and multi-layer perceptron. The best results were provided by a model where the encoder part contains convolutional and pooling layers, followed by an analogous decoder part with deconvolution and unpooling layers without the use of switch variables in the decoder part. The paper also discusses practical details of the creation of a deep convolutional auto-encoder in the very popular Caffe deep learning framework. We believe that our approach and results presented in this paper could help other researchers to build efficient deep neural network architectures in the future.

Quality-aware dual-modal saliency detection via deep reinforcement learning

Incorporating various modes of information into the machine learning procedure is becoming a new trend And data from various source can provide more information than single one no matter they are heterogeneous or homogeneous. Existing deep learning based algorithms usually directly concatenate features from each domain to represent the input data. Seldom of them take the quality of data into consideration which is a key issue in related multimodal problems. In this paper, we propose an efficient quality-aware deep neural network to model the weight of data from each domain using deep reinforcement learning (DRL). Specifically, we take the weighting of each domain as a decision-making problem and teach an agent learn to interact with the environment. The agent can tune the weight of each domain through discrete action selection and obtain a positive reward if the saliency results are improved. The target of the agent is to achieve maximum rewards after finished its sequential action selection. We validate the proposed algorithms on dual-modal saliency detection in a coarse-to-fine way. The coarse saliency maps are generated from an encoder–decoder framework which is trained with content loss and adversarial loss. The final results can be obtained via adaptive weighting of maps from each domain. Extensive experiments on two kinds of salient object detection benchmarks all obtained a significant gain by fusing two-modalities with our proposed quality-aware deep neural network.

Active Transfer Learning Network: A Unified Deep Joint Spectral–Spatial Feature Learning Model for Hyperspectral Image Classification

Deep learning has recently attracted significant attention in the field of hyperspectral images (HSIs) classification. However, the construction of an efficient deep neural network (DNN) mostly relies on a large number of labeled samples being available. To address this problem, this paper proposes a unified deep network, combined with active transfer learning that can be well-trained for HSIs classification using only minimally labeled training data. More specifically, deep joint spectral-spatial feature is first extracted through hierarchical stacked sparse autoencoder (SSAE) networks. Active transfer learning is then exploited to transfer the pre-trained SSAE network and the limited training samples from the source domain to the target domain, where the SSAE network is subsequently fine-tuned using the limited labeled samples selected from both source and target domain by corresponding active learning strategies. The advantages of our proposed method are threefold: 1) the network can be effectively trained using only limited labeled samples with the help of novel active learning strategies; 2) the network is flexible and scalable enough to function across various transfer situations, including cross-dataset and intra-image; 3) the learned deep joint spectral-spatial feature representation is more generic and robust than many joint spectral-spatial feature representation. Extensive comparative evaluations demonstrate that our proposed method significantly outperforms many state-of-the-art approaches, including both traditional and deep network-based methods, on three popular datasets.

A Gaussian-Gaussian-Restricted-Boltzmann-Machine-based Deep Neural Network Technique for Photovoltaic System Generation Forecasting

Abstract This paper proposes a new Gaussian-Gaussian-Restricted-Boltzmann-Machine-based method for forecasting photovoltaic (PV) system generation forecasting. Although renewable energy such as PV system and wind power generation has been used to suppress greenhouse gases in the world, it has a drawback that weather conditions influence the generation output significantly. Thus, it is not easy to perform Economic Load Dispatch (ELD) and Unit Commitment in power systems smoothly. From a standpoint of power system operation, more accurate predication models are required to deal with predicted values of PV system generation. In this paper, an efficient Deep Neural Network (DNN) model with Gaussian Gaussian Restricted Boltzmann Machine is presented to predict one-step-ahead PV system generation output. The model is based on Restricted Boltzmann Machine as a feature extractor and MultiLayer Perceptron (MLP) as ANN. The effectiveness of the proposed method is demonstrated for real data of a PV system.

Efficient Semantic Segmentation Using Spatio-Channel Dilated Convolutions

There has been an increasing interest in reducing the computational cost to develop efficient deep convolutional neural networks (DCNN) for real-time semantic segmentation. In this paper, we introduce an efficient convolution method, Spatio-Channel dilated convolution (SCDC) which is composed of structured sparse kernels based on the principle of split-transform-merge. Specifically, it employs the kernels whose shapes are dilated, not only in spatial domain, but also in channel domain, using a channel sampling approach. Based on SCDC, we propose an efficient convolutional module named Efficient Spatio-Channel dilated convolution (ESC). With ESC modules, we further propose ESCNet based on ESPNet architecture which is one of the state-of-the-art real-time semantic segmentation network that can be easily deployed on edge devices. We evaluated our ESCNet on the Cityscapes dataset and obtained competitive results, with a good trade-off between accuracy and computational cost. The proposed ESCNet achieves 61.5 % mean intersection over union (IoU) with only 196 K network parameters, and processes high resolution images at a rate of 164 frames per second (FPS) on a standard Titan Xp GPU. Various experimental results show that our method is reasonably accurate, light, and fast.

Move, Attend and Predict: An attention-based neural model for people’s movement prediction

Predicting people’s next location has attracted the attention of both scientists and large Internet companies, for a variety of reasons. The analysis of location data collected from smart mobile devices paves the way for the improvement of current location based services and the rise of new business models, based on rich notifications related to the right prediction of users’ next location. Moreover, the so-called attention technique has been adopted in neural networks learning, aiming at considering alignments between different parts of the source training data. This article proposes the model Move, Attend and Predict (MAP) to predict a person’s future location based on his/her mobility pattern collected by a mobile device. This is achieved by means of a computationally efficient trainable deep neural network model. Our model essentially learns which time interval in the trajectory sequences are relevant regarding a specific location. In order to extract the meaningful representation from trajectories and time sequences, the embedding representation learning technique is used. Experimental results, obtained from tests conducted on two large real-life datasets, demonstrate that our model outperforms state-of-the-art models in terms of precision, recall and F1-score performance metrics.

Multimodal 2D+3D Facial Expression Recognition With Deep Fusion Convolutional Neural Network

This paper presents a novel and efficient deep fusion convolutional neural network (DF-CNN) for multimodal 2D+3D facial expression recognition (FER). DF-CNN comprises a feature extraction subnet, a feature fusion subnet, and a softmax layer. In particular, each textured three-dimensional (3D) face scan is represented as six types of 2D facial attribute maps (i.e., geometry map, three normal maps, curvature map, and texture map), all of which are jointly fed into DF-CNN for feature learning and fusion learning, resulting in a highly concentrated facial representation (32-dimensional). Expression prediction is performed by two ways: 1) learning linear support vector machine classifiers using the 32-dimensional fused deep features, or 2) directly performing softmax prediction using the six-dimensional expression probability vectors. Different from existing 3D FER methods, DF-CNN combines feature learning and fusion learning into a single end-to-end training framework. To demonstrate the effectiveness of DF-CNN, we conducted comprehensive experiments to compare the performance of DFCNN with handcrafted features, pre-trained deep features, finetuned deep features, and state-of-the-art methods on three 3D face datasets (i.e., BU-3DFE Subset I, BU-3DFE Subset II, and Bosphorus Subset). In all cases, DF-CNN consistently achieved the best results. To the best of our knowledge, this is the first work of introducing deep CNN to 3D FER and deep learning-based featurelevel fusion for multimodal 2D+3D FER.

Deep Learning with Darwin: Evolutionary Synthesis of Deep Neural Networks

Taking inspiration from biological evolution, we explore the idea of “Can deep neural networks evolve naturally over successive generations into highly efficient deep neural networks?” by introducing the notion of synthesizing new highly efficient, yet powerful deep neural networks over successive generations via an evolutionary process from ancestor deep neural networks. The architectural traits of ancestor deep neural networks are encoded using synaptic probability models, which can be viewed as the ‘DNA’ of these networks. New descendant networks with differing network architectures are synthesized based on these synaptic probability models from the ancestor networks and computational environmental factor models, in a random manner to mimic heredity, natural selection, and random mutation. These offspring networks are then trained into fully functional networks, like one would train a newborn, and have more efficient, more diverse network architectures than their ancestor networks, while achieving powerful modeling capabilities. Experimental results for the task of visual saliency demonstrated that the synthesized ‘evolved’ offspring networks can achieve state-of-the-art performance while having network architectures that are significantly more efficient (with a staggering $$\sim $$ 48-fold decrease in synapses by the fourth generation) compared to the original ancestor network.

Polyploidism in Deep Neural Networks: m-Parent Evolutionary Synthesis of Deep Neural Networks in Varying Population Sizes

Evolutionary deep intelligence was recently proposed to organicallyproduce highly efficient deep neural network architecturesover successive generations. Thus far, current evolutionary synthesisprocesses are based on asexual reproduction, i.e., offspringneural networks are synthesized stochastically from a single parentnetwork. In this study, we investigate the effects of m-parentsexual evolutionary synthesis (m = 1, 2, 3, 5) in combination withvarying population sizes of three, five, and eight synthesized networksper generation. Experimental results were obtained usinga 10% subset of the MNIST handwritten digits dataset, and showthat increasing the number of parent networks results in improvedarchitectural efficiency of the synthesized networks (approximately150x synaptic efficiency and approximately 42–49x cluster efficiency)while resulting in only a 2–3% drop in testing accuracy.

StochasticNet in StochasticNet

Deep neural networks have been shown to outperform conventionalstate-of-the-art approaches in several structured predictionapplications. While high-performance computing devices such asGPUs has made developing very powerful deep neural networkspossible, it is not feasible to run these networks on low-cost, lowpowercomputing devices such as embedded CPUs or even embeddedGPUs. As such, there has been a lot of recent interestto produce efficient deep neural network architectures that can berun on small computing devices. Motivated by this, the idea ofStochasticNets was introduced, where deep neural networks areformed by leveraging random graph theory. It has been shownthat StochasticNet can form new networks with 2X or 3X architecturalefficiency while maintaining modeling accuracy. Motivated bythese promising results, here we investigate the idea of Stochastic-Net in StochasticNet (SiS), where highly-efficient deep neural networkswith Network in Network (NiN) architectures are formed ina stochastic manner. Such networks have an intertwining structurecomposed of convolutional layers and micro neural networksto boost the modeling accuracy. The experimental results showthat SiS can form deep neural networks with NiN architectures thathave 4X greater architectural efficiency with only a 2% dropin accuracy for the CIFAR10 dataset. The results are even morepromising for the SVHN dataset, where SiS formed deep neuralnetworks with NiN architectures that have 11.5X greater architecturalefficiency with only a 1% decrease in modeling accuracy.

Latte: a language, compiler, and runtime for elegant and efficient deep neural networks

Deep neural networks (DNNs) have undergone a surge in popularity with consistent advances in the state of the art for tasks including image recognition, natural language processing, and speech recognition. The computationally expensive nature of these networks has led to the proliferation of implementations that sacrifice abstraction for high performance. In this paper, we present Latte, a domain-specific language for DNNs that provides a natural abstraction for specifying new layers without sacrificing performance. Users of Latte express DNNs as ensembles of neurons with connections between them. The Latte compiler synthesizes a program based on the user specification, applies a suite of domain-specific and general optimizations, and emits efficient machine code for heterogeneous architectures. Latte also includes a communication runtime for distributed memory data-parallelism. Using networks described using Latte, we demonstrate 3-6x speedup over Caffe (C++/MKL) on the three state-of-the-art ImageNet models executing on an Intel Xeon E5-2699 v3 x86 CPU.