Deep Learning Scheme Research Articles

Deep Learning: The Frontier for Distributed Attack Detection in Fog-to-Things Computing

The increase in the number and diversity of smart objects has raised substantial cybersecurity challenges due to the recent exponential rise in the occurrence and sophistication of attacks. Although cloud computing has transformed the world of business in a dramatic way, its centralization hammers the application of distributed services such as security mechanisms for IoT applications. The new and emerging IoT applications require novel cybersecurity controls, models, and decisions distributed at the edge of the network. Despite the success of the existing cryptographic solutions in the traditional Internet, factors such as system development flaws, increased attack surfaces, and hacking skills have proven the inevitability of detection mechanisms. The traditional approaches such as classical machine-learning-based attack detection mechanisms have been successful in the last decades, but it has already been proven that they have low accuracy and less scalability for cyber-attack detection in massively distributed nodes such as IoT. The proliferation of deep learning and hardware technology advancement could pave a way to detecting the current level of sophistication of cyber-attacks in edge networks. The application of deep networks has already been successful in big data areas, and this indicates that fog-tothings computing can be the ultimate beneficiary of the approach for attack detection because a massive amount of data produced by IoT devices enable deep models to learn better than shallow algorithms. In this article, we propose a novel distributed deep learning scheme of cyber-attack detection in fog-to-things computing. Our experiments show that deep models are superior to shallow models in detection accuracy, false alarm rate, and scalability.

Deep Fusion Feature Learning Network for MI-EEG Classification

Brain–computer interfaces (BCIs) are used to provide a direct communication between the human brain and the external devices, such as wheelchairs and intelligent apparatus, by interpreting the electroencephalograph (EEG) signals. Recently, motor imagery EEG (MI-EEG) has become an active research field where a subject’s active intent can be detected. The accurate decoding of MI-EEG signals is essential for effective BCI systems but also very challenging due to the lack of informative correlation between the signals and the brain activities. To improve the precision performance of a BCI system, accurate feature discrimination from input signals and proper classification are necessary. However, the traditional deep learning scheme is failed to generate spatio-temporal representation simultaneously and capture the dynamic correlation for an MI-EEG sequence. To address this problem, we propose a long short-term memory network combined with a spatial convolutional network that concurrently learns spatial information and temporal correlations from raw MI-EEG signals. In addition, spectral representations of EEG signals are obtained via a discrete wavelet transformation decomposition. In order to achieve even higher learning rates and less demanding initialization, we employ a batch normalization method before training and recognition. Various experiments have been performed to evaluate the performance of the proposed deep learning architectures. Results indicate a high level of accuracy over both the public data set and the local data set. Our method can also serve as a useful and robust model for multi-task classification and subject-independent movement class decoder across many different methods.

A novel double deep ELMs ensemble system for time series forecasting

Extreme Learning Machine (ELM) has proved to be well suited to different kinds of classification and regression problems. However, failing to seek deep representation of raw data completely brought by shallow architecture has made a plenty of research work stagnant, when ELM was chosen as the basic model. Recent years, deep ELM models like Hierarchical ELM (H-ELM), deep representations learning via ELM (Dr-ELM) have been proposed to be applied in multiple applications in machine learning. In this paper, a novel double deep ELMs ensemble system (DD-ELMs-ES) is proposed to focus on the problem of time series forecasting. In the proposed system, besides H-ELM and Dr-ELM are utilized as the basic models, a novel Constrained H-ELM (CH-ELM) is presented and serves as another basic model as well. CH-ELM intends to constrain the hidden neurons’ input connection weights, so that they could be consistent with the directions of sample vectors. Whats more, a self-adaptive ReTSP-Trend pruning technique is proposed to implement ensemble pruning in DD-ELMs-ES. Benefited from the merits of combining deep learning scheme with ensemble pruning paradigm, in the empirical results, DD-ELMs-ES demonstrates better generalization performance than the basic deep ELM models and some other state-of-the-art algorithms in tackling with time series forecasting tasks.

A deep learning scheme for mental workload classification based on restricted Boltzmann machines

The mental workload (MWL) classification is a critical problem for quantitative assessment and analysis of operator functional state in many safety-critical situations with indispensable human–machine cooperation. The MWL can be measured by psychophysiological signals. In this work, we propose a novel restricted Boltzmann machine (RBM) architecture for MWL classification. In relation to this architecture, we examine two main issues: the optimal structure of RBM and selection of the most important EEG channels (electrodes) for MWL classification. The trial-and-error and entropy-based pruning methods are compared for the RBM structure identification. The degree of importance of EEG channels is calculated from the weights in a well-trained network in order to select the most relevant channels for classification task. Extensive comparative results showed that the selected EEG channels lead to accurate MWL classification across subjects.

Blind Deep S3D Image Quality Evaluation via Local to Global Feature Aggregation.

Previously, no-reference (NR) stereoscopic 3D (S3D) image quality assessment (IQA) algorithms have been limited to the extraction of reliable hand-crafted features based on an understanding of the insufficiently revealed human visual system or natural scene statistics. Furthermore, compared with full-reference (FR) S3D IQA metrics, it is difficult to achieve competitive quality score predictions using the extracted features, which are not optimized with respect to human opinion. To cope with this limitation of the conventional approach, we introduce a novel deep learning scheme for NR S3D IQA in terms of local to global feature aggregation. A deep convolutional neural network (CNN) model is trained in a supervised manner through two-step regression. First, to overcome the lack of training data, local patch-based CNNs are modeled, and the FR S3D IQA metric is used to approximate a reference ground-truth for training the CNNs. The automatically extracted local abstractions are aggregated into global features by inserting an aggregation layer in the deep structure. The locally trained model parameters are then updated iteratively using supervised global labeling, i.e., subjective mean opinion score (MOS). In particular, the proposed deep NR S3D image quality evaluator does not estimate the depth from a pair of S3D images. The S3D image quality scores predicted by the proposed method represent a significant improvement over those of previous NR S3D IQA algorithms. Indeed, the accuracy of the proposed method is competitive with FR S3D IQA metrics, having ~ 91% correlation in terms of MOS.

A resolution adaptive deep hierarchical (RADHicaL) learning scheme applied to nuclear segmentation of digital pathology images

Deep learning (DL) has recently been successfully applied to a number of image analysis problems. However, DL approaches tend to be inefficient for segmentation on large image data, such as high-resolution digital pathology slide images. For example, typical breast biopsy images scanned at 40 magnification contain billions of pixels, of which usually only a small percentage belong to the class of interest. For a typical naïve deep learning scheme, parsing through and interrogating all the image pixels would represent hundreds if not thousands of hours of compute time using high performance computing environments. In this paper, we present a resolution adaptive deep hierarchical (RADHicaL) learning scheme wherein DL networks at lower resolutions are leveraged to determine if higher levels of magnification, and thus computation, are necessary to provide precise results. We evaluate our approach on a nuclear segmentation task with a cohort of 141 ER+ breast cancer images and show we can reduce computation time on average by about 85%. Expert annotations of 12,000 nuclei across these 141 images were employed for quantitative evaluation of RADHicaL. A head-to-head comparison with a naïve DL approach, operating solely at the highest magnification, yielded the following performance metrics: .9407 vs .9854 Detection Rate, .8218 vs .8489 F-score, .8061 vs .8364 true positive rate and .8822 vs 0.8932 positive predictive value. Our performance indices compare favourably with state of the art nuclear segmentation approaches for digital pathology images.

Deep extreme feature extraction: New MVA method for searching particles in high energy physics

In this paper, we propose Deep Extreme Feature Extraction (DEFE), a new ensemble MVA method for searching ?+?- channel of Higgs bosons in high energy physics. DEFE can be viewed as a deep ensemble learning scheme that trains a strongly diverse set of neural feature learners without explicitly encouraging diversity and penalizing correlations, which is achieved by adopting an implicit neural controller (not involved in feed forward computation) that directly controls and distributes gradient flows from higher level deep prediction network. Such model-independent controller results in that every single local feature learned are used in the feature-to-output mapping stage, avoiding the blind averaging of features. DEFE makes the ensembles ?deep? in the sense that it allows deep post-process of these features that try to learn to select and abstract the ensemble of neural feature learners. Based the construction and approximation of the so-called extreme selection region, the DEFE model is able to be trained efficiently, and extract discriminative features from multiple angles and dimensions, hence the improvement of the selection region of searching new particles in HEP can be achieved. With the application of this model, a selection region full of signal processes can be obtained through the training of miniature collision events set. In comparison with the Classic Deep Neural Network, DEFE shows a state-of-the-art performance: the error rate has decreased by about 37%, the accuracy has broken through 90% for the first time, along with the discovery significance has reached a standard deviation of 6.0?. Experimental data shows that DEFE is able to train an ensemble of discriminative feature learners that boosts the over performance of final prediction. Furthermore, among high-level features, there are still some important patterns that are unidentified by DNN and are independent of low-level features, while DEFE is able to identify these significant patterns more efficiently

Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields.

In this article, we tackle the problem of depth estimation from single monocular images. Compared with depth estimation using multiple images such as stereo depth perception, depth from monocular images is much more challenging. Prior work typically focuses on exploiting geometric priors or additional sources of information, most using hand-crafted features. Recently, there is mounting evidence that features from deep convolutional neural networks (CNN) set new records for various vision applications. On the other hand, considering the continuous characteristic of the depth values, depth estimation can be naturally formulated as a continuous conditional random field (CRF) learning problem. Therefore, here we present a deep convolutional neural field model for estimating depths from single monocular images, aiming to jointly explore the capacity of deep CNN and continuous CRF. In particular, we propose a deep structured learning scheme which learns the unary and pairwise potentials of continuous CRF in a unified deep CNN framework. We then further propose an equally effective model based on fully convolutional networks and a novel superpixel pooling method, which is about 10 times faster, to speedup the patch-wise convolutions in the deep model. With this more efficient model, we are able to design deeper networks to pursue better performance. Our proposed method can be used for depth estimation of general scenes with no geometric priors nor any extra information injected. In our case, the integral of the partition function can be calculated in a closed form such that we can exactly solve the log-likelihood maximization. Moreover, solving the inference problem for predicting depths of a test image is highly efficient as closed-form solutions exist. Experiments on both indoor and outdoor scene datasets demonstrate that the proposed method outperforms state-of-the-art depth estimation approaches.

Disease Inference from Health-Related Questions via Sparse Deep Learning

Automatic disease inference is of importance to bridge the gap between what online health seekers with unusual symptoms need and what busy human doctors with biased expertise can offer. However, accurately and efficiently inferring diseases is non-trivial, especially for community-based health services due to the vocabulary gap, incomplete information, correlated medical concepts, and limited high quality training samples. In this paper, we first report a user study on the information needs of health seekers in terms of questions and then select those that ask for possible diseases of their manifested symptoms for further analytic. We next propose a novel deep learning scheme to infer the possible diseases given the questions of health seekers. The proposed scheme is comprised of two key components. The first globally mines the discriminant medical signatures from raw features. The second deems the raw features and their signatures as input nodes in one layer and hidden nodes in the subsequent layer, respectively. Meanwhile, it learns the inter-relations between these two layers via pre-training with pseudo-labeled data. Following that, the hidden nodes serve as raw features for the more abstract signature mining. With incremental and alternative repeating of these two components, our scheme builds a sparsely connected deep architecture with three hidden layers. Overall, it well fits specific tasks with fine-tuning. Extensive experiments on a real-world dataset labeled by online doctors show the significant performance gains of our scheme.

A deep analysis on age estimation

The automatic estimation of age from face images is increasingly gaining attention, as it facilitates applications including advanced video surveillance, demographic statistics collection, customer profiling, or search optimization in large databases. Nevertheless, it becomes challenging to estimate age from uncontrollable environments, with insufficient and incomplete training data, dealing with strong person-specificity and high within-range variance. These difficulties have been recently addressed with complex and strongly hand-crafted descriptors, difficult to replicate and compare. This paper presents two novel approaches: first, a simple yet effective fusion of descriptors based on texture and local appearance; and second, a deep learning scheme for accurate age estimation. These methods have been evaluated under a diversity of settings, and the extensive experiments carried out on two large databases (MORPH and FRGC) demonstrate state-of-the-art results over previous work.