Multi-layer Deep Neural Network Research Articles

A Deep Learning-Based Intrusion Detection and Preventation System for Detecting and Preventing Denial-of-Service Attacks

This document classifies, selects and trains a deep learning algorithm to create an IDS/IPS (Intrusion Prevention/Detection System) called Dique, which can detect and prevent denial of service (DoS) attacks. To mitigate DoS attacks, the IDS/IPS system, using the proposed deep learning model, classifies incoming packets to the web server into two classes: benign (which are normal traffic packets) and malicious (which the system considers to contain possible DoS attacks). Dique has a Graphical User Interface (GUI) where "in real time" you can display graphically and textually the information of captured and classified packets, and allows you to switch between the IDS mode and the IPS mode of the system operation. The proposed DoS attack classification model uses a multi-layered Deep Feed Forward neural network, the CICDDoS2019 Dataset was used for training and an accuracy of 0.994 was achieved. In addition, an offensive system called Diluvio was developed to verify the functioning of the Dique system. In Diluvio seven different types of DoS attacks were implemented (five contents in the training Datset and two that are not in said dataset) that users can selectively launch against a web server.

Artificial intelligence, machine learning, and deep learning for clinical outcome prediction.

AI is a broad concept, grouping initiatives that use a computer to perform tasks that would usually require a human to complete. AI methods are well suited to predict clinical outcomes. In practice, AI methods can be thought of as functions that learn the outcomes accompanying standardized input data to produce accurate outcome predictions when trialed with new data. Current methods for cleaning, creating, accessing, extracting, augmenting, and representing data for training AI clinical prediction models are well defined. The use of AI to predict clinical outcomes is a dynamic and rapidly evolving arena, with new methods and applications emerging. Extraction or accession of electronic health care records and combining these with patient genetic data is an area of present attention, with tremendous potential for future growth. Machine learning approaches, including decision tree methods of Random Forest and XGBoost, and deep learning techniques including deep multi-layer and recurrent neural networks, afford unique capabilities to accurately create predictions from high dimensional, multimodal data. Furthermore, AI methods are increasing our ability to accurately predict clinical outcomes that previously were difficult to model, including time-dependent and multi-class outcomes. Barriers to robust AI-based clinical outcome model deployment include changing AI product development interfaces, the specificity of regulation requirements, and limitations in ensuring model interpretability, generalizability, and adaptability over time.

MSPAN: A Memristive Spike-Based Computing Engine With Adaptive Neuron for Edge Arrhythmia Detection.

In this work, a memristive spike-based computing in memory (CIM) system with adaptive neuron (MSPAN) is proposed to realize energy-efficient remote arrhythmia detection with high accuracy in edge devices by software and hardware co-design. A multi-layer deep integrative spiking neural network (DiSNN) is first designed with an accuracy of 93.6% in 4-class ECG classification tasks. Then a memristor-based CIM architecture and the corresponding mapping method are proposed to deploy the DiSNN. By evaluation, the overall system achieves an accuracy of over 92.25% on the MIT-BIH dataset while the area is 3.438 mm2 and the power consumption is 0.178 μJ per heartbeat at a clock frequency of 500 MHz. These results reveal that the proposed MSPAN system is promising for arrhythmia detection in edge devices.

Artificial Intelligence to Detect Meibomian Gland Dysfunction From in-vivo Laser Confocal Microscopy.

Background: In recent years, deep learning has been widely used in a variety of ophthalmic diseases. As a common ophthalmic disease, meibomian gland dysfunction (MGD) has a unique phenotype in in-vivo laser confocal microscope imaging (VLCMI). The purpose of our study was to investigate a deep learning algorithm to differentiate and classify obstructive MGD (OMGD), atrophic MGD (AMGD) and normal groups.Methods: In this study, a multi-layer deep convolution neural network (CNN) was trained using VLCMI from OMGD, AMGD and healthy subjects as verified by medical experts. The automatic differential diagnosis of OMGD, AMGD and healthy people was tested by comparing its image-based identification of each group with the medical expert diagnosis. The CNN was trained and validated with 4,985 and 1,663 VLCMI images, respectively. By using established enhancement techniques, 1,663 untrained VLCMI images were tested.Results: In this study, we included 2,766 healthy control VLCMIs, 2,744 from OMGD and 2,801 from AMGD. Of the three models, differential diagnostic accuracy of the DenseNet169 CNN was highest at over 97%. The sensitivity and specificity of the DenseNet169 model for OMGD were 88.8 and 95.4%, respectively; and for AMGD 89.4 and 98.4%, respectively.Conclusion: This study described a deep learning algorithm to automatically check and classify VLCMI images of MGD. By optimizing the algorithm, the classifier model displayed excellent accuracy. With further development, this model may become an effective tool for the differential diagnosis of MGD.

Multi-layer neural network for solving problems of recognition of drained crop areas

The article deals with the design and software implementation of neural network modules for solving problems of image recognition. In particular, it describes the development of a module for network training and recognition of input pulses, which made it possible to recognize, process and analyse aerial photographs of agricultural crops as objects of identification based on the use of a multi-layer deep learning neural network. The practical use of the software tool is possible for the study and study of the peculiarities of cultivation in conditions of irrigation and differentiated placement.

Development of a Multilayer Deep Neural Network Model for Predicting Hourly River Water Temperature From Meteorological Data

Water temperature is a vital attribute of physical riverine habitat and one of the focal objectives of river engineering and management. However, in most rivers, there are not enough water temperature measurements to characterize thermal regimes and evaluate its effect on ecosystem functions such as fish migration. To aid in river restoration, machine learning-based algorithms were developed to predict hourly river water temperature. We trained, validated, and tested single-layer and multilayer linear regression (LR) and deep neural network (DNN) algorithms to predict water temperature in the Los Angeles River in southern CA, United States. For the single-layer models, we considered air temperature as the predictive feature, and for the multilayer models, relative humidity, wind speed, and barometric pressure were included in addition to air temperature as the considered features. We trained the LR and DNN algorithms on Google’s TensorFlow model using Keras artificial neural network library on Python. Results showed that multilayer predictions performed better compared to single-layer models by producing mean absolute errors (MAEs), that were 20% smaller (1.05°C), on average, compared to the single-layer models (1.3°C). The multilayer DNN algorithm outperformed the other model where the model’s coefficient of determination was 26 and 12% higher compared to the single-layer LR (the base model) and multilayer LR model, respectively. The multilayer machine learning algorithms, under proper data preparation protocols, may be considered useful tools for predicting water temperatures in sampled and unsampled rivers for current conditions and future estimations affected by different stressors such as climate and land-use change. River temperature predictions from the developed models provide valuable information for evaluating sustainability of river ecosystems and biota.

Oriented stochastic loss descent algorithm to train very deep multi-layer neural networks without vanishing gradients

Deep multi-layer neural networks represent hypotheses of very high degree polynomials to solve very complex problems. Gradient descent optimization algorithms are utilized to train such deep networks through backpropagation, which suffers from permanent problems such as the vanishing gradient problem. To overcome the vanishing problem, we introduce a new anti-vanishing back-propagated learning algorithm called oriented stochastic loss descent (OSLD). OSLD updates a random-initialized parameter iteratively in the opposite direction of its partial derivative sign by a small positive random number, which is scaled by a tuned ratio of the model loss. This paper compares OSLD to stochastic gradient descent algorithm as the basic backpropagation algorithm and Adam as one of the best backpropagation algorithms in five benchmark models. Experimental results show that OSLD is very competitive to Adam in small and moderate depth models, and OSLD outperforms Adam in very long models. Moreover, OSLD is compatible with current backpropagation architectures except for learning rates. Finally, OSLD is stable and opens more choices in front of the very deep multi-layer neural networks.

PhosIDN: an integrated deep neural network for improving protein phosphorylation site prediction by combining sequence and protein-protein interaction information.

MotivationPhosphorylation is one of the most studied post-translational modifications, which plays a pivotal role in various cellular processes. Recently, deep learning methods have achieved great success in prediction of phosphorylation sites, but most of them are based on convolutional neural network that may not capture enough information about long-range dependencies between residues in a protein sequence. In addition, existing deep learning methods only make use of sequence information for predicting phosphorylation sites, and it is highly desirable to develop a deep learning architecture that can combine heterogeneous sequence and protein–protein interaction (PPI) information for more accurate phosphorylation site prediction. ResultsWe present a novel integrated deep neural network named PhosIDN, for phosphorylation site prediction by extracting and combining sequence and PPI information. In PhosIDN, a sequence feature encoding sub-network is proposed to capture not only local patterns but also long-range dependencies from protein sequences. Meanwhile, useful PPI features are also extracted in PhosIDN by a PPI feature encoding sub-network adopting a multi-layer deep neural network. Moreover, to effectively combine sequence and PPI information, a heterogeneous feature combination sub-network is introduced to fully exploit the complex associations between sequence and PPI features, and their combined features are used for final prediction. Comprehensive experiment results demonstrate that the proposed PhosIDN significantly improves the prediction performance of phosphorylation sites and compares favorably with existing general and kinase-specific phosphorylation site prediction methods.Availability and implementationPhosIDN is freely available at https://github.com/ustchangyuanyang/PhosIDN.Supplementary information Supplementary data are available at Bioinformatics online.

Deep Learning-Based Survival Analysis for High-Dimensional Survival Data

With the development of high-throughput technologies, more and more high-dimensional or ultra-high-dimensional genomic data are being generated. Therefore, effectively analyzing such data has become a significant challenge. Machine learning (ML) algorithms have been widely applied for modeling nonlinear and complicated interactions in a variety of practical fields such as high-dimensional survival data. Recently, multilayer deep neural network (DNN) models have made remarkable achievements. Thus, a Cox-based DNN prediction survival model (DNNSurv model), which was built with Keras and TensorFlow, was developed. However, its results were only evaluated on the survival datasets with high-dimensional or large sample sizes. In this paper, we evaluated the prediction performance of the DNNSurv model using ultra-high-dimensional and high-dimensional survival datasets and compared it with three popular ML survival prediction models (i.e., random survival forest and the Cox-based LASSO and Ridge models). For this purpose, we also present the optimal setting of several hyperparameters, including the selection of a tuning parameter. The proposed method demonstrated via data analysis that the DNNSurv model performed well overall as compared with the ML models, in terms of the three main evaluation measures (i.e., concordance index, time-dependent Brier score, and the time-dependent AUC) for survival prediction performance.

A Power System Disturbance Classification Method Robust to PMU Data Quality Issues

Data quality issues exist in practical phasor measurement units (PMUs) due to communication errors or signal interferences. As a result, the performances of existing data-driven disturbance classification methods can be significantly affected. In this article, a fast disturbance classification method that is robust to PMU data quality issues is proposed. The impacts of bad PMU measurements on disturbance classification are investigated by analyzing the feature distributions of deep learning methods. A new feature extraction scheme that uses the univariate temporal convolutional denoising autoencoder (UTCN-DAE) is proposed. It allows encoding and decoding univariate disturbance data through a temporal convolutional network to capture the temporal feature representation and is robust to bad data. Based on the features of the frequency and voltage measurements encoded by the UTCN-DAE, a two-stream enhanced network, i.e., the multivariable temporal convolutional denoising network is proposed to achieve optimal feature extraction of multivariate time series by feature fusion. The classification is performed using a multilayered deep neural network and Softmax classifier. Extensive results obtained on the IEEE 39-bus system as well as a large-scale power system in China with field PMU measurements show that the proposed method achieves the highest classification accuracy and computational efficiency as compared to other deep learning algorithms.

Natural Disasters Intensity Analysis and Classification Based on Multispectral Images Using Multi-Layered Deep Convolutional Neural Network.

Natural disasters not only disturb the human ecological system but also destroy the properties and critical infrastructures of human societies and even lead to permanent change in the ecosystem. Disaster can be caused by naturally occurring events such as earthquakes, cyclones, floods, and wildfires. Many deep learning techniques have been applied by various researchers to detect and classify natural disasters to overcome losses in ecosystems, but detection of natural disasters still faces issues due to the complex and imbalanced structures of images. To tackle this problem, we propose a multilayered deep convolutional neural network. The proposed model works in two blocks: Block-I convolutional neural network (B-I CNN), for detection and occurrence of disasters, and Block-II convolutional neural network (B-II CNN), for classification of natural disaster intensity types with different filters and parameters. The model is tested on 4428 natural images and performance is calculated and expressed as different statistical values: sensitivity (SE), 97.54%; specificity (SP), 98.22%; accuracy rate (AR), 99.92%; precision (PRE), 97.79%; and F1-score (F1), 97.97%. The overall accuracy for the whole model is 99.92%, which is competitive and comparable with state-of-the-art algorithms.

GPDBN: deep bilinear network integrating both genomic data and pathological images for breast cancer prognosis prediction.

MotivationBreast cancer is a very heterogeneous disease and there is an urgent need to design computational methods that can accurately predict the prognosis of breast cancer for appropriate therapeutic regime. Recently, deep learning-based methods have achieved great success in prognosis prediction, but many of them directly combine features from different modalities that may ignore the complex inter-modality relations. In addition, existing deep learning-based methods do not take intra-modality relations into consideration that are also beneficial to prognosis prediction. Therefore, it is of great importance to develop a deep learning-based method that can take advantage of the complementary information between intra-modality and inter-modality by integrating data from different modalities for more accurate prognosis prediction of breast cancer.ResultsWe present a novel unified framework named genomic and pathological deep bilinear network (GPDBN) for prognosis prediction of breast cancer by effectively integrating both genomic data and pathological images. In GPDBN, an inter-modality bilinear feature encoding module is proposed to model complex inter-modality relations for fully exploiting intrinsic relationship of the features across different modalities. Meanwhile, intra-modality relations that are also beneficial to prognosis prediction, are captured by two intra-modality bilinear feature encoding modules. Moreover, to take advantage of the complementary information between inter-modality and intra-modality relations, GPDBN further combines the inter- and intra-modality bilinear features by using a multi-layer deep neural network for final prognosis prediction. Comprehensive experiment results demonstrate that the proposed GPDBN significantly improves the performance of breast cancer prognosis prediction and compares favorably with existing methods.Availabilityand implementationGPDBN is freely available at https://github.com/isfj/GPDBN.Supplementary informationSupplementary data are available at Bioinformatics online.

WITHDRAWN: A novel generative adversarial block truncation coding schemes for high rated image compression on E-learning resource environment

Abstract Image compression is a useful technique that is inevitable to store any images in compressed pattern. The compressed image is reconstructed using image retrieval process for any applications. This technique ensures storage efficiency in various content management applications. Particularly, E-Learning resource environment deals with more space complexity. The E-Learning storage space complexity can be reduced with the help of image compression techniques. There are different types of image compression techniques invented for building compressed resources. Block Truncation Coding (BTC) is a type of lossy compression technique for reducing grey scale quantities using blocking and quantizing phases. This is an effective technique for compressing the images. In extend, Absolute Moment BTC (AMBTC) technique is proposed for minimizing the reconstruction error rate of standard BTC model. The image blocks generation and compression are the main phases of BTC model. This can be applied for both colour images and grey scale images. However, the conventional BTC procedures lacks for edge reconstructions and noise reductions in the output images. This problem creates BTC with lack of inaccuracy in complex image compression platforms such as E-Learning image sources. In the scope, the need for more active and generative image compression procedures is necessary to improve BTC principles. This work proposes highly complex and novel Generative Adversarial BTC (GA-BTC) compression models with multiple variances. GABTC is developed with multi-layered Deep Neural Network (DNN) structures with GA neural models. The integration of both GA models and BTC principles improve the quality of block constructions and reconstructions significantly. This work evaluates the model complexity and efficiency using various E-Learning images (Colour and Grey Scale) with different compression quality measurements.

Separation of color channels from conventional colonoscopy images improves deep neural network detection of polyps.

.Significance: Colorectal cancer incidence has decreased largely due to detection and removal of polyps. Computer-aided diagnosis development may improve on polyp detection and discrimination.Aim: To advance detection and discrimination using currently available commercial colonoscopy systems, we developed a deep neural network (DNN) separating the color channels from images acquired under narrow-band imaging (NBI) and white-light endoscopy (WLE).Approach: Images of normal colon mucosa and polyps from colonoscopies were studied. Each color image was extracted based on the color channel: red/green/blue. A multilayer DNN was trained using one-channel, two-channel, and full-color images. The trained DNN was then tested for performance in detection of polyps.Results: The DNN performed better using full-colored NBI over WLE images in the detection of polyps. Furthermore, the DNN performed better using the two-channel red + green images when compared to full-color WLE images.Conclusions: The separation of color channels from full-color NBI and WLE images taken from commercially available colonoscopes may improve the ability of the DNN to detect and discriminate polyps. Further studies are needed to better determine the color channels and combination of channels to include and exclude in DNN development for clinical use.

Exploiting deep learning for predictable carbon dot design.

In this study, we developed a deep convolution neural network (DCNN) model for predicting the optical properties of carbon dots (CDs), including spectral properties and fluorescence color under ultraviolet irradiation. These results demonstrate the powerful potential of DCNN for guiding the synthesis of CDs.

Stress detection using deep neural networks

BackgroundOver 70% of Americans regularly experience stress. Chronic stress results in cancer, cardiovascular disease, depression, and diabetes, and thus is deeply detrimental to physiological health and psychological wellbeing. Developing robust methods for the rapid and accurate detection of human stress is of paramount importance.MethodsPrior research has shown that analyzing physiological signals is a reliable predictor of stress. Such signals are collected from sensors that are attached to the human body. Researchers have attempted to detect stress by using traditional machine learning methods to analyze physiological signals. Results, ranging between 50 and 90% accuracy, have been mixed. A limitation of traditional machine learning algorithms is the requirement for hand-crafted features. Accuracy decreases if features are misidentified. To address this deficiency, we developed two deep neural networks: a 1-dimensional (1D) convolutional neural network and a multilayer perceptron neural network. Deep neural networks do not require hand-crafted features but instead extract features from raw data through the layers of the neural networks. The deep neural networks analyzed physiological data collected from chest-worn and wrist-worn sensors to perform two tasks. We tailored each neural network to analyze data from either the chest-worn (1D convolutional neural network) or wrist-worn (multilayer perceptron neural network) sensors. The first task was binary classification for stress detection, in which the networks differentiated between stressed and non-stressed states. The second task was 3-class classification for emotion classification, in which the networks differentiated between baseline, stressed, and amused states. The networks were trained and tested on publicly available data collected in previous studies.ResultsThe deep convolutional neural network achieved 99.80% and 99.55% accuracy rates for binary and 3-class classification, respectively. The deep multilayer perceptron neural network achieved 99.65% and 98.38% accuracy rates for binary and 3-class classification, respectively. The networks’ performance exhibited a significant improvement over past methods that analyzed physiological signals for both binary stress detection and 3-class emotion classification.ConclusionsWe demonstrated the potential of deep neural networks for developing robust, continuous, and noninvasive methods for stress detection and emotion classification, with the end goal of improving the quality of life.

Physics-Based Deep Learning for Fiber-Optic Communication Systems

We propose a new machine-learning approach for fiber-optic communication systems whose signal propagation is governed by the nonlinear Schrodinger equation (NLSE). Our main observation is that the popular split-step method (SSM) for numerically solving the NLSE has essentially the same functional form as a deep multi-layer neural network; in both cases, one alternates linear steps and pointwise nonlinearities. We exploit this connection by parameterizing the SSM and viewing the linear steps as general linear functions, similar to the weight matrices in a neural network. The resulting physics-based machine-learning model has several advantages over “black-box” function approximators. For example, it allows us to examine and interpret the learned solutions in order to understand why they perform well. As an application, low-complexity nonlinear equalization is considered, where the task is to efficiently invert the NLSE. This is commonly referred to as digital backpropagation (DBP). Rather than employing neural networks, the proposed algorithm, dubbed learned DBP (LDBP), uses the physics-based model with trainable filters in each step and its complexity is reduced by progressively pruning filter taps during gradient descent. Our main finding is that the filters can be pruned to remarkably short lengths—as few as 3 taps/step—without sacrificing performance. As a result, the complexity can be reduced by orders of magnitude in comparison to prior work. By inspecting the filter responses, an additional theoretical justification for the learned parameter configurations is provided. Our work illustrates that combining data-driven optimization with existing domain knowledge can generate new insights into old communications problems.

Analysis of deep multilayer perceptron neural network in MWC coded optical CDMA system with LDPC code

Abstract In this paper, applying the Deep Multilayer Perceptron Neural Network (MLPNN) to the Sum-Product Algorithm (SPA) for decoding the Modified Welch-Costas (MWC) coded Optical Code Division Multiple Access (OCDMA) system with Low-Density Parity-Check (LDPC) code is analyzed. The goal is to train the MLPNN-SPA through the stochastic gradient descent (SGD) to learn and optimize the weights to each edge of the neural network decoder. Once these parameters have been trained, the decoding complexity of the MLPNN-SPA is similar to that of the SPA. Furthermore, from the simulation results, it has been shown that the MLNN-SPA can improve the system performance without additional decoding complexity as compared to the SPA.

Comparison of partial least squares-discriminant analysis, support vector machines and deep neural networks for spectrometric classification of seed vigour in a broad range of tree species

Seed vigour significantly influences the seed production and plant regeneration performance. The capability of NIR spectroscopy to identify seed vigour across multiple tree species rapidly and cost-effectively has been examined. The NIR spectra of seeds from five different tree species have been taken. Standard germination testing has also been used to verify seed vigour. Three classification models were trained, i.e., partial least squares-discriminant analysis (PLSDA), support vector machine (SVM) and multilayer deep neural network (DNN). Three types of spectral pre-processing methods and their combination were used to fit for the best classification model. The DNN model has shown good performance on all pre-processing methods and yielded higher accuracy than other models in this study, with accuracy, sensitivity, precision and specificity all equal to 1. Compared with other pre-processing methods, the second derivative spectra have shown a robust and consistent classification result in both PLSDA and DNN models. Five important regions including 1270, 1650, 1720, 2100, 2300 nm were found highly related to the seed vigour. This study has found a rapid and efficient methodology for seed vigour classification, which could serve for industrial use in a rapid and non-destructive way.

Recognition of speech emotion using custom 2D-convolution neural network deep learning algorithm

Speech emotion recognition has become the heart of most human computer interaction applications in the modern world. The growing need to develop emotionally intelligent devices has opened up a lot of research opportunities. Most researchers in this field have applied the use of handcrafted features and machine learning techniques in recognising speech emotion. However, these techniques require extra processing steps and handcrafted features are usually not robust. They are computationally intensive because the curse of dimensionality results in low discriminating power. Research has shown that deep learning algorithms are effective for extracting robust and salient features in dataset. In this study, we have developed a custom 2D-convolution neural network that performs both feature extraction and classification of vocal utterances. The neural network has been evaluated against deep multilayer perceptron neural network and deep radial basis function neural network using the Berlin database of emotional speech, Ryerson audio-visual emotional speech database and Surrey audio-visual expressed emotion corpus. The described deep learning algorithm achieves the highest precision, recall and F1-scores when compared to other existing algorithms. It is observed that there may be need to develop customized solutions for different language settings depending on the area of applications.