Self-normalizing Neural Network Research Articles

Generalized machine learning models to predict significant wave height utilizing wind and atmospheric parameters

Significant wave height (SWH) is a key parameter for wave energy extraction, ship navigation, oil and gas extraction, coastal structure construction, etc. Direct measurements of SWH using buoys are expensive and accurate over a limited area while numerical models are computationally expensive, inaccurate, with limited generalizability. Thus, this work focuses on developing generalizable machine learning models for predicting SWH from wind parameters (speed, direction, and gust) and atmospheric parameters (temperature and pressure). Two deep learning models (Artificial Neural Network (ANN) and Self Normalizing Neural Network (SNN) and two gradient boosting tree-based models (XGBoost and LightGBM) have been used in this study. Three different data sets were collected from the National Data Buoy Center: Data Set-1 (DS1): 12 years of data from 47 stations; Data Set-2 (DS2): 14 months of additional data from 6 stations randomly selected from DS1; Data Set-3 (DS3): 13 years data from completely 6 new stations. DS1 was split into training, testing, and validation datasets. Training and hyper-parameter tuning was done on the train and validation dataset, while the performance of the models was evaluated on the DS1 test dataset, DS2, and DS3, employing three error metrics: Mean Squared Error (MSE), Mean Absolute Error (MAE), and R-square score (R2). The collected data underwent data cleaning, preprocessing, and exploratory data analysis before modeling. The deep learning models have demonstrated superior fitting capacity to the tree-based models, achieving the lowest MSE (0.047), MAE (0.153), and highest R2 score (0.953) on test data. However, the gradient boosting models demonstrate better generalizing capacity than the deep learning models on DS2 and DS3. This study helps further Sustainable Development Goal 7 (SDG 7) by allowing fast and cheap assessment of wave height for ocean energy site development.

Bioengineering and Geomatics: Automatic Brain Image Segmentation using Two-Stage Pipeline with SNN and Watershed Algorithm

Digital image processing holds an increasingly essential role in the medical domain. This study emphasizes the significance of researching and implementing methods aimed at the segmentation of critical image regions and potential noise reduction, which is indispensable for medical professionals in disease diagnosis. Consequently, the investigation of software solutions in this context can substantially enhance diagnostic accuracy. In particular, neurology stands as a medical field wherein imaging plays a substantial contributory role. In pursuit of an automated brain image segmentation approach, this paper centers its attention on a two-step pipeline methodology to address the segmentation challenges inherent in medical imaging. The proposed method incorporates the use of a Self-Normalizing Neural Network (SNN) for denoising and employs the Watershed algorithm, typically employed in Geomatics imagery, for segmentation. Encouraging results are obtained, with a segmentation performance, as measured by IoU, reaching a noteworthy value of 0.93 when compared with alternative segmentation software.

Emotional Health Detection in HAR: New Approach Using Ensemble SNN

Computer recognition of human activity is an important area of research in computer vision. Human activity recognition (HAR) involves identifying human activities in real-life contexts and plays an important role in interpersonal interaction. Artificial intelligence usually identifies activities by analyzing data collected using different sources. These can be wearable sensors, MEMS devices embedded in smartphones, cameras, or CCTV systems. As part of HAR, computer vision technology can be applied to the recognition of the emotional state through facial expressions using facial positions such as the nose, eyes, and lips. Human facial expressions change with different health states. Our application is oriented toward the detection of the emotional health of subjects using a self-normalizing neural network (SNN) in cascade with an ensemble layer. We identify the subjects’ emotional states through which the medical staff can derive useful indications of the patient’s state of health.

Novel Method for Three-Dimensional Facial Expression Recognition Using Self-Normalizing Neural Networks and Mobile Devices.

Introduction To date, most ways to perform facial expression recognition rely on two-dimensional images, advanced approaches with three-dimensional data exist. These however demand stationary apparatuses and thus lack portability and possibilities to scale deployment. As human emotions, intent and even diseases may condense in distinct facial expressions or changes therein, the need for a portable yet capable solution is signified. Due to the superior informative value of three-dimensional data on facial morphology and because certain syndromes find expression in specific facial dysmorphisms, a solution should allow portable acquisition of true three-dimensional facial scans in real time. In this study we present a novel solution for the three-dimensional acquisition of facial geometry data and the recognition of facial expressions from it. The new technology presented here only requires the use of a smartphone or tablet with an integrated TrueDepth camera and enables real-time acquisition of the geometry and its categorization into distinct facial expressions. Material and Methods Our approach consisted of two parts: First, training data was acquired by asking a collective of 226 medical students to adopt defined facial expressions while their current facial morphology was captured by our specially developed app running on iPads, placed in front of the students. In total, the list of the facial expressions to be shown by the participants consisted of “disappointed”, “stressed”, “happy”, “sad” and “surprised”. Second, the data were used to train a self-normalizing neural network. A set of all factors describing the current facial expression at a time is referred to as “snapshot”. Results In total, over half a million snapshots were recorded in the study. Ultimately, the network achieved an overall accuracy of 80.54% after 400 epochs of training. In test, an overall accuracy of 81.15% was determined. Recall values differed by the category of a snapshot and ranged from 74.79% for “stressed” to 87.61% for “happy”. Precision showed similar results, whereas “sad” achieved the lowest value at 77.48% and “surprised” the highest at 86.87%. Conclusions With the present work it can be demonstrated that respectable results can be achieved even when using data sets with some challenges. Through various measures, already incorporated into an optimized version of our app, it is to be expected that the training results can be significantly improved and made more precise in the future. Currently a follow-up study with the new version of our app that encompasses the suggested alterations and adaptions, is being conducted. We aim to build a large and open database of facial scans not only for facial expression recognition but to perform disease recognition and to monitor diseases’ treatment progresses.

Machine learning-derived major adverse event prediction of patients undergoing transvenous lead extraction: Using the ESC EHRA EORP European lead extraction ConTRolled ELECTRa registry.

Transvenous lead extraction (TLE) remains a high-risk procedure. The purpose of this study was to develop a machine learning (ML)-based risk stratification system to predict the risk of major adverse events (MAEs) after TLE. A MAE was defined as procedure-related major complication and procedure-related death. We designed and evaluated an ML-based risk stratification system trained using the European Lead Extraction ConTRolled (ELECTRa) registry to predict the risk of MAEs in 3555 patients undergoing TLE and tested this on an independent registry of 1171 patients. ML models were developed, including a self-normalizing neural network (SNN), stepwise logistic regression model ("stepwise model"), support vector machines, and random forest model. These were compared with the ELECTRa Registry Outcome Score (EROS) for MAEs. There were 53 MAEs (1.7%) in the training cohort and 24 (2.4%) in the test cohort. Thirty-two clinically important features were used to train the models. ML techniques were similar to EROS by balanced accuracy (stepwise model: 0.74 vs EROS: 0.70) and superior by area under the curve (support vector machines: 0.764 vs EROS: 0.677). The SNN provided a finite risk for MAE and accurately identified MAE in 14 of 169 "high (>80%) risk" patients (8.3%) and no MAEs in all 198 "low (<20%) risk" patients (100%). ML models incrementally improved risk prediction for identifying those at risk of MAEs. The SNN has the additional advantage of providing a personalized finite risk assessment for patients. This may aid patient decision making and allow better preoperative risk assessment and resource allocation.

Differentially private self-normalizing neural networks for adversarial robustness in federated learning

The need for robust, secure and private machine learning is an important goal for realizing the full potential of the Internet of Things (IoT). Federated Learning has proven to help protect against privacy violations and information leakage. However, it introduces new risk vectors which make Machine Learning models more difficult to defend against adversarial samples. We consider the problem of improving the resilience of Federated Learning to adversarial samples without compromising the privacy preserving features which Federated Learning was primarily designed to provide.Common techniques such as adversarial training, in their basic form are not well suited for a Federated Learning environment. Our study shows that adversarial training, while improving adversarial robustness, comes at a cost of reducing the privacy guarantee of Federated Learning.We introduce DiPSeN, a Differentially Private Self-normalizing Neural Network which combines elements of differential privacy, self-normalization, and a novel optimization algorithm for adversarial client selection. Our empirical results on publicly available datasets for intrusion detection and image classification show that DiPSeN successfully improves both adversarial robustness in Federated Learning, while maintaining the privacy preserving characteristics of Federated Learning.

Pan-Cancer Classification Based on Self-Normalizing Neural Networks and Feature Selection.

Cancer is a one of the severest diseases and cancer classification plays an important role in cancer diagnosis and treatment. Some different cancers even have similar molecular features such as DNA copy number variant. Pan-cancer classification is still non-trivial at molecular level. Herein, we propose a computational method to classify cancer types by using the self-normalizing neural network (SNN) for analyzing pan-cancer copy number variation data. Since the dimension of the copy number variation features is high, the Monte Carlo feature selection method was used to rank these features. Then a classifier was built by SNN and feature selection method to select features. Three thousand six hundred ninety-four features were chosen for the prediction model, which yields the accuracy value is 0.798 and macro F1 is 0.789. We compared our model to random forest method. Results show the accuracy and macro F1 obtained by our classifier are higher than those obtained by random forest classifier, indicating the good predictive power of our method in distinguishing four different cancer types. This method is also extendable to pan-cancer classification for other molecular features.

DeepDCA: Novel Network-Based Detection of IoT Attacks Using Artificial Immune System

Recently Internet of Things (IoT) attains tremendous popularity, although this promising technology leads to a variety of security obstacles. The conventional solutions do not suit the new dilemmas brought by the IoT ecosystem. Conversely, Artificial Immune Systems (AIS) is intelligent and adaptive systems mimic the human immune system which holds desirable properties for such a dynamic environment and provides an opportunity to improve IoT security. In this work, we develop a novel hybrid Deep Learning and Dendritic Cell Algorithm (DeepDCA) in the context of an Intrusion Detection System (IDS). The framework adopts Dendritic Cell Algorithm (DCA) and Self Normalizing Neural Network (SNN). The aim of this research is to classify IoT intrusion and minimize the false alarm generation. Also, automate and smooth the signal extraction phase which improves the classification performance. The proposed IDS selects the convenient set of features from the IoT-Bot dataset, performs signal categorization using the SNN then use the DCA for classification. The experimentation results show that DeepDCA performed well in detecting the IoT attacks with a high detection rate demonstrating over 98.73% accuracy and low false-positive rate. Also, we compared these results with State-of-the-art techniques, which showed that our model is capable of performing better classification tasks than SVM, NB, KNN, and MLP. We plan to carry out further experiments to verify the framework using a more challenging dataset and make further comparisons with other signal extraction approaches. Also, involve in real-time (online) attack detection.

Prediction of Citrullination Sites on the Basis of mRMR Method and SNN.

Citrullination, an important post-translational modification of proteins, alters the molecular weight and electrostatic charge of the protein side chains. Citrulline, in protein sequences, is catalyzed by a class of Peptidyl Arginine Deiminases (PADs). Dependent on Ca2+, PADs include five isozymes: PAD 1, 2, 3, 4/5, and 6. Citrullinated proteins have been identified in many biological and pathological processes. Among them, abnormal protein citrullination modification can lead to serious human diseases, including multiple sclerosis and rheumatoid arthritis. It is important to identify the citrullination sites in protein sequences. The accurate identification of citrullination sites may contribute to the studies on the molecular functions and pathological mechanisms of related diseases. In this study, after an encoded training set (containing 116 positive and 348 negative samples) into the feature matrix, the mRMR method was used to analyze the 941- dimensional features which were sorted on the basis of their importance. Then, a predictive model based on a self-normalizing neural network (SNN) was proposed to predict the citrullination sites in protein sequences. Incremental Feature Selection (IFS) and 10-fold cross-validation were used as the model evaluation method. Three classical machine learning models, namely random forest, support vector machine, and k-nearest neighbor algorithm, were selected and compared with the SNN prediction model using the same evaluation methods. SNN may be the best tool for citrullination site prediction. The maximum value of the Matthews Correlation Coefficient (MCC) reached 0.672404 on the basis of the optimal classifier of SNN. The results showed that the SNN-based prediction methods performed better when evaluated by some common metrics, such as MCC, accuracy, and F1-Measure. SNN prediction model also achieved a better balance in the classification and recognition of positive and negative samples from datasets compared with the other three models.

Pulsar candidate selection based on self-normalizing neural networks

Pulsar candidate selection is an important step in the search task of pulsars. The traditional candidate selection is heavily dependent on human inspection. However, the human inspection is a subjective, time consuming, and error-prone process. A modern radio telescopes pulsar survey project can produce totally millions of candidates, so the manual selection becomes extremely difficult and inefficient due to a large number of candidates. Therefore, this study focuses on machine learning developed in recent years. In order to improve the efficiency of pulsar candidate selection, we propose a candidate selection method based on self-normalizing neural networks. This method uses three techniques: self-normalizing neural networks, genetic algorithm and synthetic minority over-sampling technique. The self-normalizing neural networks can improve the identification accuracy by applying deep neural networks to pulsar candidate selection. At the same time, it solves the problem of gradient disappearance and explosion in the training process of deep neural networks by using its self-normalizing property, which greatly accelerates the training process. In addition, in order to eliminate the redundancy of the sample data, we use genetic algorithm to choose sample features of pulsar candidates. The genetic algorithm for feature selection can be summarized into three steps: initializing population, assessing population fitness, and generating new populations. Decoding the individual with the largest fitness value in the last generation population, we can obtain the best subset of features. Due to radio frequency interference or noise, there are a large number of non-pulsar signals in candidates, and only a few real pulsar signals exist there. Aiming at solving the severe class imbalance problem, we use the synthetic minority over-sampling technique to increase the pulsar candidates (minority class) and reduce the imbalance degree of data. By using &lt;i&gt;k&lt;/i&gt;-nearest neighbor and linear interpolation to insert a new sample between two minority classes of samples that are close to each other according to certain rules, we can prevent the classifier from becoming biased towards the abundant non-pulsar class (majority class). Experimental results on three pulsar candidate datasets show that the self-normalizing neural network has higher accuracy and faster convergence speed than the traditional artificial neural network in the deep structure, By using the genetic algorithm and synthetic minority over-sampling technique, the selection performance of pulsar candidates can be effectively improved.

PathwayMap: Molecular Pathway Association with Self-Normalizing Neural Networks.

Drug discovery suffers from high attrition because compounds initially deemed as promising can later show ineffectiveness or toxicity resulting from a poor understanding of their activity profile. In this work, we describe a deep self-normalizing neural network model for the prediction of molecular pathway association and evaluate its performance, showing an AUC ranging from 0.69 to 0.91 on a set of compounds extracted from ChEMBL and from 0.81 to 0.83 on an external data set provided by Novartis. We finally discuss the applicability of the proposed model in the domain of lead discovery. A usable application is available via PlayMolecule.org .

Identifying Patients with Atrioventricular Septal Defect in Down Syndrome Populations by Using Self-Normalizing Neural Networks and Feature Selection.

Atrioventricular septal defect (AVSD) is a clinically significant subtype of congenital heart disease (CHD) that severely influences the health of babies during birth and is associated with Down syndrome (DS). Thus, exploring the differences in functional genes in DS samples with and without AVSD is a critical way to investigate the complex association between AVSD and DS. In this study, we present a computational method to distinguish DS patients with AVSD from those without AVSD using the newly proposed self-normalizing neural network (SNN). First, each patient was encoded by using the copy number of probes on chromosome 21. The encoded features were ranked by the reliable Monte Carlo feature selection (MCFS) method to obtain a ranked feature list. Based on this feature list, we used a two-stage incremental feature selection to construct two series of feature subsets and applied SNNs to build classifiers to identify optimal features. Results show that 2737 optimal features were obtained, and the corresponding optimal SNN classifier constructed on optimal features yielded a Matthew’s correlation coefficient (MCC) value of 0.748. For comparison, random forest was also used to build classifiers and uncover optimal features. This method received an optimal MCC value of 0.582 when top 132 features were utilized. Finally, we analyzed some key features derived from the optimal features in SNNs found in literature support to further reveal their essential roles.

Automatic segmentation method of pelvic floor levator hiatus in ultrasound using a self-normalizing neural network.

.Segmentation of the levator hiatus in ultrasound allows the extraction of biometrics, which are of importance for pelvic floor disorder assessment. We present a fully automatic method using a convolutional neural network (CNN) to outline the levator hiatus in a two-dimensional image extracted from a three-dimensional ultrasound volume. In particular, our method uses a recently developed scaled exponential linear unit (SELU) as a nonlinear self-normalizing activation function, which for the first time has been applied in medical imaging with CNN. SELU has important advantages such as being parameter-free and mini-batch independent, which may help to overcome memory constraints during training. A dataset with 91 images from 35 patients during Valsalva, contraction, and rest, all labeled by three operators, is used for training and evaluation in a leave-one-patient-out cross validation. Results show a median Dice similarity coefficient of 0.90 with an interquartile range of 0.08, with equivalent performance to the three operators (with a Williams’ index of 1.03), and outperforming a U-Net architecture without the need for batch normalization. We conclude that the proposed fully automatic method achieved equivalent accuracy in segmenting the pelvic floor levator hiatus compared to a previous semiautomatic approach.