Exponential Linear Unit Research Articles

Ferrimagnet-Based Neuromorphic Device Mimicking the Ventral Visual Pathway for High-Accuracy Target Recognition.

The ventral visual pathway (VVP) of the human brain efficiently implements target recognition by employing a deep hierarchical structure to build complex visual concepts from simple features. Artificial neural networks (ANNs) based on spintronic devices are capable of target recognition, but their poor interpretability and limited network depth hinder ANNs from mimicking the VVP. Hardware implementation of the VVP requires a biorealistic spintronic device as well as the corresponding interpretable and deep network structure, which have not been reported so far. Here, we report a ferrimagnetic neuron with a continuously differentiable exponential linear unit (CeLu) activation function, which is closer to biological neurons and could mitigate the issue of limited network depth. Meanwhile, we also demonstrate that a ferrimagnet can construct artificial synapses with high linearity and symmetry to meet the requirements of weight update algorithms. Based on these neurons and synapses, we propose an all-spin convolutional neural network (CNN) with a high interpretability and deep neural network, to mimic the VVP. Compared to the state-of-the-art spintronic-based neuromorphic computing model, the CNN with bionic function, using experimentally derived device parameters, achieves high recognition accuracies of over 91% and 98% on the CIFAR-10 datasets and the MNIST datasets, respectively, showing improvements of 1.13% and 1.76%. Our work provides a promising method to improve the bionic performance of spintronic device-based neural networks.

Modified scaled exponential linear unit

<p>Activation functions assume a crucial role in elucidating the intricacies of training dynamics and the overall performance of neural networks. Despite its simplicity and effectiveness, the ubiquitously embraced ReLU activation function harbors certain drawbacks, notably the predicament recognized as the “Dying ReLU” issue. To address such challenges, we propose the introduction of a pioneering activation function, the modified scaled exponential linear unit (M-SELU). Drawing from an array of experiments conducted across diverse computer vision tasks employing cutting-edge architectures, it becomes apparent that M-SELU exhibits superior performance compared to ReLU (used as the baseline) and various other activation functions. The simplicity of the proposed activation function (M-SELU) makes this solution particularly suitable for multi-layered deep neural architecture, including applications in CNN, CIFAR-10, and the broader field of deep learning.</p>

Periodically activated physics-informed neural networks for assimilation tasks for three-dimensional Rayleigh–Bénard convection

We apply physics-informed neural networks to three-dimensional Rayleigh–Bénard convection in a cubic cell with a Rayleigh number of Ra=106 and a Prandtl number of Pr=0.7 to assimilate the velocity vector field from given temperature fields and vice versa. With the respective ground truth data provided by a direct numerical simulation, we are able to evaluate the performance of the different activation functions applied (sine, hyperbolic tangent and exponential linear unit) and different numbers of neurons (32, 64, 128, 256) for each of the five hidden layers of the multi-layer perceptron. The main result is that the use of a periodic activation function (sine) typically benefits the assimilation performance in terms of the analyzed metrics, correlation with the ground truth and mean average error. The higher quality of results from sine-activated physics-informed neural networks is also manifested in the probability density function and power spectra of the inferred velocity or temperature fields. Regarding the two assimilation directions, the assimilation of temperature fields based on velocities appears to be more challenging in the sense that it exhibits a sharper limit on the number of neurons below which viable assimilation results cannot be achieved.

Crop disease diagnosis and prediction using two-stream hybrid convolutional neural networks

Crop diseases significantly impact yield and quality, posing a direct threat to food security. The application of Convolutional Neural Networks (CNN) in crop disease recognition has notably improved diagnosis accuracy and efficiency. This study presents an innovative crop disease classification model based on the VGG-16 network. Enhancements include the incorporation of Batch Normalization (BN) and a novel activation function synergizing with Exponential Linear Units (ELU), improving model convergence speed and accuracy. Additionally, Global Average Pooling (GAP) is integrated to streamline the network architecture, and the InceptionV2 module is introduced to extract leaf disease features from different dimensions, enhancing model robustness. Validation on the PlantVillage dataset shows an accuracy rate of 98.89%, demonstrating the model's competitiveness and its potential to support sustainable agricultural production.

MARNet: Multi-head attention residual network for rolling bearing fault diagnosis under noisy condition

Rolling bearings are crucial components of rotating machinery, and their health states directly affect the overall performance of the machinery. Therefore, it is exceedingly necessary to detect and diagnose bearing faults. Numerous bearing fault diagnosis methods have been successfully used for ensuring the safe operation of rotating machinery. However, in practical working environments, there is a considerable amount of noise, resulting in traditional methods incapable of achieving accurate fault diagnosis. This paper proposes a new multi-head attention residual network (MARNet) for rolling bearing fault diagnosis under noisy condition. MARNet optimizes residual units by simplifying multi-layer convolutions into a single-layer convolution and replaces the rectified linear unit (ReLU) function with the exponential linear unit (ELU) function to obtain a more appropriate activation function. Additionally, the multi-head attention mechanism is introduced into the residual block to capture correlation information between any two time sequences, enhancing the network’s feature extraction capability. The effectiveness and superiority of the MARNet in noisy environments are demonstrated through conducting the two bearing datasets from Case Western Reserve University (CWRU) and Paderborn University (PU). The experiment results show that the proposed method exhibits anti-noise characteristics and generalization capability compared with several up-to-date deep learning methods for fault diagnosis of rolling bearings.

Graph attention network with convolutional layer for predicting gene regulations from single-cell ribonucleic acid sequence data

Reconstructing gene regulatory networks (GRNs) is an important task to reveal the regulatory relationship between genes and understand the mechanism of intracellular gene expression regulation. With the development of single-cell ribonucleic acid sequencing (scRNA-seq) technology, researchers begin to attempt to infer GRN within cells. In this paper, we propose graph attention network with convolutional layer (GATCL) to infer the latent interactions between transcription factors (TFs) and target genes in GRN. Firstly, GATCL uses graph attention network (GAT), which can effectively extract information about genes and TFs. Secondly, we combine multi-head attention layer with one-head attention layer and propose a new method of using convolution instead of weight matrix. Thirdly, we use the exponential linear unit (ELU) activation function to replace the leaky rectified linear unit (LReLU) commonly used in GAT, which further improves the accuracy of GATCL. The AUROC of our method on seven scRNA-seq datasets with four types of ground-truth networks reached an average of 0.827, which is higher than other state-of-the-art models. GATCL applies a supervised deep learning algorithm to solve the problems existing in the inference of GRN from scRNA-seq in the engineering field, and the effectiveness of this method is verified by a large number of experiments.

Improved dendritic learning: Activation function analysis

This study conducted a thorough evaluation of an improved dendritic learning (DL) framework, focusing specifically on its application in power load forecasting. The objective was to optimise the activation functions within the synapses and somas of DL to enhance their adaptability across diverse real-world scenarios. Through a rigorous analysis involving 25 experiments across five activation functions (sigmoid, hyperbolic tangent (tanh), rectified linear unit (ReLU), leaky ReLU, and exponential linear unit (ELU)), we elucidated their impacts on both regression and classification performance. Notably, the leaky ReLU–tanh combination demonstrated exceptional mean performance and effectiveness across 14 benchmark datasets from the University of California Irvine Machine Learning Repository, surpassing alternative combinations. When applied to power load forecasting, this combination outperformed other models, particularly transformer and LSTM. These findings underscore the significant advantages of the leaky ReLU–tanh-based DL framework in accurately predicting electricity load in smart grids, as evidenced by the lowest mean absolute error (39.27), root mean squared error (29.13), and mean absolute percentage error (2.84).

Large-scale self-normalizing neural networks

Self-normalizing neural networks (SNN) regulate the activation and gradient flows through activation functions with the self-normalization property. As SNNs do not rely on norms computed from minibatches, they are more friendly to data parallelism, kernel fusion, and emerging architectures such as ReRAM-based accelerators. However, existing SNNs have mainly demonstrated their effectiveness on toy datasets and fall short in accuracy when dealing with large-scale tasks like ImageNet. They lack the strong normalization, regularization, and expression power required for wider, deeper models and larger-scale tasks. To enhance the normalization strength, this paper introduces a comprehensive and practical definition of the self-normalization property in terms of the stability and attractiveness of the statistical fixed points. It is comprehensive as it jointly considers all the fixed points used by existing studies: the first and second moment of forward activation and the expected Frobenius norm of backward gradient. The practicality comes from the analytical equations provided by our paper to assess the stability and attractiveness of each fixed point, which are derived from theoretical analysis of the forward and backward signals. The proposed definition is applied to a meta activation function inspired by prior research, leading to a stronger self-normalizing activation function named “bi-scaled exponential linear unit with backward standardized” (bSELU-BSTD). We provide both theoretical and empirical evidence to show that it is superior to existing studies. To enhance the regularization and expression power, we further propose scaled-Mixup and channel-wise scale & shift. With these three techniques, our approach achieves 75.23% top-1 accuracy on the ImageNet with Conv MobileNet V1, surpassing the performance of existing self-normalizing activation functions. To the best of our knowledge, this is the first SNN that achieves comparable accuracy to batch normalization on ImageNet.

An object detection method for catenary component images based on improved Faster R-CNN

Catenary components are an important part of electrified railways. Especially for catenary support devices, there are various types of components with significant differences in scale. According to statistical data, there is a high risk of failure for the catenary support device components during the operation of the catenary system. Therefore, in order to ensure the safe operation of the railways, it is critical to accurately locate and recognize the components in the catenary images. In this paper, we propose an improved method based on faster region-based convolutional neural networks (Faster R-CNN) framework to realize the detection and extraction of the components on the catenary support devices. Firstly, the anchor box parameters are reset using the K-means clustering method, which greatly improves the localization precision of the predicted box. Secondly, scaled exponential linear units activation function is introduced to improve the algorithm performance. Moreover, ResNet-34, the backbone of Faster R-CNN, is optimized. We design a transition structure for multi-scale filter combination convolution to avoid missing feature information and eliminate some redundant convolution structures. This modification substantially enhances the capability of the model to recognize a wide variety of component types. Finally, we conduct some control experiments comparing with single shot multibox detector and you only look once (YOLO) series (YOLOv3, YOLOv5 and YOLOv7) models. They are faster but less accurate, especially for small objects. The results show that the proposed method has better detection performance, achieving a mean average precision of 96.50% and running at 17.79 frames per second. In addition, our model has the highest average recall of 69.27%, which is 2.66% higher than the original model.

MRI super-resolution using similarity distance and multi-scale receptive field based feature fusion GAN and pre-trained slice interpolation network

Challenges arise in achieving high-resolution Magnetic Resonance Imaging (MRI) to improve disease diagnosis accuracy due to limitations in hardware, patient discomfort, long acquisition times, and high costs. While Convolutional Neural Networks (CNNs) have shown promising results in MRI super-resolution, they often don't look into the structural similarity and prior information available in consecutive MRI slices. By leveraging information from sequential slices, more robust features can be obtained, potentially leading to higher-quality MRI slices. We propose a multi-slice two-dimensional (2D) MRI super-resolution network that combines a Generative Adversarial Network (GAN) with feature fusion and a pre-trained slice interpolation network to achieve three-dimensional (3D) super-resolution. The proposed model requires consecutively acquired three low-resolution (LR) MRI slices along a specific axis, and achieves the reconstruction of the MRI slices in the remaining two axes. The network effectively enhances both in-plane and out-of-plane resolution along the sagittal axis while addressing computational and memory constraints in 3D super-resolution. The proposed generator has a in-plane and out-of-plane Attention (IOA) network that fuses both in-plane and out-plane features of MRI dynamically. In terms of out-of-plane attention, the network merges features by considering the similarity distance between features and for in-plane attention, the network employs a two-level pyramid structure with varying receptive fields to extract features at different scales, ensuring the inclusion of both global and local features. Subsequently, to achieve 3D MRI super-resolution, a pre-trained slice interpolation network is used that takes two consecutive super-resolved MRI slices to generate a new intermediate slice. To further enhance the network performance and perceptual quality, we introduce a feature up-sampling layer and a feature extraction block with Scaled Exponential Linear Unit (SeLU). Moreover, our super-resolution network incorporates VGG loss from a fine-tuned VGG-19 network to provide additional enhancement. Through experimental evaluations on the IXI dataset and BRATS dataset, using the peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM) and the number of training parameters, we demonstrate the superior performance of our method compared to the existing techniques. Also, the proposed model can be adapted or modified to achieve super-resolution for both 2D and 3D MRI data.

A novel DbneAlexnet with Gazelle Hunting Optimization Algorithm enabled wild animal detection in WMSN data communication in IoT environment

SummaryThe main goal of this strategy is to detect the occurrence of wild animals in farmsteads and human habitats using an improved deep learning in order to avoid loss of life and damage to agricultural crops. This paper formulates a technique for recognizing wild animals in an Internet of Things (IoT) environment utilizing deep batch normalized exponential linear unit AlexNet (DbneAlexnet) and the Gazelle Hunting Optimization Algorithm (GHOA). The IoT‐Multimedia Sensor Networks (WMSN) network is first simulated, with the IoT nodes capturing the images needed for wild animal identification. The suggested GHOA is used for routing, which sends the captured images to the base station (BS). The input wild animal image is transmitted to image preprocessing at the BS, where it is processed using the Weiner filter (WF) to remove undesirable noise from the image. The denoised output is sent into salient map extraction, which extracts salient map to determine the regions that are conspicuous or noticeable at every place in the field of vision and guides the selection of attended location. Finally, the saliency map is forwarded to DbneAlexnet for detecting the wild animals, where the DbneAlexnet is trained using proposed GHOA. The suggested GHOA algorithm is created by combining the Gazelle Optimization Algorithm (GOA) and the deer hunting optimization algorithm (DHOA). Furthermore, detection is appraised by precision, recall, and f1‐score, which provide values of 90.2%, 89.1%, and 89.6%, respectively.

Defect Detection of Cell Phone Screen Using a Faster Regional Convolutional Neural Network with Multi-head Attention Mechanism

Since the glass outer screen of a cell phone is the main sensory part of the human eye when using a cell phone, the advantages and disadvantages of the cell phone screen directly affect people's sense of use. Therefore, the defect detection requirements for cell phone screens are high and need to meet the needs of high-volume factory inspection. Most of the traditional defect detection methods use visual methods, the detection results are overly dependent on the subjectivity and experience of workers, the efficiency of this method is low, and the accuracy is poor. Currently, machine learning-based detection methods are applied in numerous industries. In this paper, a faster Regional Convolutional Neural Network (R-CNN) with multi-head attention mechanism for defect detection of cell phone screen is proposed. To enhance the network's capability in extracting feature information, a four-head attention mechanism is added to the last convolutional layer of the ResNet50 network. An improved Region of Interest (ROI) Align is proposed to replace the original ROI Pooling to reduce the localization error of cell phone screen defects. Replace the original Rectified Linear Unit (ReLU) activation function with the Copy Exponential Linear Unit (CELU) activation function to expedite the convergence capability of the network. Finally, by comparing with other classical model training, the evaluation results indicate that the proposed method achieved an average accuracy of 95.71%, which is a 5.34% improvement compared to the original faster R-CNN network.

OOA-modified Bi-LSTM network: An effective intrusion detection framework for IoT systems

Currently, the Internet of Things (IoT) generates a huge amount of traffic data in communication and information technology. The diversification and integration of IoT applications and terminals make IoT vulnerable to intrusion attacks. Therefore, it is necessary to develop an efficient Intrusion Detection System (IDS) that guarantees the reliability, integrity, and security of IoT systems. The detection of intrusion is considered a challenging task because of inappropriate features existing in the input data and the slow training process. In order to address these issues, an effective meta heuristic based feature selection and deep learning techniques are developed for enhancing the IDS. The Osprey Optimization Algorithm (OOA) based feature selection is proposed for selecting the highly informative features from the input which leads to an effective differentiation among the normal and attack traffic of network. Moreover, the traditional sigmoid and tangent activation functions are replaced with the Exponential Linear Unit (ELU) activation function to propose the modified Bi-directional Long Short Term Memory (Bi-LSTM). The modified Bi-LSTM is used for classifying the types of intrusion attacks. The ELU activation function makes gradients extremely large during back-propagation and leads to faster learning. This research is analysed in three different datasets such as N-BaIoT, Canadian Institute for Cybersecurity Intrusion Detection Dataset 2017 (CICIDS-2017), and ToN-IoT datasets. The empirical investigation states that the proposed framework obtains impressive detection accuracy of 99.98 %, 99.97 % and 99.88 % on the N-BaIoT, CICIDS-2017, and ToN-IoT datasets, respectively. Compared to peer frameworks, this framework obtains high detection accuracy with better interpretability and reduced processing time.

Infrared image target detection for substation electrical equipment based on improved faster region-based convolutional neural network algorithm.

Substation electrical equipment generates a massive number of infrared images during operation. However, the overall quality of the infrared images is low and it lacks image detail information. When using traditional target detection algorithms for detection, feature extraction poses great difficulties. Therefore, to address this problem, this paper proposes a target detection algorithm based on the improved faster region-based convolutional neural network (Faster R-CNN). It achieves the correct identification of different types of electrical equipment in infrared images. First, the algorithm improves the backbone network of Faster R-CNN for feature extraction. An InResNet structure is proposed to replace the residual block structure of the original ResNet-34 network, which enhances the richness of feature extraction. Second, the rectified linear unit activation function in the original feature extraction network is replaced by the exponential linear unit activation function, and group normalization is used instead of batch normalization as the network normalization method. Then, the dense connection structure is introduced into the ResNet-34 network, and the whole network is called residual dense connection network. Finally, the improved Faster R-CNN is compared to the original Faster R-CNN, a single-shot multibox detector, and you only look once v3 plus spatial pyramid pooling. The experimental results show that the improved algorithm has the highest mean average precision and average recall for most of the substation electrical equipment in infrared images. Moreover, from the confidence level of the detected electrical equipment and the accuracy of the prediction box, the improved Faster R-CNN has the best performance.

TabTranSELU: A transformer adaptation for solving tabular data

Tabular data are most prevalent datasets in real world, yet the integration of deep learning algorithms in tabular data often garners less attention despite their widespread utilization in other field. This phenomenon could be attributed to the dominance of the classical algorithms in their simplicity and interpretability, and the superior performance of the gradient boosting tree models in tabular data. In this paper, a simple yet affective adaptation of the Transformer architecture tailored specifically for tabular data is presented, not only achieving good performance but also retains a high degree of explain ability. The model encodes both continuous and categorical features, alongside their respective names, and feed them into an enhanced Transformer structure enriched with Scaled Exponential Linear Unit activation. Through rigorous experimentation, our model not only outperforms classical algorithms and similar Transformer-based counterparts, but also are comparable to the performance of gradient boosting tree models.

Enhancing deterministic prediction in unidirectional ocean waves using an Artificial Neural Network with exponential linear unit

The phase-resolved wave prediction based on physical methods is difficult to ensure both accuracy and efficiency simultaneously. In recent years, Artificial Neural Network (ANN) has been successfully utilized for deterministic ocean wave prediction with efficiency and high precision. In this work, an ANN with the exponential linear unit (ELU) activation function is developed for deterministic wave prediction in unidirectional sea states. High-order Spectral method is used to generate the data on wave elevation. Instead of the traditional rectified linear unit (ReLU), we propose to use ELU as the activation function in the present ANN. Numerical results show that the ANN with the ELU activation function provides a more accurate prediction than the ANN with the ReLU activation function. Based on wave elevation measured at a fixed position, wave elevation prediction at another fixed position and wave elevation prediction in a fixed zone, are both studied by making use of the ANN with the ELU. Moreover, the factors including sea state and time length of wave elevation measured at a fixed position are considered to demonstrate the efficiency of the prediction results.

Classifying Coal Mine Pillar Stability Areas with Multiclass SVM on Ensemble Learning Models

Pillars are key structural components in coal mining. The safety requirements of underground coal mines are non-negotiable. Accurately classifying the areas of pillar stability helps ensure safety in coal mines. This study aimed to classify new pillar stability categories and their stability areas. The multiclass support vector machine (SVM) method was implemented with two types of kernel functions (polynomial and radial basis function (RBF) kernels) on pillar stability data with four new categories: failed or intact, either with or without an appropriate safety factor. This classification uses three basic ensemble learning models: Artificial Neural Network-Backpropagation Rectified Linear Unit, Artificial Neural Network-Backpropagation Exponential Linear Unit, and Artificial Neural Network-Backpropagation Gaussian Error Linear Unit. The results with four data proportions and ten experiments had an average accuracy and standard deviation of 92.98% and 0.56%-1.64% respectively. The accuracies of the multiclass SVM method using the polynomial kernel and the RBF kernel with Bayesian parameter optimization to classify the areas of pillar stability were 91% and 92%, respectively. The multiclass SVM method with the RBF kernel captured 96.6% of potentially dangerous pillars. The visualization of classification areas showed that areas with intact pillars may also have failed pillars.

L-Net: A lightweight convolutional neural network for devices with low computing power

Deep learning (DL) has demonstrated exceptional success across various domains, including computer vision, natural language processing, and speech recognition. However, the training and inference processes of DL models typically require substantial computational resources and storage space, presenting a significant challenge within the Internet of Things (IoT) domain. This study contributes theoretically to the field of lightweight DL by proposing L-Net, a lightweight convolutional neural network designed specifically for low-compute devices. The L-Net addresses challenges associated with channel interaction disparities and vanishing gradients. To further improve the network performance, we introduce the residual enhanced channel attention (or R-ECA) module, which combines a bypass mechanism derived from simplified residual learning with the attention mechanism's cross-channel interaction. Additionally, we replace the rectified linear unit function (or ReLU) with an exponential linear unit (or ELU) function to enhance the network's nonlinear expression capability and training speed. We conducted object recognition experiments and compared the accuracy and prediction stability of L-Net with well-known models, such as AlexNet, VGG11, SqueezeNet, ResNet, and MobileNet, to assess its efficacy. Using the CIFAR-10 dataset and our custom dataset of apple tree leaf diseases, our experimental results demonstrate that, with relatively smaller model parameters, L-Net performs exceptionally well in terms of mean Average Precision (mAP), achieving 0.906. Furthermore, when applied to our custom dataset, L-Net exhibits relatively consistent performance across various dataset splits under different ratios, outperforming the majority of models.

Enhanced SSD framework for detecting defects in cigarette appearance using variational Bayesian inference under limited sample conditions.

In high-speed cigarette manufacturing industries, occasional minor cosmetic cigarette defects and a scarcity of samples significantly hinder the rapid and accurate detection of defects. To tackle this challenge, we propose an enhanced single-shot multibox detector (SSD) model that uses variational Bayesian inference for improved detection of tiny defects given sporadic occurrences and limited samples. The enhanced SSD model incorporates a bounded intersection over union (BIoU) loss function to reduce sensitivity to minor deviations and uses exponential linear unit (ELU) and leaky rectified linear unit (ReLU) activation functions to mitigate vanishing gradients and neuron death in deep neural networks. Empirical results show that the enhanced SSD300 and SSD512 models increase the model's detection accuracy mean average precision (mAP) by up to 1.2% for small defects. Ablation studies further reveal that the model's mAP increases by 1.5%, which reduces the computational requirements by 5.92 GFLOPs. The model also shows improved inference in scenarios with limited samples, thus highlighting its effectiveness and applicability in high-speed, precision-oriented cigarette manufacturing industries.

A deep convolutional neural network architecture for breast mass classification using mammogram images

<p>Breast cancer is one of the second most common cancer occurring worldwide. Early identification of the disease is a major interest that promises to propose several diagnostic procedures to prevent further surgical interventions. This research paper aims to develop a breast mass classifier system using deep learning to differentiate breast mass images from normal mammographic images. The benchmark mammographic datasets CBIS-DDSM, INbreast, and mini-MIAS are used for constructing the proposed model DELU-BM-CNN. The region of interest is identified by applying image processing techniques (median filter, binarization and dilation) and the images are enhanced and sharpened using adaptive histogram equalization and unsharp masking techniques. The pre-processed images are trained with a minimum of five deep convolutional layers activated by an Exponential Linear Unit (ELU) which is developed from scratch for feature learning and classifying the given whole mammographic images. Dropout, Data normalization, and Global average pooling are some of the regularization techniques adopted to prevent the model from over-fitting. The proposed models are able to classify CBIS-DDSM images with an accuracy of 96.60%, INbreast images with 96.20% and MIAS images with 97.40%. The experimental results are also compared with conventional Rectified Linear Unit (ReLU) and Leaky ReLU activation function that promises the proposed model as a good prognosticator than the state-of-art models for cancer diagnosis using mammogram images as input.</p>