Scaled Exponential Linear Unit Research Articles

MRI super-resolution using similarity distance and multi-scale receptive field based feature fusion GAN and pre-trained slice interpolation network

Challenges arise in achieving high-resolution Magnetic Resonance Imaging (MRI) to improve disease diagnosis accuracy due to limitations in hardware, patient discomfort, long acquisition times, and high costs. While Convolutional Neural Networks (CNNs) have shown promising results in MRI super-resolution, they often don't look into the structural similarity and prior information available in consecutive MRI slices. By leveraging information from sequential slices, more robust features can be obtained, potentially leading to higher-quality MRI slices. We propose a multi-slice two-dimensional (2D) MRI super-resolution network that combines a Generative Adversarial Network (GAN) with feature fusion and a pre-trained slice interpolation network to achieve three-dimensional (3D) super-resolution. The proposed model requires consecutively acquired three low-resolution (LR) MRI slices along a specific axis, and achieves the reconstruction of the MRI slices in the remaining two axes. The network effectively enhances both in-plane and out-of-plane resolution along the sagittal axis while addressing computational and memory constraints in 3D super-resolution. The proposed generator has a in-plane and out-of-plane Attention (IOA) network that fuses both in-plane and out-plane features of MRI dynamically. In terms of out-of-plane attention, the network merges features by considering the similarity distance between features and for in-plane attention, the network employs a two-level pyramid structure with varying receptive fields to extract features at different scales, ensuring the inclusion of both global and local features. Subsequently, to achieve 3D MRI super-resolution, a pre-trained slice interpolation network is used that takes two consecutive super-resolved MRI slices to generate a new intermediate slice. To further enhance the network performance and perceptual quality, we introduce a feature up-sampling layer and a feature extraction block with Scaled Exponential Linear Unit (SeLU). Moreover, our super-resolution network incorporates VGG loss from a fine-tuned VGG-19 network to provide additional enhancement. Through experimental evaluations on the IXI dataset and BRATS dataset, using the peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM) and the number of training parameters, we demonstrate the superior performance of our method compared to the existing techniques. Also, the proposed model can be adapted or modified to achieve super-resolution for both 2D and 3D MRI data.

MAGNETIC RESONANCE IMAGE DENOIZING USING A DUAL-CHANNEL DISCRIMINATIVE DENOIZING NETWORK

Magnetic Resonance Imaging (MRI) provides detailed information about soft tissues, which is essential for disease analysis. However, the presence of Rician noise in MR images introduces uncertainties that challenge medical practitioners during analysis. The objective of our research paper is to introduce an innovative dual-channel deep learning (DL) model designed to effectively denoize MR images. The methodology of this model integrates two distinct pathways, each equipped with unique normalization and activation techniques, facilitating the creation of a wide range of image features. Specifically, we employ Group Normalization in combination with Parametric Rectified Linear Units (PRELU) and Local Response Normalizations (LRN) alongside Scaled Exponential Linear Units (SELU) within both channels of our denoizing network. The outcomes of our proposed network exhibit clinical relevance, empowering medical professionals to conduct more efficient disease analysis. When evaluated by experienced radiologists, our results were deemed satisfactory. The network achieved a noteworthy improvement in performance metrics without requiring retraining. Specifically, there was a (6.65 ± 0.03)% enhancement in Peak Signal-to-Noise Ratio (PSNR) values and a (3.9 ± 0.02)% improvement in Structural Similarity Index (SSIM) values. Furthermore, when evaluated on the dataset on which the network was initially trained, the increase in PSNR and SSIM values was even more pronounced, with a (7.5 ± 0.02)% improvement in PSNR and a (4.3 ± 0.01)% enhancement in SSIM. Evaluation metrics, such as SSIM and PSNR, demonstrated a notable enhancement in the results obtained using our network. The statistical significance of our findings is evident, with p-values consistently less than 0.05 (p < 0.05). Importantly, our network demonstrates exceptional generalizability, as it performs remarkably well on different datasets without the need for retraining.

QC_SANE: Robust Control in DRL Using Quantile Critic With Spiking Actor and Normalized Ensemble.

Recently introduced deep reinforcement learning (DRL) techniques in discrete-time have resulted in significant advances in online games, robotics, and so on. Inspired from recent developments, we have proposed an approach referred to as Quantile Critic with Spiking Actor and Normalized Ensemble (QC_SANE) for continuous control problems, which uses quantile loss to train critic and a spiking neural network (NN) to train an ensemble of actors. The NN does an internal normalization using a scaled exponential linear unit (SELU) activation function and ensures robustness. The empirical study on multijoint dynamics with contact (MuJoCo)-based environments shows improved training and test results than the state-of-the-art approach: population coded spiking actor network (PopSAN).

Performance Analysis of Neural Network Architecture in Developing Real-Time Malaysian River Water Quality Model

According to the world health organization, 485,000 people died each year due to water-related health diseases which are mainly contributed by poor river water quality. As a result, water quality monitoring stations have been deployed across the world. Unfortunately, due to the complex nature of the off-site water quality parameters, the water quality index (WQI) cannot be assessed in real-time. This has led to a significant push for the scientific community to develop an accurate and robust water quality prediction model. The dynamic and nonlinear nature of water quality parameters are major challenges for traditional machine learning algorithms such as multi linear regression to capture. In this study, the water quality index prediction model was developed using the feedforward artificial neural network (FANN) algorithms utilizing only on-site parameters. The performance of different nonlinear activation functions in the hidden neurons was thoroughly analysed which includes rectified linear unit (ReLU), scaled exponential linear unit (SELU), and exponential linear unit (ELU). Additionally, various initialization and optimization algorithms were also evaluated for maximum performance and efficiency. The results shows that FANN-ELU model coupled with Glorot initialization technique and AdaGrad optimizer outperformed other model combinations with an R2 value of 0.88 and mean squared error (MSE) value of 22.74.

A New Fault Diagnosis Method for Unbalanced Data Based on 1DCNN and L2-SVM

In general, the measured health condition data from rolling bearings usually exhibit imbalanced distribution. However, traditional intelligent fault diagnosis methods usually assume that the data categories are balanced. To improve the diagnosis accuracy of unbalanced datasets, a new fault diagnosis method for unbalanced data based on 1DCNN and L2-SVM is proposed in this paper. Firstly, to prevent the minority class samples from being heavily suppressed by the rectified linear unit (ReLU) activation function in the traditional convolutional neural network (CNN), ReLU is improved by linear and scaled exponential linear units (SELUs). Secondly, to solve the problem where the cross-entropy loss treats all input samples equally, it is replaced by the L2-support vector machine (L2-SVM) loss. Furthermore, a dynamic adjustment parameter is introduced to assign less misclassification cost to the majority of class samples. Finally, we add a new modulation factor that reduces the weight of more distinguishable samples to generate more focus on training indiscernible samples. The proposed method is carried out on two kinds of bearing datasets. The experimental results illustrate a significant improvement in recognition accuracy and the higher diagnosis performance of the model when dealing with unbalanced data compared with other intelligent methods.

Lung Cancer Nodules Detection via an Adaptive Boosting Algorithm Based on Self-Normalized Multiview Convolutional Neural Network.

Lung cancer is the deadliest cancer killing almost 1.8 million people in 2020. The new cases are expanding alarmingly. Early lung cancer manifests itself in the form of nodules in the lungs. One of the most widely used techniques for both lung cancer early and noninvasive diagnosis is computed tomography (CT). However, the intensive workload of radiologists to read a large number of scans for nodules detection gives rise to issues like false detection and missed detection. To overcome these issues, we proposed an innovative strategy titled adaptive boosting self-normalized multiview convolution neural network (AdaBoost-SNMV-CNN) for lung cancer nodules detection across CT scans. In AdaBoost-SNMV-CNN, MV-CNN function as a baseline learner while the scaled exponential linear unit (SELU) activation function normalizes the layers by considering their neighbors' information and a special drop-out technique (α-dropout). The proposed method was trained and tested using the widely Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) and Early Lung Cancer Action Program (ELCAP) datasets. AdaBoost-SNMV-CNN achieved an accuracy of 92%, sensitivity of 93%, and specificity of 92% for lung nodules detection on the LIDC-IDRI dataset. Meanwhile, on the ELCAP dataset, the accuracy for detecting lung nodules was 99%, sensitivity 100%, and specificity 98%. AdaBoost-SNMV-CNN outperformed the majority of the model in accuracy, sensitivity, and specificity. The multiviews confer the model's good generalization and learning ability for diverse features of lung nodules, the model architecture is simple, and has a minimal computational time of around 102 minutes. We believe that AdaBoost-SNMV-CNN has good accuracy for the detection of lung nodules and anticipate its potential application in the noninvasive clinical diagnosis of lung cancer. This model can be of good assistance to the radiologist and will be of interest to researchers involved in the designing and development of advanced systems for the detection of lung nodules to accomplish the goal of noninvasive diagnosis of lung cancer.

A Novel Multi-Task Learning Model with PSAE Network for Simultaneous Estimation of Surface Quality and Tool Wear in Milling of Nickel-Based Superalloy Haynes 230.

For data-driven intelligent manufacturing, many important in-process parameters should be estimated simultaneously to control the machining precision of the parts. However, as two of the most important in-process parameters, there is a lack of multi-task learning (MTL) model for simultaneous estimation of surface roughness and tool wear. To address the problem, a new MTL model with shared layers and two task-specific layers was proposed. A novel parallel-stacked auto-encoder (PSAE) network based on stacked denoising auto-encoder (SDAE) and stacked contractive auto-encoder (SCAE) was designed as the shared layers to learn deep features from cutting force signals. To enhance the performance of the MTL model, the scaled exponential linear unit (SELU) was introduced as the activation function of SDAE. Moreover, a dynamic weight averaging (DWA) strategy was implemented to dynamically adjust the learning rate of different tasks. Then, the time-domain features were extracted from raw cutting signals and low-frequency reconstructed wavelet packet coefficients. Frequency-domain features were extracted from the power spectrum obtained by the Fourier transform. After that, all features were combined as the input vectors of the proposed MTL model. Finally, surface roughness and tool wear were simultaneously predicted by the trained MTL model. To verify the superiority and effectiveness of the proposed MTL model, nickel-based superalloy Haynes 230 was machined under different cutting parameter combinations and tool wear levels. Some other intelligent algorithms were also implemented to predict surface roughness and tool wear. The results showed that compared with the support vector regression (SVR), kernel extreme learning machine (KELM), MTL with SDAE (MTL_SDAE), MTL with SCAE (MTL_SCAE), and single-task learning with PSAE (STL_PSAE), the estimation accuracy of surface roughness was improved by 30.82%, 16.67%, 14.06%, 26.17%, and 16.67%, respectively. Meanwhile, the prediction accuracy of tool wear was improved by 46.74%, 39.57%, 41.51%, 38.68%, and 39.57%, respectively. For practical engineering application, the dimensional deviation and surface quality of the machined parts can be controlled through the established MTL model.

AdaBoost maximum entropy deep inverse reinforcement learning with truncated gradient

Studying the representational capacity of neural networks to learn nonlinear rewards is necessary in a complex and nonlinear environment. Over recent years, the maximum entropy deep inverse reinforcement learning algorithm (ME-DIRL) has been increasingly applied to the learning of nonlinear rewards. However, under cases of limited and imbalanced expert demonstration data, complex calculations, or overfitting, the learning nonlinear rewards remains a challenging problem. A novel ME-DIRL with AdaBoost algorithm (AME-DIRL) is our proposed solution. The focus of AME-DIRL is to utilize the AdaBoost algorithm. This combines multiple ME-DIRL processes to form a strong learner and thus overcome the imbalance of the data set. Furthermore, to deal with the complex calculations in AME-DIRL, a truncated gradient (TG) method is applied for getting the sparse rewards obtained by the strong learner, thus reducing the model complexity. To prevent overfitting, a correction factor is then added to the linear combination of weak learners. AME-DIRL models the relationship between input features and output rewards. Rewards are approximated by means of a convolutional neural network (CNN) with scaled exponential linear units (SELUs). Numerical results indicate that our proposed AME-DIRL shows higher accuracy in learning rewards when compared with several classical inverse reinforcement learning (IRL) algorithms.

Robust Beamforming Based on Complex-Valued Convolutional Neural Networks for Sensor Arrays

Robust adaptive beamforming (RAB) plays a vital role in modern communications by ensuring the reception of high-quality signals. This paper proposes a deep learning approach to robust adaptive beamforming. In particular, we propose a novel RAB approach where the sample covariance matrix (SCM) is used as the input of a deep 1D Complex-Valued Convolutional Neural Network (CVCNN). The network employs complex convolutional and pooling layers, as well as a Cartesian Scaled Exponential Linear Unit activation function to directly compute the nearly-optimum weight vector through the training process and without prior knowledge about the direction of arrival of the desired signal. This means that reconstruction of the interference plus noise (IPN) covariance matrix is not required. The trained CVCNN accurately computes the nearly-optimum weight vector for data not used during training. The computed weight vector is employed to estimate the signal-to-interference plus noise ratio. Simulations show that the proposed RAB can provide performance close to that of the optimal beamformer.

A data-driven method for dynamic load forecasting of scraper conveyer based on rough set and multilayered self-normalizing gated recurrent network

The dynamic load forecasting of scraper conveyer is one of the key problems that need to be solved in unmanned coal mining. The dynamic load forecasting system of scraper conveyer is a complex, multivariable, and nonlinear system, and there are coupling relations between every variable. It is very difficult to establish precise mathematic model. Therefore, based on rough set and the gated recurrent units (GRU), this study proposes a data-driven method for dynamic load forecasting of scraper conveyer based on rough set and multilayered self-normalizing GRU network. First, the rough set was applied to carry on for a variety of factors affecting load forecasting of scraper conveyer to optimize the model input, and the importance of each attribute for load of scraper conveyer was obtained. Then, a multilayered self-normalizing gated recurrent units (MS-GRU) model is proposed for the dynamic load forecasting of scraper conveyer. This model introduces scaled exponential linear units (SELU) activation function to squash the hidden states to calculate the output of the model, and the exploding and vanishing gradient problem are overcome in a stacked GRU neural network. Finally, an experiment is applied to verify the proposed model in this paper. The experimental results show that this article Compared with the existing methods, the model shows a higher accuracy rate 95.8%, which can well complete the prediction of the operating parameters of the shearer.

Artificial Intelligence-Enabled ECG Algorithm Based on Improved Residual Network for Wearable ECG

Heart disease is the leading cause of death for men and women globally. The residual network (ResNet) evolution of electrocardiogram (ECG) technology has contributed to our understanding of cardiac physiology. We propose an artificial intelligence-enabled ECG algorithm based on an improved ResNet for a wearable ECG. The system hardware consists of a wearable ECG with conductive fabric electrodes, a wireless ECG acquisition module, a mobile terminal App, and a cloud diagnostic platform. The algorithm adopted in this study is based on an improved ResNet for the rapid classification of different types of arrhythmia. First, we visualize ECG data and convert one-dimensional ECG signals into two-dimensional images using Gramian angular fields. Then, we improve the ResNet-50 network model, add multistage shortcut branches to the network, and optimize the residual block. The ReLu activation function is replaced by a scaled exponential linear units (SELUs) activation function to improve the expression ability of the model. Finally, the images are input into the improved ResNet network for classification. The average recognition rate of this classification algorithm against seven types of arrhythmia signals (atrial fibrillation, atrial premature beat, ventricular premature beat, normal beat, ventricular tachycardia, atrial tachycardia, and sinus bradycardia) is 98.3%.

Convolutional neural network for transient grating frequency-resolved optical gating trace retrieval and its algorithm optimization* *Project supported by the National Key R&D Program of China (Grant No. 2017YFB0405202) and the National Natural Science Foundation of China (Grant Nos. 61690221, 91850209, and 11774277).

A convolutional neural network is employed to retrieve the time-domain envelop and phase of few-cycle femtosecond pulses from transient-grating frequency-resolved optical gating (TG-FROG) traces. We use theoretically generated TG-FROG traces to complete supervised trainings of the convolutional neural networks, then use similarly generated traces not included in the training dataset to test how well the networks are trained. Accurate retrieval of such traces by the neural network is realized. In our case, we find that networks with exponential linear unit (ELU) activation function perform better than those with leaky rectified linear unit (LRELU) and scaled exponential linear unit (SELU). Finally, the issues that need to be addressed for the retrieval of experimental data by this method are discussed.

Fault Diagnosis of a Rotor-Bearing System Under Variable Rotating Speeds Using Two-Stage Parameter Transfer and Infrared Thermal Images

Current fault diagnosis methods for rotor-bearing system are mostly based on analyzing the vibration signals collected at steady rotating speeds. In those methods, the data collected under one operating condition cannot be accurately used for diagnosis under a different condition. Moreover, in vibration monitoring, installing the necessary sensors will affect the equipment structure and hence the vibration response itself. The present paper proposes a new method based on two-stage parameter transfer and infrared thermal images for fault diagnosis of rotor-bearing system under variable rotating speeds. The method of parameter transfer enables the use of data (or parameters) acquired under one operating condition (called the source domain) to be extended for use in a different operating condition (called the target domain). First, scaled exponential linear unit (SELU) and modified stochastic gradient descent (MSGD) are used to construct an enhanced convolutional neural network (ECNN). Second, a stacked convolutional auto-encoder (CAE) trained based on unlabeled source-domain thermal images is employed to initialize a source-domain ECNN. Third, model parameters from the pre-trained source-domain ECNN are transferred to the target-domain ECNN to adapt to the characteristics of the target domain. The collected thermal images for a rotor-bearing system under variable speeds are used to test the transfer diagnosis performance of the proposed method. The experimental results demonstrate the performance improvement and the advantages of the proposed method.

Trainable activation function with differentiable negative side and adaptable rectified point

Activation function is one of the keys for Artificial Neural Network in learning complex mapping function. A recently proposed activation function called Scaled Exponential Linear Unit (SELU) has a unique characteristic in the ability to automatically normalize its output toward predefined mean and variance. In this paper, we introduce Parametric Scaled Exponential Linear Unit (PSELU), a modification of SELU where the parameters are adaptively learned during the training phase via backpropagation algorithm. We then add further modifications with the aim of having a stronger gradient in the negative part and produce more negative output from our proposed method. Our proposed method is evaluated using various artificial neural network models on diverse image classification tasks. In addition, we also measure the performance of our proposed method in comparison with some other popular activation functions. The evaluation results in this paper present some empirical proof of improvement in term of network generalization performance. Moreover, our experiments strengthen our initial hypothesis regarding the advantages obtained from using our proposed method. Finally, the behavior of the trained parameters along with the training phase dynamic is observed in order to better understand the increased performance of our proposed method

Intelligent fault diagnosis among different rotating machines using novel stacked transfer auto-encoder optimized by PSO

Intelligent fault diagnosis techniques cross rotating machines have great significances in theory and engineering For this purpose, this paper presents a novel method using novel stacked transfer auto-encoder (NSTAE) optimized by particle swarm optimization (PSO). First, novel stacked auto-encoder (NSAE) model is designed with scaled exponential linear unit (SELU), correntropy and nonnegative constraint. Then, NSTAE is constructed using NSAE and parameter transfer strategy to enable the pre-trained source-domain NSAE to adapt to the target-domain samples. Finally, PSO is used to flexibly decide the hyperparameters of NSTAE. The effectiveness and superiority of the presented method are investigated through analyzing the collected experimental data of bearings and gears from different rotating machines.

An enhanced convolutional neural network for bearing fault diagnosis based on time–frequency image

Deep learning theory has been widely used for diagnosing bearing faults. However, this method still has same drawbacks. For example, single time or frequency domain analysis methods cannot effectively extract features, the ReLU function is greatly affected by the learning rate, and it is difficult to achieve satisfactory results using the same regularization for different layers. To overcome the aforementioned deficiencies: (1) short-time Fourier transform theory to obtain an input image, (2) the scaled exponential linear unit (SELU) function is introduced to avoid excessive “dead” nodes during the training process, and (3) the use of hierarchical regularization to obtain better training results. Small sample datasets were used for the test experiment in two bearing fault simulators. The experiment results showed that the proposed method has a higher fault diagnosis accuracy than existing deep learning diagnosis methods.

DE-Net: Deep Encoding Network for Building Extraction from High-Resolution Remote Sensing Imagery

Deep convolutional neural networks have promoted significant progress in building extraction from high-resolution remote sensing imagery. Although most of such work focuses on modifying existing image segmentation networks in computer vision, we propose a new network in this paper, Deep Encoding Network (DE-Net), that is designed for the very problem based on many lately introduced techniques in image segmentation. Four modules are used to construct DE-Net: the inceptionstyle downsampling modules combining a striding convolution layer and a max-pooling layer, the encoding modules comprising six linear residual blocks with a scaled exponential linear unit (SELU) activation function, the compressing modules reducing the feature channels, and a densely upsampling module that enables the network to encode spatial information inside feature maps. Thus, DE-Net achieves stateoftheart performance on the WHU Building Dataset in recall, F1-Score, and intersection over union (IoU) metrics without pretraining. It also outperformed several segmentation networks in our self-built Suzhou Satellite Building Dataset. The experimental results validate the effectiveness of DE-Net on building extraction from aerial imagery and satellite imagery. It also suggests that given enough training data, designing and training a network from scratch may excel fine-tuning models pre-trained on datasets unrelated to building extraction.

Evaluation of maxout activations in deep learning across several big data domains

This study investigates the effectiveness of multiple maxout activation function variants on 18 datasets using Convolutional Neural Networks. A network with maxout activation has a higher number of trainable parameters compared to networks with traditional activation functions. However, it is not clear if the activation function itself or the increase in the number of trainable parameters is responsible in yielding the best performance for different entity recognition tasks. This paper investigates if an increase in the number of convolutional filters on traditional activation functions performs equal-to or better-than maxout networks. Our experiments compare the Rectified Linear Unit, Leaky Rectified Linear Unit, Scaled Exponential Linear Unit, and Hyperbolic Tangent activations to four maxout function variants. We observe that maxout networks train relatively slower than networks with traditional activation functions, e.g. Rectified Linear Unit. In addition, we found that on average, across all datasets, the Rectified Linear Unit activation function performs better than any maxout activation when the number of convolutional filters is increased. Furthermore, adding more filters enhances the classification accuracy of the Rectified Linear Unit networks, without adversely affecting their advantage over maxout activations with respect to network-training speed.

Population Statistics Algorithm Based on MobileNet

In today’s society, intelligent video surveillance plays an important role in social security, traffic scheduling, national security and other fields. One of the research hotspots is people statistics based on image processing, which has strategic significance in practical applications. Aimed at the problem that the low accuracy in the actual application scenario, the limited hardware resources, and the low operation efficiency, this paper proposes a multi-feature target detection model based on the lightweight deep learning network MobileNet [1], which can be used in intelligent terminals. The basic feature-extraction network MobileNet as a lightweight network can provide a flexible alternative configuration in terms of efficiency and accuracy. The underlying detection network selects a single deep nerual network, named SSD [2]. The algorithm can achieve multi-scale target detection, and uses the target position and category to perform one-time regression. In this paper, the activation function of SSD is changed into SeLU (scaled exponential linear units) [3], which improves the robustness of the algorithm. At the same time, the work of sample diversity and data enhancement has been made, and the characteristics of the human body above the shoulders have been fully utilized. Experiments have shown that the improved network structure based on MobileNet has higher detection accuracy, lower delay, excellent robustness, while the number of model parameters is effectively reduced.

XNORNet and Minimum Barrier Detection for Efficient Face Recognition.

Deep Convolutional Neural Networks (CNNs) have been extremely effective in many areas of computer vision, particularly in the domain of facial recognition where many CNN models have delivered a near perfect performance on long-standing benchmarks. A major challenge being faced by CNN architectures today is their need for a large amount of memory and processing power to deliver their exceptional performance. This hampers their deployment on less powerful devices like smart-phones whose ubiquity is unmatched. This paper critically analyzes a CNN based network architecture wherein convolutions are approximated by XNOR operations and bit-counts which are primarily binary operations hence making the network much faster and more memory efficient than standard CNN architectures. The aim of this paper is to assess the performance of the aforementioned architecture for the task of facial recognition with Minimum Barrier Salient Object Detection (MBD) as a preprocessing step and to examine the affect self-normalizing properties induced by the Scaled Exponential Linear Unit (SELU) activation function on the convergence of the network and benchmark it against the convergence performance of the same network incorporating the widely used Rectified Linear Unit (RELU) as its activation function. Two widely known datasets, Faces95 and Faces96, pre-processed using Minimum Barrier Detection (MBD) for saliency detection were used benchmarking the performance of the proposed networks. It was observed that the network using SELU converged faster than RELU for the Faces95 dataset and both converged at similar rates for Faces96 dataset. The network with SELU gave better accuracy than RELU for both the datasets.