Leaky ReLU Research Articles

Construction and Simulation of Market Risk Warning Model Based on Deep Learning

The market is intricate and complicated, and the existing risk warning models have problems of low efficiency and poor generalization in predicting market risk data. Aiming at the problems, this study takes stock market risk warning as the research object and proposes a market risk warning model based on LSTM-VaR. 15 variables in the three categories of basic transaction data, statistical technical indicators, and moving interval data are selected as the stock market characteristic indicators, the LSTM(Long Short Term Memory) prediction model is constructed and the standard deviation of stock returns is predicted. Based on the predicted results, the probability distribution of return rate under the conditional distribution is obtained, and the VaR(Value at Risk) is calculated. 1% and 5% sample quantiles are taken as the warning line, and the LSTM-VaR warning model is obtained. The results show that the RMSE value of the model is the smallest, which is 0.013762, when the activation function of the LSTM-VaR model is the Leaky ReLu function, the training periods epochs are 10, the time window length N is 9, the batch size is 8, the number of neurons in each layer is 50, the dropout probability is 0.1, and adam is used as the optimizer. Compared with traditional prediction models such as MLP, the proposed model has better performance and can well realize market risk warning.

The Application of a Pavement Distress Detection Method Based on FS-Net

In order to solve the problem of difficulties in pavement distress detection in the field of pavement maintenance, a pavement distress detection algorithm based on a new deep learning method is proposed. Firstly, an image data set of pavement distress is constructed, including large-scale image acquisition, expansion and distress labeling; secondly, the FReLU structure is used to replace the leaky ReLU activation function to improve the ability of two-dimensional spatial feature capture; finally, in order to improve the detection ability of this model for long strip pavement distress, the strip pooling method is used to replace the maximum pooling method commonly used in the existing network, and a new method is formed which integrates the FReLU structure and the strip pooling method, named FS-Net in this paper. The results show that the average accuracy of the proposed method is 4.96% and 3.67% higher than that of the faster R-CNN and YOLOv3 networks, respectively. The detection speed of 4 K images can reach about 12 FPS. The accuracy and computational efficiency can meet the actual needs in the field of road detection. A set of lightweight detection equipment for highway pavement was formed in this paper by purchasing hardware, developing software, designing brackets and packaging shells, and the FS-Net was burned into the equipment. The recognition rate of pavement distress is more than 90%, and the measurement error of the crack width is within ±0.5 mm through application testing. The lightweight detection equipment for highway pavement with burning of the pavement distress detection algorithm based on FS-Net can detect pavement conditions quickly and identify the distress and calculate the distress parameters, which provide a large amount of data support for the pavement maintenance department to make maintenance decisions.

Feature Enhancement-Based Ship Target Detection Method in Optical Remote Sensing Images

Ship targets in ORSIs (Optical Remote Sensing Images) have the characteristics of various scales, and most of them are medium and small-scale targets. When the existing target detection algorithms are applied to ship target detection in ORSIs, the detection accuracy is low. There are two main reasons for the above problems, one is the mismatch of the receptive fields, and the other is the lack of feature information. For resolving the problem that multi-scale ship targets are difficult to detect, this paper proposes a ship target detection algorithm based on feature enhancement. Firstly, EIRM (Elastic Inception Residual Module) is proposed for feature enhancement, which can capture feature information of different dimensions and provide receptive fields of different scales for mid- and low-level feature maps. Secondly, the SandGlass-L block is proposed by replacing the ReLu6 activation function of the SandGlass block with the Leaky ReLu activation function. Leaky ReLu solves the problem of 0 output when ReLu6 has negative input, so the SandGlass-L block can retain more feature information. Finally, based on SandGlass-L, SGLPANet (SandGlass-L Path Aggregation Network) is proposed to alleviate the problem of information loss caused by dimension transformation and retain more feature information. The backbone network of the algorithm in this paper is CSPDarknet53, and the SPP module and EIRM act after the backbone network. The neck network is SGLPANet. Experiments on the NWPU VHR-10 dataset show that the algorithm in this paper can well solve the problem of low detection accuracy caused by mismatched receptive fields and missing feature information. It not only improves the accuracy of ship target detection, but also achieves good results when extended to other categories. At the same time, the extended experiments on the LEVIR dataset show that the algorithm also has certain applicability on different datasets.

Lyapunov-Derived Control and Adaptive Update Laws for Inner and Outer Layer Weights of a Deep Neural Network

Lyapunov-based real-time update laws are well-known for neural network (NN)-based adaptive controllers that control nonlinear dynamical systems using single-hidden-layer NNs. However, developing real-time weight update laws for deep NNs (DNNs) remains an open question. This letter presents the first result with Lyapunov-based real-time weight adaptation laws for each layer of a feedforward DNN-based control architecture, with stability guarantees. Additionally, the developed method allows nonsmooth activation functions to be used in the DNN to facilitate improved transient performance. A nonsmooth Lyapunov-based stability analysis proves global asymptotic tracking error convergence. Simulation results are provided for a nonlinear system using DNNs with leaky rectified linear unit (LReLU) and hyperbolic tangent activation functions to demonstrate the efficacy of the developed method.

An Optimized Convolutional Neural Network with Combination Blocks for Chinese Sign Language Identification

(Aim) Chinese sign language is an essential tool for hearing-impaired to live, learn and communicate in deaf communities. Moreover, Chinese sign language plays a significant role in speech therapy and rehabilitation. Chinese sign language identification can provide convenience for those hearing impaired people and eliminate the communication barrier between the deaf community and the rest of society. Similar to the research of many biomedical image processing (such as automatic chest radiograph processing, diagnosis of chest radiological images, etc.), with the rapid development of artificial intelligence, especially deep learning technologies and algorithms, sign language image recognition ushered in the spring. This study aims to propose a novel sign language image recognition method based on an optimized convolutional neural network. (Method) Three different combinations of blocks: Conv-BN-ReLU-Pooling, Conv-BN-ReLU, Conv-BN-ReLU-BN were employed, including some advanced technologies such as batch normalization, dropout, and Leaky ReLU. We proposed an optimized convolutional neural network to identify 1320 sign language images, which was called as CNN-CB method. Totally ten runs were implemented with the hold-out randomly set for each run. (Results) The results indicate that our CNN-CB method gained an overall accuracy of 94.88 ± 0.99%. (Conclusion) Our CNN-CB method is superior to thirteen state-ofthe-art methods: eight traditional machine learning approaches and five modern convolutional neural network approaches

A Novel Gray Image Denoising Method Using Convolutional Neural Network

In order to make the image denoising more effective in high noise level environment, we propose a gray image denoising method using convolutional neural network (ConvNet). By constructing symmetric and dilated convolutional residual network and combining leaky ReLU (Rectified Linear Unit) and ReLU dual-functions. Experimental data show that this method achieves better objective results based on small resolution input & large resolution input and small dataset & large dataset compared with other methods. A series of subjective visual comparison results also show that proposed method can perform image denoising well in high noise level environment, and there are no problems such as damaged edge or texture details, boundary artifacts and poor definition of the denoised image. The proposed method not only improves the image denoising ability, but also improves the denoising productivity to some extent. In the related work, we have made a comprehensive discussion on the current mainstream image denoising methods based on traditional machine learning and deep learning, which to some extent makes up for the lack of such reviews. In addition, the existing problems and future development direction of image denoising are discussed in detail.

Convolutional Neural Network-Based Skin Lesion Classification With Variable Nonlinear Activation Functions

One of the worst forms of skin cancer is melanoma which can be curable if it is diagnosed at an early stage. The earlier the cancer is diagnosed, the better is the outcome. The risk of death from melanoma is directly related to the delay in identifying a lesion. A Deep Learning-based computer diagnosing system can be an automated solution in clinical assessments to overcome this problem. Convolutional Neural Network (CNN) can help to improve the classification rate of skin lesions from dermoscopic images without the need for any human assistance. The linear and nonlinear activation functions act as a node placed at hidden layers or output layers of a Neural Network to play a role in deciding whether that node should pass information to the following layer or not. This critical mathematical mechanism influences the accuracy rate of the CNN. To obtain acceptable performance, CNN requires a large amount of training data. This research shows how fast and effective different types of nonlinear activation functions work on a CNN with limited image datasets. Experimental analysis reveals that the proposed CNN model with parameterized Leaky ReLU function outperforms (97.50% accuracy, 98.00% precision, and 98.00% sensitivity) the same network with distinct nonlinear activation functions for the problem of melanoma recognition by classifying skin lesion into three classes. All experimental studies are carried out using images from PH2 (a dermoscopic image database obtained at the Hospital Pedro Hispano Dermatology Service in Matosinhos, Portugal) and International Skin Imaging Collaboration (ISIC) archive datasets.

An Improved Deep Network-Based Scene Classification Method for Self-Driving Cars

A self-driving car is a hot research topic in the field of the intelligent transportation system, which can greatly alleviate traffic jams and improve travel efficiency. Scene classification is one of the key technologies of self-driving cars, which can provide the basis for decision-making in self-driving cars. In recent years, deep learning-based solutions have achieved good results in the problem of scene classification. However, some problems should be further studied in the scene classification methods, such as how to deal with the similarities among different categories and the differences among the same category. To deal with these problems, an improved deep network-based scene classification method is proposed in this article. In the proposed method, an improved faster region with convolutional neural network features (RCNN) network is used to extract the features of representative objects in the scene to obtain local features, where a new residual attention block is added to the Faster RCNN network to highlight local semantics related to driving scenarios. In addition, an improved Inception module is used to extract global features, where a mixed Leaky ReLU and ELU function is presented, to reduce the possible redundancy of the convolution kernel and enhance the robustness. Then, the local features and the global features are fused to realize the scene classification. Finally, a private dataset is built from the public datasets for the specialized application of scene classification in the self-driving field, and the proposed method is tested on the proposed dataset. The experimental results show that the accuracy of the proposed method can reach 94.76%, which is higher than the state-of-the-art methods.

Flood Prediction using Deep Learning Models

Deep learning has recently appeared as one of the best reliable approaches for forecasting time series. Even though there are numerous data-driven models for flood prediction, most studies focus on prediction using a single flood variable. The creation of various data-driven models may require unfeasible computing resources when estimating multiple flood variables. Furthermore, the trends of several flood variables can only be revealed by analysing long-term historical observations, which conventional data-driven models do not adequately support. This study proposed a time series model with layer normalization and Leaky ReLU activation function in multivariable long-term short memory (LSTM), bidirectional long-term short memory (BI-LSTM) and deep recurrent neural network (DRNN). The proposed models were trained and evaluated by using the sensory historical data of river water level and rainfall in the east coast state of Malaysia. It were then, compared to the other six deep learning models. In terms of prediction accuracy, the experimental results also demonstrated that the deep recurrent neural network model with layer normalization and Leaky ReLU activation function performed better than other models.

An efficient real-time colonic polyp detection with YOLO algorithms trained by using negative samples and large datasets

Colorectal cancer (CRC) is one of the common types of cancer with a high mortality rate. Colonoscopy is the gold standard for CRC screening and significantly reduces CRC mortality. However, due to many factors, the rate of missed polyps, which are the precursors of colorectal cancer, is high in practice. Therefore, many artificial intelligence-based computer-aided diagnostic systems have been presented to increase the detection rate of missed polyps. In this article, we present deep learning-based methods for reliable computer-assisted polyp detection. The proposed methods differ from state-of-the-art methods as follows. First, we improved the performances of YOLOv3 and YOLOv4 object detection algorithms by integrating Cross Stage Partial Network (CSPNet) for real-time and high-performance automatic polyp detection. Then, we utilized advanced data augmentation techniques and transfer learning to improve the performance of polyp detection. Next, for further improving the performance of polyp detection using negative samples, we substituted the Sigmoid-weighted Linear Unit (SiLU) activation functions instead of the Leaky ReLU and Mish activation functions, and Complete Intersection over Union (CIoU) as the loss function. In addition, we present a comparative analysis of these activation functions for polyp detection. We applied the proposed methods on the recently published novel datasets, which are the SUN polyp database and the PICCOLO database. Additionally, we investigated the proposed models for MICCAI Sub-Challenge on Automatic Polyp Detection in Colonoscopy dataset. The proposed methods outperformed the other studies in both real-time performance and polyp detection accuracy.

Optimized leaky ReLU for handwritten Arabic character recognition using convolution neural networks

Optimized leaky ReLU for handwritten Arabic character recognition using convolution neural networks

Методология построения нейронной нечеткой сети в области информационной безопасности

This paper proposes the use of hybrid models based on neural networks and fuzzy systems to build intelligent intrusion detection systems based on the theory of fuzzy rules. The presented system will be able to generate rules based on the results using fuzzy logic neurons. To avoid oversaturation and assist in determining the necessary network topology, training models based on extreme learning machine and regularization theory will be used to find the most significant neurons. In this paper, a type of SQL injection cyberattack is considered, which actively exploits errors in systems that communicate with the database via SQL commands, and for this reason is considered a kind of straightforward attack. The fuzzy neural network architecture used in detecting SQL injection attacks is a multi-component structure. The first two layers of the model are considered as a fuzzy inference system capable of extracting knowledge from data and transforming it into fuzzy rules. These rules help build automated systems for detecting SQL injection attacks. The third layer consists of a simple neuron that has an activation function called a leaky ReLU. The first layer consists of fuzzy neurons, the activation functions of which are Gaussian membership functions of fuzzy sets, defined in accordance with the partitioning of the input variables. The technique uses the concept of a simple linear regression model to solve the problem of choosing the best subsets of neurons. To perform model selection, the paper used the widely used least angular regression (LARS) algorithm.

Quantum ReLU activation for Convolutional Neural Networks to improve diagnosis of Parkinson’s disease and COVID-19

This study introduces a quantum-inspired computational paradigm to address the unresolved problem of Convolutional Neural Networks (CNNs) using the Rectified Linear Unit (ReLU) activation function (AF), i.e., the ‘dying ReLU’. This problem impacts the accuracy and the reliability in image classification tasks for critical applications, such as in healthcare. The proposed approach builds on the classical ReLU and Leaky ReLU, applying the quantum principles of entanglement and superposition at a computational level to derive two novel AFs, respectively the ‘Quantum ReLU’ (QReLU) and the ‘modified-QReLU’ (m-QReLU). The proposed AFs were validated when coupled with a CNN using seven image datasets on classification tasks involving the detection of COVID-19 and Parkinson’s Disease (PD). The out-of-sample/test classification accuracy and reliability (precision, recall and F1-score) of the CNN were compared against those of the same classifier when using nine classical AFs, including ReLU-based variations. Findings indicate higher accuracy and reliability for the CNN when using either QReLU or m-QReLU on five of the seven datasets evaluated. Whilst retaining the best classification accuracy and reliability for handwritten digits recognition on the MNIST dataset (ACC = 99%, F1-score = 99%), avoiding the ‘dying ReLU’ problem via the proposed quantum AFs improved recognition of PD-related patterns from spiral drawings with the QReLU especially, which achieved the highest classification accuracy and reliability (ACC = 92%, F1-score = 93%). Therefore, with these increased accuracy and reliability, QReLU and m-QReLU can aid critical image classification tasks, such as diagnoses of COVID-19 and PD.

Deep Neural Network for Electroencephalogram based Emotion Recognition

Emotion recognition using electroencephalogram (EEG) signals is an aspect of affective computing. The EEG refers to recording brain responses via electrical signals by showing external stimuli to the participants. This paper proposes the prediction of valence, arousal, dominance and liking for EEG signals using a deep neural network (DNN). The EEG data is obtained from the AMIGOS dataset, a publicly available dataset for mood and personality research. Two features, normalized and power and normalized wavelet energy, are extracted using Fourier and wavelet transform, respectively. A DNN with three different activation functions (exponential linear unit, rectified linear unit [ReLU] and leaky ReLU) has been applied for single and combined features. The result of combined features with leaky ReLU is found to be the best, with a classification accuracy of 85.47, 81.87, 84.04 and 86.63 for valence, arousal, dominance and liking, respectively.

Multi-Path Deep CNN with Residual Inception Network for Single Image Super-Resolution

Recent research on single-image super-resolution (SISR) using deep convolutional neural networks has made a breakthrough and achieved tremendous performance. Despite their significant progress, numerous convolutional neural networks (CNN) are limited in practical applications, owing to the requirement of the heavy computational cost of the model. This paper proposes a multi-path network for SISR, known as multi-path deep CNN with residual inception network for single image super-resolution. In detail, a residual/ResNet block with an Inception block supports the main framework of the entire network architecture. In addition, remove the batch normalization layer from the residual network (ResNet) block and max-pooling layer from the Inception block to further reduce the number of parameters to preventing the over-fitting problem during the training. Moreover, a conventional rectified linear unit (ReLU) is replaced with Leaky ReLU activation function to speed up the training process. Specifically, we propose a novel two upscale module, which adopts three paths to upscale the features by jointly using deconvolution and upsampling layers, instead of using single deconvolution layer or upsampling layer alone. The extensive experimental results on image super-resolution (SR) using five publicly available test datasets, which show that the proposed model not only attains the higher score of peak signal-to-noise ratio/structural similarity index matrix (PSNR/SSIM) but also enables faster and more efficient calculations against the existing image SR methods. For instance, we improved our method in terms of overall PSNR on the SET5 dataset with challenging upscale factor 8× as 1.88 dB over the baseline bicubic method and reduced computational cost in terms of number of parameters 62% by deeply-recursive convolutional neural network (DRCN) method.

Activation functions of artificial-neural-network-based nonlinear equalizers for optical nonlinearity compensation

We investigated the performance of artificial neural network (ANN)-based nonlinear equalizers for optical nonlinearity compensation by comparing activation functions, including a sigmoid function, ReLU, and Leaky ReLU. We compared the learning speeds and compensation performances by evaluating the resulting error vector magnitudes of the compensated signals. The performance was investigated using simulated 100-km optical fiber transmission of 10-GSymbol/s 16QAM signals. When the number of hidden-layer units in the ANN was small, the sigmoid function showed better performance in learning speed than ReLU and Leaky ReLU. This point is important because the number of ANN units has to be reduced in order to improve the computational complexity of the ANN-based nonlinear equalizer.

A deep learning approach for Indian sign language gestures classification with different backgrounds

The sign language recognition system recently has drawn the attention of various researchers as there is no universal sign language, moreover, it consists of many patterns and postures. Many methods for extracting features and classifying sign language have been proposed in the literature, most of them are based on machine learning techniques. In this article, a deep learning method has been adopted by designing a Convolution Neural Network (CNN) model to extract the sign language features where for classification softmax layer is used. All alphabets in simple as well as complex backgrounds have been considered, where data is collected from 100 subjects in different lighting conditions. The effect of various optimization techniques (Adam, Sgdm, RMSProp), activation functions (ReLU and Leaky ReLU) for generalization ability is also observed. The proposed approach has succeeded in attaining the testing accuracy of 99.10%, 92.69%, and 95.95% on the Indian sign language dataset with simple, complex backgrounds and on a mixed background scenarios, respectively. The model is also tested on NUS dataset-I, NUS dataset-II, their combination, and achieved the accuracy of 100%, 95.95%, and 97.22% respectively.

HSV model‐based segmentation driven facial acne detection using deep learning

AbstractAcne is a skin disease mainly caused by bacteria, the hair follicles exposed to oil, and dying skin cells. These sometimes trigger whiteheads, blackheads or pimples, usually on the neck, arms, arm and back of shoulders. Acne in adolescents is the most severe, even though it affects people of any generation. Doctors can easily detect acne by seeing a patient's skin, but automatic acne detection is not easy for machines. Deep learning (DL) approaches have been quite successful for various aspects like classification and detection of objects in real life. This paper proposes an enhanced DL CNN model with the Leaky ReLU activation function. DermNet NZ's facial acne images dataset is used for the experiments. Three different techniques‐ K‐Means, Texture Analysis and HSV Model‐Based Segmentation, are applied for image segmentation to extract the acne region from skin images. After applying all the above image segmentation methods five times for each method, output images from K‐Means and HSV (5 + 5 images) are collected and combined with the dataset. Using that dataset, one SVM model using Scikit‐learn and two CNN models‐ one with the ReLU activation function and another with the LeakyReLU activation function, is trained. Out of these three models, the proposed CNN (LeakyReLU) model achieved a 97.54% accuracy.

Segmentation and quantification of COVID-19 infections in CT using pulmonary vessels extraction and deep learning.

At the end of 2019, the World Health Organization (WHO) reported pneumonia that started in Wuhan, China, as a global emergency problem. Researchers quickly advanced in research to try to understand this COVID-19 and sough solutions for the front-line professionals fighting this fatal disease. One of the tools to aid in the detection, diagnosis, treatment, and prevention of this disease is computed tomography (CT). CT images provide valuable information on how this new disease affects the lungs of patients. However, the analysis of these images is not trivial, especially when researchers are searching for quick solutions. Detecting and evaluating this disease can be tiring, time-consuming, and susceptible to errors. Thus, in this study, we aim to automatically segment infections caused by COVID19 and provide quantitative measures of these infections to specialists, thus serving as a support tool. We use a database of real clinical cases from Pedro Ernesto University Hospital of the State of Rio de Janeiro, Brazil. The method involves five steps: lung segmentation, segmentation and extraction of pulmonary vessels, infection segmentation, infection classification, and infection quantification. For the lung segmentation and infection segmentation tasks, we propose modifications to the traditional U-Net, including batch normalization, leaky ReLU, dropout, and residual block techniques, and name it as Residual U-Net. The proposed method yields an average Dice value of 77.1% and an average specificity of 99.76%. For quantification of infectious findings, the proposed method achieves results like that of specialists, and no measure presented a value of ρ < 0.05 in the paired t-test. The results demonstrate the potential of the proposed method as a tool to help medical professionals combat COVID-19. fight the COVID-19.

An improved DualGAN for near-infrared image colorization

This paper focuses on the colorization problem of near-infrared (NIR) images. Traditional colorization methods of grayscale images usually depend on users’ intervention and cannot be extended to NIR image colorization due to inherent complexity, such as the same near-infrared lights emitted by objects with different colors. Furthermore, a large number of paired and labeled images, which cannot be guaranteed for the addressed problem, need to be provided during the training phase, whether for some traditional reference-based coloring methods or for CNN-based automatic coloring ones. Benefiting from the advantages of deep learning and generative adversarial networks (GANs) in the image-to-image translation, an improved DualGAN architecture is constructed to deal with the investigated problem. The developed architecture contains four blocks and any two adjacent blocks exist a direct connection channel, where convolution layers in each block enclose batch normalization and leaky ReLU nonlinearities. The adoption of dual deep learning networks is to establish the conversion translation relationship between NIR images and RGB images without paired and labeled requirements. Besides, a mixed loss function by integrating generator loss for discriminators’ training is designed to decrease the occurrence of incorrect images generated by generators. Finally, an intensive comparison analysis based on common data sets is conducted to verify superiority over leading-edge methods in qualitative and quantitative visual assessments.