Leaky ReLU Research Articles

A practical Approach to Handwritten Digit Generation with Generative Adversarial Networks on MNIST

Implementation of a Generative Adversarial Network (GAN) utilizing the MNIST (Modified National Institute of Standards and Technology database) dataset of handwritten digits, the GAN comprises a generator creating synthetic images from random noise and a discriminator to classifying these images as real or fake. The Generator utilizes leaky ReLU (rectified linear unit) activations, Batch normalization and reshaping to produce 28x28 grayscale images, while the Discriminator uses dense layers, leaky ReLU, and dropout to enhance classification accuracy. Both networks are trained using the Adam optimizer to improve stability and performance. The GAN is trained in an adversarial setup where the Generator seeks to create convincing images and the Discriminator aims to correctly classify them. Results demonstrate the GAN's ability to generate high-quality handwritten digit images, displaying its effectiveness and providing valuable insights into best practices for GAN implementation and training.

Efficient quantum circuits for machine learning activation functions including constant T-depth ReLU

In recent years, Quantum Machine Learning (QML) has increasingly captured the interest of researchers. Among the components in this domain, activation functions hold a fundamental and indispensable role. Our research focuses on the development of activation functions quantum circuits for integration into fault-tolerant quantum computing architectures, with an emphasis on minimizing T-depth. Specifically, we present novel implementations of ReLU and leaky ReLU activation functions, achieving constant T-depths of 4 and 8, respectively. Leveraging quantum lookup tables, we extend our exploration to other activation functions such as the sigmoid. This approach enables us to customize precision and T-depth by adjusting the number of qubits, making our results more adaptable to various application scenarios. This study represents a significant advancement towards enhancing the practicality and application of quantum machine learning. Published by the American Physical Society 2024

IMPLEMENTATION OF BACKPROPAGATION AND HYBRID ARIMA-NN METHODS IN PREDICTING ACCURACY LEVELS OF RAINFALL IN MAKASSAR CITY

Hybrid ARIMA-NN is a combined approach of the ARIMA model used to capture linear patterns in time series data and Artificial Neural Networks (ANN) to handle non-linear and stochastic patterns. Using a gradient descent algorithm, backpropagation adjusts synaptic weights based on the error between the network's prediction and actual training data values. In this study, a comparison was made between the Backpropagation method and Hybrid ARIMA-NN in forecasting rainfall in Makassar City. Rainfall data in Makassar City uses data from the rainfall measuring station at the Paotere Maritime Meteorological Station in Makassar. The activation functions used are ReLU and Leaky ReLU with epoch parameters set at 350, and learning rates of 0.01, 0.001, 0.0001, and 0.00001. The two best methods selected for further evaluation are Backpropagation with architecture 12-32-16-8-1 and Hybrid ARIMA-NN (ARIMA [4,0,1]-NN 12-256-128-64-1). The ARIMA model (4,0,1) with AIC values of 1303.4 and RMSE 162,369 is the best compared to other models, which aligns with the advantages of backpropagation architecture. The results showed that the Backpropagation method excelled with an RMSE value of 137.320 or 0.1149, indicating high accuracy in forecasting changes in seasonal trends and patterns. Hybrid ARIMA-NN gives good results with RMSE 145.834, as residues contain better nonlinearity compared to ARIMA models (4,0,1), although it shows a slightly higher error rate compared to Backpropagation.

AONet: Attention network with optional activation for unsupervised video anomaly detection

AbstractAnomaly detection in video surveillance is crucial but challenging due to the rarity of irregular events and ambiguity of defining anomalies. We propose a method called AONet that utilizes a spatiotemporal module to extract spatiotemporal features efficiently, as well as a residual autoencoder equipped with an attention network for effective future frame prediction in video anomaly detection. AONet utilizes a novel activation function called OptAF that combines the strengths of the ReLU, leaky ReLU, and sigmoid functions. Furthermore, the proposed method employs a combination of robust loss functions to address various aspects of prediction errors and enhance training effectiveness. The performance of the proposed method is evaluated on three widely used benchmark datasets. The results indicate that the proposed method outperforms existing state‐of‐the‐art methods and demonstrates comparable performance, achieving area under the curve values of 97.0%, 86.9%, and 73.8% on the UCSD Ped2, CUHK Avenue, and ShanghaiTech Campus datasets, respectively. Additionally, the high speed of the proposed method enables its application to real‐time tasks.

An Innovative Deep Neural Network Model for Precise Calorie Burn Prediction from Physical Activity Data

Accurate prediction of calories burned during physical activities is crucial for various applications in health monitoring, fitness tracking, and personalized nutrition. Traditional methods often lack the precision needed for individualized estimates, which has increased interest in advanced machine learning approaches. This research introduces a deep learning model designed to predict calories burned with enhanced accuracy by capturing complex, non-linear relationships in the data. The model employs a multilayer perceptron neural network, Leaky ReLU activations, dropout regularization, and the Adam optimizer to improve generalizability and prevent overfitting. The evaluation of training and validation loss over epochs demonstrated the model's robustness and capacity to generalize effectively to novel data. The model's performance was evaluated using various metrics, achieving superior results with a remarkable Mean Absolute Error (MAE) of 0.27% and an accuracy of 99.73%, outperforming other models discussed in the literature. These findings indicate that deep learning offers significant potential for improving calorie prediction models, providing more reliable fitness and health management tools.

Gradient Descent Provably Escapes Saddle Points in the Training of Shallow ReLU Networks

AbstractDynamical systems theory has recently been applied in optimization to prove that gradient descent algorithms bypass so-called strict saddle points of the loss function. However, in many modern machine learning applications, the required regularity conditions are not satisfied. In this paper, we prove a variant of the relevant dynamical systems result, a center-stable manifold theorem, in which we relax some of the regularity requirements. We explore its relevance for various machine learning tasks, with a particular focus on shallow rectified linear unit (ReLU) and leaky ReLU networks with scalar input. Building on a detailed examination of critical points of the square integral loss function for shallow ReLU and leaky ReLU networks relative to an affine target function, we show that gradient descent circumvents most saddle points. Furthermore, we prove convergence to global minima under favourable initialization conditions, quantified by an explicit threshold on the limiting loss.

Phase unwrapping in digital holography based on SRDU-net

In digital holographic measurement, the hologram phase is extracted using an inverse tangent function, resulting in a wrapped phase that is constrained to be (-π,π]. However, the phase contains the height information of the object, which is crucial for accurate measurement of the object contour. Therefore, a digital holographic phase unwrapping method based on SRDU-Net (Separable-Residual-Dense-Inverted U-Net) is proposed in this paper. The SRDU-Net network is constructed by introducing depth-separable convolution, inverted residuals and dense blocks with U-Net as the network framework to achieve high-precision hologram phase recovery. This network adopts depth-separable convolution instead of traditional convolution, combines the inverse residual connection to construct a lightweight convolution structure, and defines dense blocks with this structure to form a lightweight grouped deep convolution network. Meanwhile, the Leaky ReLU activation function is used to introduce the learning rate mechanism to optimize the network parameters with Huber and MSE (mean square error) as the combined loss function. The simulated phase dataset is used to train the network, the trained network model is tested for speckle noise immunity, and phase unwrapping experiments are performed on the collected holograms of the test samples. The results show that SRDU-Net improves the SSIM by 0.1% and reduces the RMSE by 86% over Res-UNet (Residual-UNet). Therefore, the proposed method can realize high-precision recovery of digital holographic wrapped phases, and has a good robustness to phase unwrapping of holograms containing a high degree of speckle noise.

Evaluating Activation Functions in GAN Models for Virtual Inpainting: A Path to Architectural Heritage Restoration

Computer vision has advanced much in recent years. Several tasks, such as image recognition, classification, or image restoration, are regularly solved with applications using artificial intelligence techniques. Image restoration comprises different use cases such as style transferring, improvement of quality resolution, or completing missing parts. The latter is also known as image inpainting, virtual image inpainting in this case, which consists of reconstructing missing regions or elements. This paper explores how to evaluate the performance of a deep learning method to do virtual image inpainting to reconstruct missing architectonical elements in images of ruined Greek temples to measure the performance of different activation functions. Unlike a previous study related to this work, a direct reconstruction process without segmented images was used. Then, two evaluation methods are presented: the objective one (mathematical metrics) and an expert (visual perception) evaluation to measure the performance of the different approaches. Results conclude that ReLU outperforms other activation functions, while Mish and Leaky ReLU perform poorly, and Swish’s professional evaluations highlight a gap between mathematical metrics and human visual perception.

Interpolation-split: a data-centric deep learning approach with big interpolated data to boost airway segmentation performance

The morphology and distribution of airway tree abnormalities enable diagnosis and disease characterisation across a variety of chronic respiratory conditions. In this regard, airway segmentation plays a critical role in the production of the outline of the entire airway tree to enable estimation of disease extent and severity. Furthermore, the segmentation of a complete airway tree is challenging as the intensity, scale/size and shape of airway segments and their walls change across generations. The existing classical techniques either provide an undersegmented or oversegmented airway tree, and manual intervention is required for optimal airway tree segmentation. The recent development of deep learning methods provides a fully automatic way of segmenting airway trees; however, these methods usually require high GPU memory usage and are difficult to implement in low computational resource environments. Therefore, in this study, we propose a data-centric deep learning technique with big interpolated data, Interpolation-Split, to boost the segmentation performance of the airway tree. The proposed technique utilises interpolation and image split to improve data usefulness and quality. Then, an ensemble learning strategy is implemented to aggregate the segmented airway segments at different scales. In terms of average segmentation performance (dice similarity coefficient, DSC), our method (A) achieves 90.55%, 89.52%, and 85.80%; (B) outperforms the baseline models by 2.89%, 3.86%, and 3.87% on average; and (C) produces maximum segmentation performance gain by 14.11%, 9.28%, and 12.70% for individual cases when (1) nnU-Net with instant normalisation and leaky ReLU; (2) nnU-Net with batch normalisation and ReLU; and (3) modified dilated U-Net are used respectively. Our proposed method outperformed the state-of-the-art airway segmentation approaches. Furthermore, our proposed technique has low RAM and GPU memory usage, and it is GPU memory-efficient and highly flexible, enabling it to be deployed on any 2D deep learning model.

Predictive Modeling of Conveyor Belt Deterioration in Coal Mines Using AI Techniques

Conveyor belts are vital for material transportation in coal mines due to their cost-effectiveness and versatility. These belts endure significant wear from harsh operating conditions, risking substantial financial losses if they fail. This study develops five artificial neural network (ANN) models to predict conveyor belt damage using 11 parameters from the Belchatow brown coal mine in Poland. The models target five outputs: number of repairs and cable cuts, cumulative number of repairs and cable cuts, and their ages. Various optimizers (Adam, Nadam, RMSprop, Adamax, and stochastic gradient descent or SGD) and activation functions (ReLU, Swish, sigmoid, tanh, Leaky ReLU, and softmax) were tested to find the optimal configurations. The predictive performance was evaluated using three error indicators against actual mine data. Superior models can forecast belt behavior under specific conditions, aiding proactive maintenance. The study also advocates for the Diagbelt+ system over human inspections for failure detection. This modeling approach enhances proactive maintenance, preventing total system breakdowns due to belt wear.

Graph attention network with convolutional layer for predicting gene regulations from single-cell ribonucleic acid sequence data

Reconstructing gene regulatory networks (GRNs) is an important task to reveal the regulatory relationship between genes and understand the mechanism of intracellular gene expression regulation. With the development of single-cell ribonucleic acid sequencing (scRNA-seq) technology, researchers begin to attempt to infer GRN within cells. In this paper, we propose graph attention network with convolutional layer (GATCL) to infer the latent interactions between transcription factors (TFs) and target genes in GRN. Firstly, GATCL uses graph attention network (GAT), which can effectively extract information about genes and TFs. Secondly, we combine multi-head attention layer with one-head attention layer and propose a new method of using convolution instead of weight matrix. Thirdly, we use the exponential linear unit (ELU) activation function to replace the leaky rectified linear unit (LReLU) commonly used in GAT, which further improves the accuracy of GATCL. The AUROC of our method on seven scRNA-seq datasets with four types of ground-truth networks reached an average of 0.827, which is higher than other state-of-the-art models. GATCL applies a supervised deep learning algorithm to solve the problems existing in the inference of GRN from scRNA-seq in the engineering field, and the effectiveness of this method is verified by a large number of experiments.

AIR POLLUTION PREDICTION USING MODIFIED GRU DEEP LEARNING MODEL

<p>In recent days air quality has been very poor in urban society due to a lot of industries, vehicles and so on. To overcome poor quality, an air quality prediction is much needed to manage an urban area and control pollution. In this article, deep learning (DL) namely a Modified Gated Recurrent Unit (GRU) neural network is used to forecast air pollution. Initially, a new activation function named Dual-Slope Leaky ReLU (DSL-ReLU) is introduced. The DSL-ReLU activation function includes two hyperparameters which control the slopes for positive and negative input values to achieve fine-tuned responses for varying data inputs. Then, the model parameters like learning rate, batch size, and dropout rates are tuned using the Spider Wasp Optimization (SWO) algorithm. The SWO algorithm is based on the female spider wasp’s hunting and nesting behaviour that is applied to solve the optimization issues. This SWO model is used to GRU model parameters and provides robustness in the predictive modelling of air quality monitoring. For validation, the Kaggle data set is used whereas the proposed optimized GRU model considerably improves the accuracy of air pollution forecasts</p>

MultiTumor Analyzer (MTA-20–55): A network for efficient classification of detected brain tumors from MRI images

Brain cancer, one of the leading causes of mortality worldwide, is caused by brain tumors. Early diagnosis of tumors and predicting their progression can help doctors to save lives. In this article, we have designed an automated approach for locating and classifying tumors from MRI images. The novelties of the research work include the following two stages: Developing an encoder-decoder type 20-Layered deep neural network (DNN) named MultiTumor Analyzer (MTA-20) with 15 down-sampling layers and 4 up-sampling layers, the segmentation is performed in the initial stage. Here, we have adhered a Leaky ReLU activation function instead of ReLU which learn a parameter with negative values that may have valuable information which is essential specifically for image segmentation. Further, a 55-layered DNN using multistage feature fusion is developed in the second stage of the work for the classification of localized tumors. The classification is performed using developed MultiTumor Analyzer (MTA-55) DNN with Softmax classifier. The efficacy of the designed network is validated using highly cited quantitative measures such as accuracy, sensitivity, specificity, dice similarity coefficient (DSC), precision, and F1-measure. It is observed that the proposed MTA-20 DNN attains the average accuracy, sensitivity, specificity, DSC, and precision of 99.2 %, 94.6 %, 99.3 %, 88 %, and 82.5 % respectively against seven state-of-the-art techniques. Also, it is found that, the proposed MTA-55 DNN provides the overall accuracy, recall, specificity, F1-measure, precision, and DSC of 99.8 %, 99.633 %, 99.844 %, 99.659 %, 99.689 %, and 99.656 % respectively as compared to thirteen state-of-the-art techniques. These results corroborate the superiority of the proposed technique.

Improved dendritic learning: Activation function analysis

This study conducted a thorough evaluation of an improved dendritic learning (DL) framework, focusing specifically on its application in power load forecasting. The objective was to optimise the activation functions within the synapses and somas of DL to enhance their adaptability across diverse real-world scenarios. Through a rigorous analysis involving 25 experiments across five activation functions (sigmoid, hyperbolic tangent (tanh), rectified linear unit (ReLU), leaky ReLU, and exponential linear unit (ELU)), we elucidated their impacts on both regression and classification performance. Notably, the leaky ReLU–tanh combination demonstrated exceptional mean performance and effectiveness across 14 benchmark datasets from the University of California Irvine Machine Learning Repository, surpassing alternative combinations. When applied to power load forecasting, this combination outperformed other models, particularly transformer and LSTM. These findings underscore the significant advantages of the leaky ReLU–tanh-based DL framework in accurately predicting electricity load in smart grids, as evidenced by the lowest mean absolute error (39.27), root mean squared error (29.13), and mean absolute percentage error (2.84).

A transfer learning enabled approach for ocular disease detection and classification.

Ocular diseases pose significant challenges in timely diagnosis and effective treatment. Deep learning has emerged as a promising technique in medical image analysis, offering potential solutions for accurately detecting and classifying ocular diseases. In this research, we propose Ocular Net, a novel deep learning model for detecting and classifying ocular diseases, including Cataracts, Diabetic, Uveitis, and Glaucoma, using a large dataset of ocular images. The study utilized an image dataset comprising 6200 images of both eyes of patients. Specifically, 70% of these images (4000 images) were allocated for model training, while the remaining 30% (2200 images) were designated for testing purposes. The dataset contains images of five categories that include four diseases, and one normal category. The proposed model uses transfer learning, average pooling layers, Clipped Relu, Leaky Relu and various other layers to accurately detect the ocular diseases from images. Our approach involves training a novel Ocular Net model on diverse ocular images and evaluating its accuracy and performance metrics for disease detection. We also employ data augmentation techniques to improve model performance and mitigate overfitting. The proposed model is tested on different training and testing ratios with varied parameters. Additionally, we compare the performance of the Ocular Net with previous methods based on various evaluation parameters, assessing its potential for enhancing the accuracy and efficiency of ocular disease diagnosis. The results demonstrate that Ocular Net achieves 98.89% accuracy and 0.12% loss value in detecting and classifying ocular diseases by outperforming existing methods.

An Improved YOLOv7-Tiny Method for the Segmentation of Images of Vegetable Fields

In response to the limitations of existing methods in differentiating between vegetables and all types of weeds in farmlands, a new image segmentation method is proposed based on the improved YOLOv7-tiny. Building on the original YOLOv7-tiny framework, we replace the CIoU loss function with the WIoU loss function, substitute the Leaky ReLU loss function with the SiLU activation function, introduce the SimAM attention mechanism in the neck network, and integrate the PConv convolution module into the backbone network. The improved YOLOv7-tiny is used for vegetable target detection, while the ExG index, in combination with the OTSU method, is utilized to obtain a foreground image that includes both vegetables and weeds. By integrating the vegetable detection results with the foreground image, a vegetable distribution map is generated. Subsequently, by excluding the vegetable targets from the foreground image using the vegetable distribution map, a single weed target is obtained, thereby achieving accurate segmentation between vegetables and weeds. The experimental results show that the improved YOLOv7-tiny achieves an average precision of 96.5% for vegetable detection, with a frame rate of 89.3 fps, Params of 8.2 M, and FLOPs of 10.9 G, surpassing the original YOLOv7-tiny in both detection accuracy and speed. The image segmentation algorithm achieves a mIoU of 84.8% and an mPA of 97.8%. This method can effectively segment vegetables and a variety of weeds, reduce the complexity of segmentation with good feasibility, and provide a reference for the development of intelligent plant protection robots.

Performance Evaluation of VGG_19 Model for Brain Tumor Datasets

This paper presents a comprehensive assessment of the VGG19 model for detecting brain tumors using deep learning methods, particularly convolutional neural networks (CNNs). The study evaluates key performance metrics such as accuracy, precision, recall, F1 score, and area under the curve (AUC). The investigation delves into the influence of different activation functions, learning rates, and epochs on the VGG19 model's performance. Activation functions are crucial for feature extraction and model convergence, with a comparison made between ReLU and Leaky ReLU to determine their effectiveness in enhancing brain tumor detection accuracy. Additionally, the study examines the impact of various learning rates (0.01, 0.001, 0.0001) on the model's convergence speed and overall performance. Different numbers of training epochs (15, 25, 50) are also considered to strike a balance between computational efficiency and model effectiveness. Evaluation metrics encompass accuracy, precision, recall, F1 score, and AUC, providing a comprehensive assessment of the model's performance across different configurations. The goal is to identify optimal combinations of activation functions, learning rates, and epochs to maximize the VGG19 model's accuracy in detecting brain tumors. This research contributes to advancing efficient and accurate methodologies for brain tumor detection, facilitating early diagnosis and treatment planning. The insights gained from this study can guide the development of more effective deep learning-based systems for medical image analysis, benefiting both patients and healthcare professionals.

Seatbelt Detection Algorithm Improved with Lightweight Approach and Attention Mechanism

Precise and rapid detection of seatbelts is an essential research field for intelligent traffic management. In order to improve the detection precision of seatbelts and speed up algorithm inference velocity, a lightweight seatbelt detection algorithm is proposed. Firstly, by adding the G-ELAN module designed in this paper to the YOLOv7-tiny network, the optimization of construction and reduction of parameters are accomplished, and the ResNet is compressed with the channel pruning approach to decrease computational overheads. Then, the Mish activation function is utilized to replace the Leaky Relu in the neck to enhance the non-linear competence of the network. Finally, the triplet attention module is integrated into the model after pruning to make up for the underlying performance reduction caused by the previous stage and upgrade overall detection precision. The experimental results based on the self-built seatbelt dataset showed that, compared to the initial network, the Mean Average Precision (mAP) achieved by the proposed GM-YOLOv7 was improved by 3.8%, while the volume and the computation amount were lowered by 20% and 24.6%, respectively. Compared with YOLOv3, YOLOX, and YOLOv5, the mAP of GM-YOLOv7 increased by 22.4%, 4.6%, and 4.2%, respectively, and the number of computational operations decreased by 25%, 63%, and 38%, respectively. In addition, the accuracy of the improved RST-Net increased to 98.25%, while the parameter value was reduced by 48% compared to the basic model, effectively improving the detection performance and realizing a lightweight structure.

Comparison of Activation Functions in Convolutional Neural Network for Poisson Noisy Image Classification

Deep learning, specifically the Convolutional Neural Network (CNN), has been a significant technology tool for image processing and human health. CNNs, which mimic the working principles of the human brain, can learn robust representations of images. However, CNNs are susceptible to noise interference, which can impact classification performance. Choosing the right activation function can improve CNNs performance and accuracy. This research aims to test the accuracy of CNN with ResNet50, VGG16, and GoogleNet architectures combined with several activation functions such as ReLU, Leaky ReLU, Sigmoid, and Tanh in the classification of images that experience Poisson noise. Poisson noise is applied to each test data to evaluate CNN accuracy. The data used in this study consists of three scenarios of different numbers of classes, namely 3 classes, 5 classes, and 10 classes. The results showed that combining ResNet50 with the ReLU activation function produced the best performance in class recognition in each scenario of the number of classes experiencing Poisson noise interference. The model achieved 97% accuracy for 3-class data, 95% for 5-class data, and 90% for 10-class data. These results show that using ResNet50 with the ReLU activation function can provide excellent resistance to Poisson noise in image processing. It was found that as the number of classes increases, the accuracy of image recognition tends to decrease. This shows that the more complex the image classification task is with a larger number of classes, the more difficult it is for CNNs to distinguish between different classes. Doi: 10.28991/ESJ-2024-08-02-014 Full Text: PDF

Automatic Liver Segmentation from Multiphase CT Using Modified SegNet and ASPP Module

Liver cancer is one of the dominant causes of cancer death worldwide. Computed Tomography (CT) is the commonly used imaging modality for diagnosing it. Computer-based liver cancer diagnosis systems can assist radiologists in image interpretation and improve diagnosis speed and accuracy. Since liver segmentation is crucial to such systems, researchers are relentlessly pursuing various segmentation approaches. A clinically viable computer-aided system requires examining multiphase CT images. However, most of the research focuses only on the portal venous phase. In this work, we developed an automatic and efficient Deep Learning (DL) method using SegNet, atrous spatial pyramid pooling module and leaky ReLU layers for liver segmentation from quadriphasic abdominal CT volumes. The proposed method was validated on two datasets, an internal institutional dataset consisting of multiphase CT and a public dataset of portal venous phase CT volumes. The Dice Coefficients (DC) obtained were greater than 96% for the latter dataset and the portal venous phase of the former. For arterial, delayed and plain CT phases of the former dataset, the DC achieved were 94.61%, 95.01% and 93.23%, respectively. Experiments showed that our model performed better than the other state-of-the-art DL models. Ablation studies have revealed that the proposed model leverages the strengths of all the three components that make it up. The promising performance of the proposed method suggests that it is appropriate for incorporation in hepatic cancer diagnosis systems.