ReLU Activation Function Research Articles

Estimation of the Regression Model Using M-Estimation Method and Artificial Neural Networks in the Presence of Outliers

This study aimed to predict using regression models in the presence of outliers in the study data. The research delved into outliers, their detection, and model estimation through robust methods, represented by the M estimator and multilayer artificial neural networks. A comparison between these methods was conducted, and they were applied to real data representing a survey of private-sector power generators for 2021. Model evaluation was performed using Mean Squared Error (MSE) and the determination coefficient. The results indicated that the M-estimator with the Huber function outperformed its counterpart with the Tukey function. The best-performing architecture for the artificial neural network was ML-FF (5, 16, 32, 46, 128, 1) with the relu activation function. This network effectively handled extreme values and exhibited strong predictive capabilities. The choice of activation function and the number of hidden layers significantly impacted the neural network's performance, with the results showing the superiority of this artificial neural network over the robust estimators. Paper type Research paper

AUTOMATED PNEUMONIA IDENTIFICATION THROUGH CNN-BASED ANALYSIS

Pneumonia, a condition caused by infectious agents such as bacteria, viruses, fungi, or parasites, results in lung infections and the accumulation of pus in affected tissues. Incorrect diagnosis or improper treatment can significantly impact a patient's quality of life. This project focuses on the precise identification and categorization of pneumonia-afflicted patients through the analysis of their chest X-rays. Leveraging advancements in deep learning, healthcare experts can make more accurate diagnostic decisions for various illnesses. This study presents an innovative approach utilizing Convolutional Neural Networks (CNN) to predict and distinguish between patients affected by pneumonia and those who are not, based on chest X-ray images. The methodology involves a series of convolutional and max pooling layers activated using the ReLU activation function. Subsequently, the processed data is passed through dense layers, culminating in the activation of the output neuron through the sigmoidal function. During the training process, the model exhibits increasing accuracy and decreasing loss, demonstrating its effectiveness. To prevent overfitting, data augmentation techniques are applied before model fitting. These strategies collectively enhance the robustness and reliability of the deep learning models, resulting in accurate and persuasive detection of pneumonia from chest X-ray images. Keywords: Pneumonia, CNN, VGG-16, VGG-19, Chest X-rays, Classification, Deeep learning, Data augmenatation, ReLU, softmax.

Measurement and diagnostic system for detecting and classifying faults in the rotary hay tedder using multilayer perceptron neural networks

The implementation of preventive maintenance measures, which aim to assure the optimal performance of rotary machines and minimize the occurrence of substantial damage, has gained significant importance in the agricultural machinery industry. As part of this work, several measurements and computations were conducted to develop effective methods for detecting and classifying machine damages based on raw measurement data from accelerometers acquired both in the laboratory and real-life conditions. For this purpose a rotary hay tedder was equipped with a self-developed measuring and diagnostic system. The hardware and network communication components of this system were described. Machine learning methods were applied and the multi-layer perceptron (MLP) neural-network based models were created. The diagnostic models were made using different machine learning algorithms with 90 different configurations of neural-networks in total, including 30 different hidden layer arrangements and three activation functions (relu, softmax and sigmoid). Several learning scenarios were considered in order to determine and to distinguish between a healthy state and a machine failure, such as a missing or broken tedder tine. Based on laboratory experiments and meadow tests, 20 best metrics were selected for the input feature vector. When conducting laboratory and on-site tests, the best efficiency was achieved for the multi-layer perceptron neural-network architecture with two hidden layers (size of 20 and 20) and the relu activation function, and the Adam optimizer. Accuracy, as the ratio of correct predictions to total predictions made, for different diagnostic models was estimated. The confusion matrices were given to present the results for 10 rotors. Data efficiency for different diagnostic models and hyperparameters was also presented. For tests on the meadow, the average accuracy was 79% for detecting a missing tine, and 69% for detecting a tine failure (a missing or broken tine). In the case of the laboratory-based experimental results, the average accuracy for detecting a broken tine in the rotor was 92.5%. Therefore, based on the results, the proposed approach can be used for rotary hay tedder fault diagnosis.

Predicting Diffusion Coefficients in Nafion Membranes during the Soaking Process Using a Machine Learning Approach.

Nafion, a versatile polymer used in electrochemistry and membrane technologies, exhibits complex behaviors in saline environments. This study explores Nafion membrane's IR spectra during soaking and subsequent drying processes in salt solutions at various concentrations. Utilizing the principles of Fick's second law, diffusion coefficients for these processes are derived via exponential approximation. By harnessing machine learning (ML) techniques, including the optimization of neural network hyperparameters via a genetic algorithm (GA) and leveraging various regressors, we effectively pinpointed the optimal model for predicting diffusion coefficients. Notably, for the prediction of soaking coefficients, our model is composed of layers with 64, 64, 32, and 16 neurons, employing ReLU, ELU, sigmoid, and ELU activation functions, respectively. Conversely, for drying coefficients, our model features two hidden layers with 16 and 12 neurons, utilizing sigmoid and ELU activation functions, respectively.

Aircraft Objection Detection Method of Airport Surface based on Improved YOLOv5

An aircraft object detection method on the basis of improved YOLOv5 was proposed to address the issues of large model size, high number of parameters, and inability to meet real-time monitoring requirements of aircrafts in traditional object detection. Firstly, the basic unit of ShuffleNetv2 network was optimized through replacing 3x3 convolution with 5x5 convolution and removing subsequent 1x1 convolution. Simultaneously, the original ReLU activation function was replaced with PReLU. Secondly, CBAM (Convolutional Block Attention Module) attention mechanism was developed to enhance the detection accuracy of the improved network. Finally, improved ShuffleNetv2 network was applied as the backbone structure of YOLOv5. Experimental results revealed that the parameter number of the improved YOLOv5 method introduced in this paper was decreased by 18 times, with a model size of 1.03M. Therefore, a 20.8% increase was achieved in frames per second (FPS) in GPU environments and a 234.6% increase was observed in FPS in CPU environments, while a mean average precision (mAP@0.5) of 0.99 was maintained compared with traditional YOLOv5 network. Because of the advantages of fewer parameters, faster recognition speed, higher localization accuracy, and smaller memory requirement, the developed method was found to be suitable for real-time monitoring of aircrafts in airport surface.

Robust nonparametric regression based on deep ReLU neural networks

In this paper, we consider robust nonparametric regression using deep neural networks with ReLU activation function. While several existing theoretically justified methods are geared towards robustness against identical heavy-tailed noise distributions, the rise of adversarial attacks has emphasized the importance of safeguarding estimation procedures against systematic contamination. We approach this statistical issue by shifting our focus towards estimating conditional distributions. To address it robustly, we introduce a novel estimation procedure based on ℓ-estimation. Under a mild model assumption, we establish general non-asymptotic risk bounds for the resulting estimators, showcasing their robustness against contamination, outliers, and model misspecification. We then delve into the application of our approach using deep ReLU neural networks. When the model is well-specified and the regression function belongs to an α-Hölder class, employing ℓ-type estimation on suitable networks enables the resulting estimators to achieve the minimax optimal rate of convergence. Additionally, we demonstrate that deep ℓ-type estimators can circumvent the curse of dimensionality by assuming the regression function closely resembles the composition of several Hölder functions. To attain this, new deep fully-connected ReLU neural networks have been designed to approximate this composition class. This approximation result can be of independent interest.

EPIDL: Towards efficient and privacy‐preserving inference in deep learning

SummaryDeep learning has shown its great potential in real‐world applications. However, users(clients) who want to use deep learning applications need to send their data to the deep learning service provider (server), which can make the client's data leak to the server, resulting in serious privacy concerns. To address this issue, we propose a protocol named EPIDL to perform efficient and secure inference tasks on neural networks. This protocol enables the client and server to complete inference tasks by performing secure multi‐party computation (MPC) and the client's private data is kept secret from the server. The work in EPIDL can be summarized as follows: First, we optimized the convolution operation and matrix multiplication, such that the total communication can be reduced; Second, we proposed a new method for truncation following secure multiplication based on oblivious transfer and garbled circuits, which will not fail and can be executed together with the ReLU activation function; Finally, we replace complex activation function with MPC‐friendly approximation function. We implement our work in C++ and accelerate the local matrix computation with CUDA support. We evaluate the efficiency of EPIDL in privacy‐preserving deep learning inference tasks, such as the time to execute a secure inference on the MNIST dataset in the LeNet model is about 0.14 s. Compared with the state‐ofthe‐art work, our work is 1.8–98 faster over LAN and WAN, respectively. The experimental results show that our EPIDL is efficient and privacy‐preserving.

An Enhance CNN Model for Brain Tumor Detection from MRI Images

A Convolutional Neural Network (CNN) model for the precise classification and detection of brain tumors from Magnetic Resonance (MR) images is presented in this research. Leveraging the power of deep learning, the proposed CNN model demonstrates remarkable accuracy in distinguishing between different tumor types, aiding in effective diagnosis. The network is trained on a diverse dataset, enabling robust generalization to unseen cases. The proposed method not only enhances the classification performance but also facilitates early tumor detection. Operating on 256 x 256 image inputs, the proposed model boasts a complex architecture featuring six Conv2D layers, batch normalization, five MaxPooling2D layers, six dense layers, and an input layer. Throughout these layers, the ReLU activation function dominates with the final dense layer utilizing the sigmoid function. The potential of sophisticated CNN architectures in medical imaging is highlighted by this study. The outcomes validate the effectiveness of the CNN-based method, advancing computer-aided neuroimaging diagnostic tools and offering a useful tool for medical practitioners. The increased accuracy values demonstrate how well the sophisticated CNN model recognizes brain tumors and point to its potential as a more complex and dependable technique. It also presents a viable avenue for accurate and timely brain tumor identification.

Trish: an efficient activation function for CNN models and analysis of its effectiveness with optimizers in diagnosing glaucoma

Glaucoma is an eye disease that spreads over time without showing any symptoms at an early age and can result in vision loss in advanced ages. The most critical issue in this disease is to detect the symptoms of the disease at an early age. Various researches are carried out on machine learning approaches that will provide support to the expert for this diagnosis. The activation function plays a pivotal role in deep learning models, as it introduces nonlinearity, enabling neural networks to learn complex patterns and relationships within data, thus facilitating accurate predictions and effective feature representations. In this study, it is focused on developing an activation function that can be used in CNN architectures using glaucoma disease datasets. The developed function (Trish) was compared with ReLU, LReLU, Mish, Swish, Smish, and Logish activation functions using SGD, Adam, RmsProp, AdaDelta, AdaGrad, Adamax, and Nadam optimizers in CNN architectures. Datasets consisting of retinal fundus images named ACRIMA and HRF were used within the scope of the experiments. These datasets are widely known and currently used in the literature. To strengthen the test validity, the proposed function was also tested on the CIFAR-10 dataset. As a result of the study, 97.22% validation accuracy performance was obtained. It should be stated that the acquired performance value is at a significant level for the detection of glaucoma.

CD-BTMSE: A Concept Drift detection model based on Bidirectional Temporal Convolutional Network and Multi-Stacking Ensemble learning

The existence of concept drift phenomenon seriously affects the quality of data, it is an urgent need to investigate accurate concept drift detection methods to improve the data quality. Concept drift detection is widely used in the application scenarios of anomaly detection, intrusion detection, fake reviews detection, etc. In recent years, although scholars have proposed many concept drift detection methods, these methods can only cope with a single type of concept drift and are unable to effectively deal with time-series data. In this paper, we propose a Concept Drift detection model based on Bidirectional Temporal convolutional network and Multi-Stacking Ensemble learning (namely CD-BTMSE). CD-BTMSE choose six suitable base learners to solve the problems of overfitting, weak generalization ability and poor robustness of existing ensemble learning-based concept drift detection models, it also innovatively utilizes the Bidirectional Temporal Convolutional Network (BiTCN) model to improve the detection accuracy of concept drift through considering the temporal characteristics of the data as well as the bidirectional semantics in the detection process; At the same time, it utilizes the multi-stacking ensemble learning model to solve the problem of low accuracy of concept drift detection caused by the relatively high generalization error rate and poor generalization ability of the existing ensemble learning-based methods. In addition, the negative log-likelihood loss function in BiTCN model is replaced by a Focal Loss to solve the problem of category inhomogeneity, and the ReLU activation function is replaced by LeakyReLU to solve the neuron ”death” and gradient problems. Extensive experiments on six public datasets show that the proposed CD-BTMSE model outperforms seven state-of-the-art concept drift detection models in terms of accuracy, precision, recall and F1-measure, and it also has better stable performance.

LAMA-WAVELET: IMAGE IMPAINTING WITH HIGH QUALITY OF FINE DETAILS AND OBJECT EDGES

Context. The problem of the image impainting in computer graphic and computer vision systems is considered. The subject of the research is deep learning convolutional neural networks for image inpainting.
 Objective. The objective of the research is to improve the image inpainting performance in computer vision and computer graphics systems by applying wavelet transform in the LaMa-Fourier network architecture.
 Method. The basic LaMa-Fourier network decomposes the image into global and local texture. Then it is proposed to improve the network block, processing the global context of the image, namely, the spectral transform block. To improve the block of spectral transform, instead of Fourier Unit Structure the Simple Wavelet Convolution Block elaborated by the authors is used. In this block, 3D wavelet transform of the image on two levels was initially performed using the Daubechies wavelet db4. The obtained coefficients of 3D wavelet transform are splitted so that each subband represents a separate feature of the image. Convolutional layer, batch normalization and ReLU activation function are sequentially applied to the results of splitting of coefficients on each level of wavelet transform. The obtained subbands of wavelet coefficients are concatenated and the inverse wavelet transform is applied to them, the result of which is the output of the block. Note that the wavelet coefficients at different levels were processed separately. This reduces the computational complexity of calculating the network outputs while preserving the influence of the context of each level on image inpainting. The obtained neural network is named LaMa-Wavelet. The FID, PSNR, SSIM indexes and visual analysis were used to estimate the quality of images inpainted with LaMa-Wavelet network.
 Results. The proposed LaMa-Wavelet network has been implemented in software and researched for solving the problem of image inpainting. The PSNR of images inpainted using the LaMa-Wavelet exceeds the results obtained using the LaMa-Fourier network for narrow and medium masks in average by 4.5%, for large masks in average by 6%. The LaMa-Wavelet applying can enhance SSIM by 2–4% depending on a mask size. But it takes 3 times longer to inpaint one image with LaMa-Wavelet than with LaMa-Fourier network. Analysis of specific images demonstrates that both networks show similar results of inpainting of a homogeneous background. On complex backgrounds with repeating elements the LaMa-Wavelet is often more effective in restoring textures.
 Conclusions. The obtained LaMa-Wavelet network allows to improve the image inpainting with large masks due to applying wavelet transform in the LaMa network architecture. Namely, the quality of reconstruction of image edges and fine details is increased.

Development of a portable electronic nose for the classification of tea quality based on tea dregs aroma

Abstract The current assessment of tea quality is considered subjective. This study aims to develop a portable electronic nose to assess the aroma of tea dregs objectively by relying on the aromatic capture process through sensors and using multilayer perceptron (MLP). A MLP with some hyperparameter variations is used and compared with five machine-learning classifiers. The classification using MLP model with ReLU activation function and 3 hidden layers with 100 hidden nodes resulted in the highest accuracy of 0.8750 ± 0.0241. The MLP model using ReLU activation function is better than Sigmoid while increasing the number of hidden layers and hidden nodes does not necessarily enhance its performance. In the future, this research can be improved by adding sensors to the portable electronic nose, increasing the number of datasets used, and using ensemble learning or deep learning models.

Comparison of Activation Functions in Convolutional Neural Network for Poisson Noisy Image Classification

Deep learning, specifically the Convolutional Neural Network (CNN), has been a significant technology tool for image processing and human health. CNNs, which mimic the working principles of the human brain, can learn robust representations of images. However, CNNs are susceptible to noise interference, which can impact classification performance. Choosing the right activation function can improve CNNs performance and accuracy. This research aims to test the accuracy of CNN with ResNet50, VGG16, and GoogleNet architectures combined with several activation functions such as ReLU, Leaky ReLU, Sigmoid, and Tanh in the classification of images that experience Poisson noise. Poisson noise is applied to each test data to evaluate CNN accuracy. The data used in this study consists of three scenarios of different numbers of classes, namely 3 classes, 5 classes, and 10 classes. The results showed that combining ResNet50 with the ReLU activation function produced the best performance in class recognition in each scenario of the number of classes experiencing Poisson noise interference. The model achieved 97% accuracy for 3-class data, 95% for 5-class data, and 90% for 10-class data. These results show that using ResNet50 with the ReLU activation function can provide excellent resistance to Poisson noise in image processing. It was found that as the number of classes increases, the accuracy of image recognition tends to decrease. This shows that the more complex the image classification task is with a larger number of classes, the more difficult it is for CNNs to distinguish between different classes. Doi: 10.28991/ESJ-2024-08-02-014 Full Text: PDF

An Empirical Evaluation of Adapting Hybrid Parameters for CNN-based Sentiment Analysis

Sentiment analysis aims to understand human emotions and perceptions through various machine-learning pipelines. However, feature engineering and inherent semantic gap constraints often hinder conventional machine learning techniques and limit their accuracy. Newer neural network models have been proposed to automate the feature learning process and enrich learned features with word contextual embeddings to identify their semantic orientations to address these challenges. This article aims to analyze the influence of different factors on the accuracy of sentiment classification predictions by employing Feedforward and Convolutional Neural Networks. To assess the performance of these neural network models, we utilize four diverse real-world datasets, namely 50,000 movie reviews from IMDB, 10,662 sentences from LightSide Movie_Reviews, 300 public movie reviews, and 1,600,000 tweets extracted from Sentiment140. We experimentally investigate the impact of exploiting GloVe word embeddings on enriching feature vectors extracted from sentiment sentences. Findings indicate that using larger dimensions of GloVe word embeddings increases the sentiment classification accuracy. In particular, results demonstrate that the accuracy of the CNN with a larger feature map, a smaller filter window, and the ReLU activation function in the convolutional layer was 90.56% using the IMDB dataset. In comparison, it was 80.73% and 77.64% using the sentiment140 and the 300 sentiment sentences dataset, respectively. However, it is worth mentioning that, with large-size sentiment sentences (LightSide’s Movie Reviews) and using the same parameters, only a 64.44% level of accuracy was achieved.

Determination of Radiation Value by Month Using Artificial Neural Network Model; Ankara, Sivas, Erzurum example

This research examines the estimation of solar radiation using artificial neural network (ANN) models in Turkish cities with similar latitude values such as Ankara, Sivas and Erzurum. The aim of this study is to investigate whether cities at similar latitudes exhibit similar trends in solar radiation values, despite their geographical differences. In this study, solar radiation prediction was made for 3 cities with a single-layer neural network. Monthly solar radiation intensity was estimated for the 10-year period between 2012 and 2022 with a total of 4764 samples taken from the General Directorate of State Meteorology. An artificial neural network model was developed with 8 units in the first hidden layer and 4 units in the second hidden layer. The optimizer used in compiling the model was determined as Adam, the loss function as 'mean_squared_error' and the metric as 'mse'. ReLU activation function was used in the input layer and hidden layers. A 10-year solar radiation intensity value was used in the output layer. 70% of the data set is reserved for training and 30% for testing data set. As a result, similar solar radiation trends were obtained in the same latitude regions, the results were confirmed by meteorological data.

Complexity of Neural Network Training and ETR: Extensions with Effectively Continuous Functions

The training problem of neural networks (NNs) is known to be ER-complete with respect to ReLU and linear activation functions. We show that the training problem for NNs equipped with arbitrary activation functions is polynomial-time bireducible to the existential theory of the reals extended with the corresponding activation functions. For effectively continuous activation functions (e.g., the sigmoid function), we obtain an inclusion to low levels of the arithmetical hierarchy. Consequently, the sigmoid activation function leads to the existential theory of the reals with the exponential function, and hence the decidability of training NNs using the sigmoid activation function is equivalent to the decidability of the existential theory of the reals with the exponential function, a long-standing open problem. In contrast, we obtain that the training problem is undecidable if sinusoidal activation functions are considered.

DeepBern-Nets: Taming the Complexity of Certifying Neural Networks Using Bernstein Polynomial Activations and Precise Bound Propagation

Formal certification of Neural Networks (NNs) is crucial for ensuring their safety, fairness, and robustness. Unfortunately, on the one hand, sound and complete certification algorithms of ReLU-based NNs do not scale to large-scale NNs. On the other hand, incomplete certification algorithms are easier to compute, but they result in loose bounds that deteriorate with the depth of NN, which diminishes their effectiveness. In this paper, we ask the following question; can we replace the ReLU activation function with one that opens the door to incomplete certification algorithms that are easy to compute but can produce tight bounds on the NN's outputs? We introduce DeepBern-Nets, a class of NNs with activation functions based on Bernstein polynomials instead of the commonly used ReLU activation. Bernstein polynomials are smooth and differentiable functions with desirable properties such as the so-called range enclosure and subdivision properties. We design a novel Interval Bound Propagation (IBP) algorithm, called Bern-IBP, to efficiently compute tight bounds on DeepBern-Nets outputs. Our approach leverages the properties of Bernstein polynomials to improve the tractability of neural network certification tasks while maintaining the accuracy of the trained networks. We conduct experiments in adversarial robustness and reachability analysis settings to assess the effectiveness of the approach. Our proposed framework achieves high certified accuracy for adversarially-trained NNs, which is often a challenging task for certifiers of ReLU-based NNs. This work establishes Bernstein polynomial activation as a promising alternative for improving NN certification tasks across various NNs applications.

ASPP+-LANet: A Multi-Scale Context Extraction Network for Semantic Segmentation of High-Resolution Remote Sensing Images

Semantic segmentation of remote sensing (RS) images is a pivotal branch in the realm of RS image processing, which plays a significant role in urban planning, building extraction, vegetation extraction, etc. With the continuous advancement of remote sensing technology, the spatial resolution of remote sensing images is progressively improving. This escalation in resolution gives rise to challenges like imbalanced class distributions among ground objects in RS images, the significant variations of ground object scales, as well as the presence of redundant information and noise interference. In this paper, we propose a multi-scale context extraction network, ASPP+-LANet, based on the LANet for semantic segmentation of high-resolution RS images. Firstly, we design an ASPP+ module, expanding upon the ASPP module by incorporating an additional feature extraction channel, redesigning the dilation rates, and introducing the Coordinate Attention (CA) mechanism so that it can effectively improve the segmentation performance of ground object targets at different scales. Secondly, we introduce the Funnel ReLU (FReLU) activation function for enhancing the segmentation effect of slender ground object targets and refining the segmentation edges. The experimental results show that our network model demonstrates superior segmentation performance on both Potsdam and Vaihingen datasets, outperforming other state-of-the-art (SOTA) methods.

BI-TST_YOLOv5: Ground Defect Recognition Algorithm Based on Improved YOLOv5 Model

Pavement defect detection technology stands as a pivotal component within intelligent driving systems, demanding heightened precision and rapid detection rates. Addressing the complexities arising from diverse defect types and intricate backgrounds in visual sensing, this study introduces an enhanced approach to augment the network structure and activation function within the foundational YOLOv5 algorithm. Initially, modifications to the YOLOv5′s architecture incorporate an adjustment to the Leaky ReLU activation function, thereby enhancing regression stability and accuracy. Subsequently, the integration of bi-level routing attention into the network’s head layer optimizes the attention mechanism, notably improving overall efficiency. Additionally, the replacement of the YOLOv5 backbone layer’s C3 module with the C3-TST module enhances initial convergence efficiency in target detection. Comparative analysis against the original YOLOv5s network reveals a 2% enhancement in map50 and a 1.8% improvement in F1, signifying an overall advancement in network performance. The initial convergence rate of the algorithm has been improved, and the accuracy and operational efficiency have also been greatly improved, especially on models with small-scale training sets.

Low dimensional approximation and generalization of multivariate functions on smooth manifolds using deep ReLU neural networks

The expressive power of deep neural networks is manifested by their remarkable ability to approximate multivariate functions in a way that appears to overcome the curse of dimensionality. This ability is exemplified by their success in solving high-dimensional problems where traditional numerical solvers fail due to their limitations in accurately representing high-dimensional structures. To provide a theoretical framework for explaining this phenomenon, we analyze the approximation of Hölder functions defined on a d-dimensional smooth manifold M embedded in RD, with d≪D, using deep neural networks. We prove that the uniform convergence estimates of the approximation and generalization errors by deep neural networks with ReLU activation functions do not depend on the ambient dimension D of the function but only on its lower manifold dimension d, in a precise sense. Our result improves existing results from the literature where approximation and generalization errors were shown to depend weakly on D.