Activation Function Of Neural Network Research Articles

An Improved Activation Function in Convolution Neural Network to Estimate the Hazardous Air Pollutant Based on Images

An Improved Activation Function in Convolution Neural Network to Estimate the Hazardous Air Pollutant Based on Images

Activation Functions for Neural Networks: Application and Performance-based Comparison

Past decade has seen explosive growth of Deep Learning (DL) algorithms based on Artificial Neural Networks (ANNs) and its applications in vast emerging domains to solve real world complex problems. The DL architecture uses Activation Functions (AFs), to perform the task of finding relationship between the input feature and the output. Essential building blocks of any ANN are AFs which bring the required non-linearity of the output in the Output layer of network. Layers of ANNs are combinations of linear and nonlinear AFs. Most extensively used AFs are Sigmoid, Hyperbolic Tangent (Tanh), Rectified Linear Unit (ReLU) etc to name a few. Choosing an AF for a particular AF depends on various factors such as Nature of Application, Design of ANN, Optimizers used in the network, Complexity of Data etc. This paper presents a survey on most widely used AFs along with the important consideration while selecting an AF on a specific problem domain. A broad guideline on selecting an AF based on the literature survey has been presented to help researchers in employing suitable AF in their problem domain.

Comparative study of integer-order and fractional-order artificial neural networks: Application for mathematical function generation

This paper investigates the impact of fractional derivatives on the activation functions of an artificial neural network (ANN). Based on the results and analysis, a three-layer backpropagation neural network model with fractional and integer derivatives employed in the activation function and fractional gradient descent method backpropagation learning algorithm has been proposed. Specifically, three perceptrons have been proposed based on fractional and integer derivatives applied to activation functions and learning algorithms. They are fractional derivative activation function (FDAF) perceptron, integer derivative activation function (IDAF)-fractional derivative learning algorithm (FDLA) perceptron, and fractional derivative learning algorithm (FDLA) perceptron. The Riemann-Liouville (RL) fractional derivative, Grunwald-Letnikov (GL) derivative, Caputo-Fabrizio (CF) fractional derivative, Caputo (C) fractional derivative, and Atangana-Baleanu (AB) fractional derivative have been employed. The impact of these derivatives’ fractional-order (FO) is investigated in a wide range from 0.1−0.9 on testing mean square error (MSE) and time required to train the perceptron. FO-based and IO-based perceptrons are compared with the help of performance metrics such as testing MSE and the time required to train the models. The training and testing simulation results illustrate that Caputo-Fabrizio derivative-based perceptrons outperform other fractional derivatives. Also, the performance of the FO-based perceptron is better in terms of the least MSE.

An Innovative Deep Learning Approach for Image Semantic and Instance Segmentation

In this study, we propose a segmentation model based on convolutional neural networks (CNNs) to address image segmentation challenges in computer vision. Prior to designing the model, the activation function and other modules of the convolutional neural network were optimized to meet specific requirements. The segmentation task was transformed into binary classification problem to simplify network calculations and improve efficiency. Additionally, the model utilized a mask map obtained from the semantic segmentation model to aid in instance segmentation. Class activation technology was introduced to extract feature mapping maps. The corresponding thermal maps were obtained to achieve target instance segmentation. To further validate the effectiveness of the segmentation model, simulation experiments were conducted on semantic segmentation and instance segmentation respectively. The results show that the accuracy of the basic semantic segmentation model reached 87.58%, while the average accuracy of the entire class of the optimized instance segmentation model reached 97.9%. Therefore, the research and design of image segmentation models demonstrate high accuracy and good robustness.

ReActHE: A homomorphic encryption friendly deep neural network for privacy-preserving biomedical prediction

The growing distribution of deep learning models to individuals’ devices on sensitive healthcare data introduces challenging privacy and security problems when computation is being operated on an untrusted server. Homomorphic encryption (HE) is one of the appropriate cryptographic techniques to provide secure machine learning computation by directly computing over encrypted data, so that allows the data owner and model owner to outsource processing of sensitive information to an untrusted server without leaking any information about the data. However, most current HE schemes only support limited arithmetic operations, which significantly hinder their applications to implement a secure deep learning algorithm, especially on the nonlinear activation function of a deep neural network. In this paper, we develop a novel HE-friendly deep neural network, named REsidue ACTivation HE (ReActHE), to implement a precise and privacy-preserving algorithm with a non-approximating HE scheme on the activation function. We consider a residue activation strategy with a scaled power activation function in a deep neural network for HE-friendly nonlinear activation. Moreover, we propose a residue activation network structure to constrain the latent space in the training process to alleviate the optimization difficulty. We comprehensively evaluate the proposed ReActHE method using various biomedical datasets and widely-used image datasets. Our results demonstrate that ReActHE outperforms other alternative solutions to secure machine learning with HE and achieves low approximation errors in classification and regression tasks.

Comparison of Activation Functions in Convolutional Neural Network for Poisson Noisy Image Classification

Deep learning, specifically the Convolutional Neural Network (CNN), has been a significant technology tool for image processing and human health. CNNs, which mimic the working principles of the human brain, can learn robust representations of images. However, CNNs are susceptible to noise interference, which can impact classification performance. Choosing the right activation function can improve CNNs performance and accuracy. This research aims to test the accuracy of CNN with ResNet50, VGG16, and GoogleNet architectures combined with several activation functions such as ReLU, Leaky ReLU, Sigmoid, and Tanh in the classification of images that experience Poisson noise. Poisson noise is applied to each test data to evaluate CNN accuracy. The data used in this study consists of three scenarios of different numbers of classes, namely 3 classes, 5 classes, and 10 classes. The results showed that combining ResNet50 with the ReLU activation function produced the best performance in class recognition in each scenario of the number of classes experiencing Poisson noise interference. The model achieved 97% accuracy for 3-class data, 95% for 5-class data, and 90% for 10-class data. These results show that using ResNet50 with the ReLU activation function can provide excellent resistance to Poisson noise in image processing. It was found that as the number of classes increases, the accuracy of image recognition tends to decrease. This shows that the more complex the image classification task is with a larger number of classes, the more difficult it is for CNNs to distinguish between different classes. Doi: 10.28991/ESJ-2024-08-02-014 Full Text: PDF

Amplifying Sine Unit: An Oscillatory Activation Function for Deep Neural Networks to Recover Nonlinear Oscillations Efficiently

Many industrial and real-life problems exhibit highly nonlinear periodic behaviors and the conventional methods may fall short of finding their analytical or closed-form solutions. Such problems demand cutting-edge computational tools with increased functionality and reduced cost. Recently, deep neural networks have gained massive research interest due to their ability to handle large data and universality to learn complex functions. In this work, we put forward a methodology based on deep neural networks with responsive layers structure to deal nonlinear oscillations in microelectromechanical systems. We incorporated some oscillatory and non-oscillatory activation functions such as growing cosine unit known as GCU, Sine, Mish and Tanh in our designed network to have a comprehensive analysis on their performance for highly nonlinear problems. Integrating oscillatory activation functions with deep neural networks definitely outperform in predicting the periodic patterns of underlying systems. To support oscillatory actuation for nonlinear systems, we have proposed a novel oscillatory activation function called Amplifying Sine Unit denoted as ASU which is more efficient than GCU for complex vibratory systems such as microelectromechanical systems. Experimental results show that the designed network with our proposed activation function ASU is more reliable and robust to handle the challenges posed by oscillations

Performance of Fourier-based activation function in physics-informed neural networks for patient-specific cardiovascular flows

Background and ObjectivesPhysics-informed neural networks (PINNs) can be used to inversely model complex physical systems by encoding the governing partial differential equations and training data into the neural network. However, neural networks are known to be biased towards learning less complex functions, called spectral bias. This has important implications in modeling cardiovascular flows, where spatial frequencies can vary substantially across anatomies and pathologies (e.g., aneurysms or stenoses). Recent evidence suggests that Fourier-based activation functions have desirable properties, and can potentially reduce spectral bias; however, the performance and adequacy of such Fourier activation functions have not yet been evaluated in patient-specific cardiovascular flow applications. MethodsThe performance of sine activation function was evaluated against tanh and swish activation functions in a 1D advection-diffusion problem, an eccentric 2D stenosis model (Re=5000), and a patient-specific 3D aortic model (Re=823) under pulsatile flow conditions. CFD simulations were performed at high spatio-temporal resolution and data points were extracted for training the neural network. The number of training data points were normalized by L/D. The performance of the PINNs framework was evaluated with increasing number of training data points and across all three activation functions. ResultsOur results demonstrate that sine activation function presents desirable characteristics, such as monotonic reduction in errors, relatively faster convergence, and accurate eigen spectra at higher modes, compared to tanh and swish activation functions. Interestingly, for all activation functions, the domain-averaged errors tended to asymptote at ≈15−20% despite substantial increase in training point density. For 2D eccentric stenosis, errors asymptoted at a sensor point density of 40L/D. For 3D patient-specific aorta, this asymptote was achieved at 180L/D for all three activation functions with an error of ≈15% although sine activation function demonstrated relatively faster convergence. ConclusionsWe have demonstrated that Fourier-based activation functions have higher performance in terms of accuracy and convergence properties for cardiovascular flow applications; however, inherent challenges of neural networks (e.g., spectral bias) can limit the accuracy to ≈15% under physiological, 3D patient-specific blood flow conditions.

ModSwish: a new activation function for neural network

modSwish: a new activation function for neural network

A Formal Characterization of Activation Functions in Deep Neural Networks.

In this article, a mathematical formulation for describing and designing activation functions in deep neural networks is provided. The methodology is based on a precise characterization of the desired activation functions that satisfy particular criteria, including circumventing vanishing or exploding gradients during training. The problem of finding desired activation functions is formulated as an infinite-dimensional optimization problem, which is later relaxed to solving a partial differential equation. Furthermore, bounds that guarantee the optimality of the designed activation function are provided. Relevant examples with some state-of-the-art activation functions are provided to illustrate the methodology.

Multiobjective Multiverse Optimizer for Multirobotic U-Shaped Disassembly Line Balancing Problems

The development of technology accelerates the upgrade of products, which results in a significant number of obsolete products. This research aims to solve the Multi-robotic Multi-product U-shaped Disassembly-line-balancing Problem (M2UDP), in which different products are disassembled on a U-shaped model in a pre-set cycle time and assigned tasks to robots in each workstation reasonably. A linear mixed-integer model is established to maximize disassembly profit and minimize carbon emissions. An improved multi-objective multi-verse optimizer (IMMO) that utilizes the sigmoid activation function in neural networks is proposed to find the optimal plan for the model. The improved algorithm is verified via a set of real-life instances and compared with three classical multi-objective optimization algorithms. The experimental results show that the proposed IMMO performs better than those peers in solving the M2UDP problems.

An Application of Fuzzy Symbolic Time-Series for Energy Demand Forecasting

In this paper, we present a new fuzzy symbolization technique for energy load forecasting with neural networks, FPLS-Sym. Symbolization techniques transform a numerical time series into a smaller string of symbols, providing a high-level representation of time series by combining segmentation, aggregation and discretization. The dimensional reduction obtained with symbolization can speed up substantially the time required to train neural networks, however, it can also lead to considerable information losses that could lead to a less accurate forecast. FPLS-Sym introduces the use of fuzzy logic in the discretization process, maintaining more information about each segment of the neural network at the expense of requiring more space in memory. Extensive experimentation was made to evaluate FPLS-Sym with various neural-network-based models, including different neural network architectures and activation functions. The evaluation was done with energy demand data from Spain taken from 2009 to 2019. Results show that FPLS-Sym provides better quality metrics than other symbolization techniques and outperforms the use of the standard numerical time series representation in both quality metrics and training time.

10 GHz Optoelectronic Circuit with Variable Sigmoid Shaped Activation Function for Photonic Neural Networks

10 GHz Optoelectronic Circuit with Variable Sigmoid Shaped Activation Function for Photonic Neural Networks

Deep convolutional neural network and IoT technology for healthcare.

Deep convolutional neural network and IoT technology for healthcare.

Convolutional Neural Network Activation Function Performance on Image Recognition of The Batak Script

Deep Learning is a sub-set of Machine learning, Deep Learning is widely used to solve problems in various fields. One of the popular deep learning architectures is The Convolutional Neural Network (CNN), CNN has a layer that transforms feature extraction automatically so it is widely used in image recognition. However, CNN's performance using the tanh function is still relatively low, therefore it is necessary to select the right activation function to improve accuracy performance. This study analyzes the use of the activation function in image recognition of the Batak script. The result of this study is that the CNN model using the ReLU and eLU functions produces the highest accuracy compared to the CNN model using the tanh function. The CNN model using eLU produces the best accuracy performance in the training process, which is 99.71% with an error value of 0.0108. Meanwhile, in the testing process, the highest accuracy value is generated by the CNN Model using the ReLU function with an accuracy of 94.11%, an error value of 0.3282, a precision value of 0.9411, a recall of 0.9411, and an f1-score of 0.9416.

HeFUN: Homomorphic Encryption for Unconstrained Secure Neural Network Inference

Homomorphic encryption (HE) has emerged as a pivotal technology for secure neural network inference (SNNI), offering privacy-preserving computations on encrypted data. Despite active developments in this field, HE-based SNNI frameworks are impeded by three inherent limitations. Firstly, they cannot evaluate non-linear functions such as ReLU, the most widely adopted activation function in neural networks. Secondly, the permitted number of homomorphic operations on ciphertexts is bounded, consequently limiting the depth of neural networks that can be evaluated. Thirdly, the computational overhead associated with HE is prohibitively high, particularly for deep neural networks. In this paper, we introduce a novel paradigm designed to address the three limitations of HE-based SNNI. Our approach is an interactive approach that is solely based on HE, called iLHE. Utilizing the idea of iLHE, we present two protocols: ReLU, which facilitates the direct evaluation of the ReLU function on encrypted data, tackling the first limitation, and HeRefresh, which extends the feasible depth of neural network computations and mitigates the computational overhead, thereby addressing the second and third limitations. Based on HeReLU and HeRefresh protocols, we build a new framework for SNNI, named HeFUN. We prove that our protocols and the HeFUN framework are secure in the semi-honest security model. Empirical evaluations demonstrate that HeFUN surpasses current HE-based SNNI frameworks in multiple aspects, including security, accuracy, the number of communication rounds, and inference latency. Specifically, for a convolutional neural network with four layers on the MNIST dataset, HeFUN achieves 99.16% accuracy with an inference latency of 1.501 s, surpassing the popular HE-based framework CryptoNets proposed by Gilad-Bachrach, which achieves 98.52% accuracy with an inference latency of 3.479 s.

Application of multi-directional FG material to improve natural frequencies of perovskite solar cells under mechanical shock validated by deep-learning approach

In order to generate electricity from sunlight, spacecraft that operate within the inner solar system often utilize photovoltaic solar panels that are controlled by power electronics. For this purpose, the emerging technologies of perovskite solar cells (PSCs) have attracted plenty of attention recently. Improving the stability of this kind of cell is the main purpose of this research. In the current work, for the first time, dynamic stability analysis of the PSCs, including the multi-directional functionally graded (MD-FG) layer as the host layer, is presented. Also, due to the complex situation of solar cells, this kind of system may be under mechanical shock loading. Various types of PSCs, including cylindrical, spherical, and hyperbolic paraboloid solar cells, by changing curvature factors are modeled. For considering the size-dependency of the PSCs, modified couple stress theory (MCST) with one length scale factor on the strain energy is presented. The forced vibration caused by mechanical shock loading is illustrated by inverting the Laplace transform using the fully modified representation of Abate and Dubner’s solution. A deep neural network (DNN) with independent variables, hidden, input, and output layers, intrinsic neural network parameters, activation function, hidden neurons, and loss function is offered to train the current appropriate structure once the results of mathematical modeling have been obtained. Finally, by applying the two-dimensional differential quadrature method (2D-DQM) as an accurate integral operator, the discretized motion equations can be derived from the weak form of the equations. Finally, in the results section, some suggestion for improving the stability and frequency of the PSCs is discussed.

A surface-normal photodetector as nonlinear activation function in diffractive optical neural networks

Optical neural networks (ONNs) enable high speed, parallel, and energy efficient processing compared to their conventional digital electronic counterparts. However, realizing large scale ONN systems is an open problem. Among various integrated and non-integrated ONNs, free-space diffractive ONNs benefit from a large number of pixels of spatial light modulators to realize millions of neurons. However, a significant fraction of computation time and energy is consumed by the nonlinear activation function that is typically implemented using a camera sensor. Here, we propose a novel surface-normal photodetector (SNPD) with an optical-in–electrical-out (O–E) nonlinear response to replace the camera sensor that enables about three orders of magnitude faster (5.7 µs response time) and more energy efficient (less than 10 nW/pixel) response. Direct efficient vertical optical coupling, polarization insensitivity, inherent nonlinearity with no control electronics, low optical power requirements, and the possibility of implementing large scale arrays make the SNPD a promising O–E nonlinear activation function for diffractive ONNs. To show the applicability of the proposed neural nonlinearity, successful classification simulations of the MNIST and Fashion MNIST datasets using the measured response of SNPD with accuracy comparable to that of an ideal ReLU function are demonstrated.

Implementation of the SoftMax Activation for Reconfigurable Neural Network Hardware Accelerators

In recent decades, machine-learning algorithms have been extensively utilized to tackle various complex tasks. To achieve the high performance and efficiency of these algorithms, various hardware accelerators are used. Typically, these devices are specialized for specific neural network architectures and activation functions. However, state-of-the-art complex autonomous and mobile systems may require different algorithms for different tasks. Reconfigurable accelerators can be used to resolve this problem. They possess the capability to support diverse neural network architectures and allow for significant alterations to the implemented model at runtime. Thus, a single device can be used to address entirely different tasks. Our research focuses on dynamically reconfigurable accelerators based on reconfigurable computing environments (RCE). To implement the required neural networks on such devices, their algorithms need to be adapted to the homogeneous structure of RCE. This article proposes the first implementation of the widely used SoftMax activation for hardware accelerators based on RCE. The implementation leverages spatial distribution and incorporates several optimizations to enhance its performance. The timing simulation of the proposed implementation on FPGA shows a high throughput of 1.12 Gbps at 23 MHz. The result is comparable to counterparts lacking reconfiguration capability. However, this flexibility comes at the expense of the increased consumption of logic elements.

Graphene/silicon heterojunction for reconfigurable phase-relevant activation function in coherent optical neural networks

Optical neural networks (ONNs) herald a new era in information and communication technologies and have implemented various intelligent applications. In an ONN, the activation function (AF) is a crucial component determining the network performances and on-chip AF devices are still in development. Here, we first demonstrate on-chip reconfigurable AF devices with phase activation fulfilled by dual-functional graphene/silicon (Gra/Si) heterojunctions. With optical modulation and detection in one device, time delays are shorter, energy consumption is lower, reconfigurability is higher and the device footprint is smaller than other on-chip AF strategies. The experimental modulation voltage (power) of our Gra/Si heterojunction achieves as low as 1 V (0.5 mW), superior to many pure silicon counterparts. In the photodetection aspect, a high responsivity of over 200 mA/W is realized. Special nonlinear functions generated are fed into a complex-valued ONN to challenge handwritten letters and image recognition tasks, showing improved accuracy and potential of high-efficient, all-component-integration on-chip ONN. Our results offer new insights for on-chip ONN devices and pave the way to high-performance integrated optoelectronic computing circuits.