Activation Function Of Neural Network Research Articles

Dynamics analysis and FPGA implementation of discrete memristive cellular neural network with heterogeneous activation functions

The activation function, as an important component of artificial neural networks, endows neural networks with rich dynamical phenomena by virtue of its nonlinear properties. However, especially for cellular neural networks (CNNs), the exploration of neural networks consisting of heterogeneous activation functions has not been sufficient. In this article, we introduce heterogeneous activation functions in discrete cellular neural network (DCNN) for the first time. In order to enhance the dynamics of the original DCNN, we propose our discrete memristive cellular neural network (DMCNN) based on a memristor with a sinusoidal function. Using various methods of dynamical characterization, such as Lyapunov exponential analysis, bifurcation diagrams and spectral entropy, the rich dynamical behaviour of the proposed model is comprehensively investigated, including hyperchaos, transient chaos, high spectral entropy and multiple types of coexisting attractors in the proposed model. Last but not least, the hardware platform of the proposed model is implemented by field programmable gate array (FPGA), and the dynamics of the proposed model is verified by a combination of software simulation and hardware experiments. The new exploration of DMCNN with heterogeneous activation functions in this article lays the foundation for further research into neural networks with complex dynamical behaviour.

Degree of Lp Approximation Using Activated Singular Integrals

In this article we present the Lp, p≥1, approximation properties of activated singular integral operators over the real line. We establish their approximation to the unit operator with rates. The kernels here come from neural network activation functions and we employ the related density functions. The derived inequalities use the high order Lp modulus of smoothness.

Comparative Performance of Different Convolutional Neural Network Activation Functions on Image Classification

Comparative Performance of Different Convolutional Neural Network Activation Functions on Image Classification

Adaptive Morphing Activation Function for Neural Networks

Adaptive Morphing Activation Function for Neural Networks

A reconfigurable multi-precision quantization-aware nonlinear activation function hardware module for DNNs

In recent years, the increasing variety of nonlinear activation functions (NAFs) in deep neural networks (DNNs) has led to higher computational demands. However, hardware implementation faces challenges such as lack of flexibility, high hardware cost, and limited accuracy. This paper proposes a highly flexible and low-cost hardware solution for implementing activation functions to overcome these issues. Based on the piecewise linear (PWL) approximation method, our method supports NAFs with different accuracy configurations through a customized implementation strategy to meet the requirements in different scenario applications. In this paper, the symmetry of the activation function is investigated, and incorporate curve translation preprocessing and data quantization to significantly reduce hardware storage costs. The modular hardware architecture proposed in this study supports NAFs of multiple accuracies, which is suitable for designing deep learning neural network accelerators in various scenarios, avoiding the need to design dedicated hardware circuits for the activation function layer and enhances circuit design efficiency. The proposed hardware architecture is validated on the Xilinx XC7Z010 development board. The experimental results show that the average absolute error (AAE) is reduced by about 35.6 % at a clock frequency of 312.5 MHz. Additionally, the accuracy loss of the model is maximized to −0.684 % after replacing the activation layer function of DNNs under the PyTorch framework.

Computational Investigation of the Potential and Limitations of Machine Learning with Neural Network Circuits Based on Synaptic Transistors.

Synaptic transistors have been proposed to implement neuron activation functions of neural networks (NNs). While promising to enable compact, fast, inexpensive, and energy-efficient dedicated NN circuits, they also have limitations compared to digital NNs (realized as codes for digital processors), including shape choices of the activation function using particular types of transistor implementation, and instabilities due to noise and other factors present in analog circuits. We present a computational study of the effects of these factors on NN performance and find that, while accuracy competitive with traditional NNs can be realized for many applications, there is high sensitivity to the instability in the shape of the activation function, suggesting that, when highly accurate NNs are required, high-precision circuitry should be developed beyond what has been reported for synaptic transistors to date.

Quantitative Uniform Approximation by Activated Singular Operators

In this article, we study the approximation properties of activated singular integral operators over the real line. We establish their convergence to the unit operator with rates. The kernels here derive from neural network activation functions and their corresponding density functions. The estimates are mostly sharp, and they are pointwise and uniform. The derived inequalities involve the higher order modulus of smoothness.

A Neural-Network-Based Watermarking Method Approximating JPEG Quantization.

We propose a neural-network-based watermarking method that introduces the quantized activation function that approximates the quantization of JPEG compression. Many neural-network-based watermarking methods have been proposed. Conventional methods have acquired robustness against various attacks by introducing an attack simulation layer between the embedding network and the extraction network. The quantization process of JPEG compression is replaced by the noise addition process in the attack layer of conventional methods. In this paper, we propose a quantized activation function that can simulate the JPEG quantization standard as it is in order to improve the robustness against the JPEG compression. Our quantized activation function consists of several hyperbolic tangent functions and is applied as an activation function for neural networks. Our network was introduced in the attack layer of ReDMark proposed by Ahmadi et al. to compare it with their method. That is, the embedding and extraction networks had the same structure. We compared the usual JPEG compressed images and the images applying the quantized activation function. The results showed that a network with quantized activation functions can approximate JPEG compression with high accuracy. We also compared the bit error rate (BER) of estimated watermarks generated by our network with those generated by ReDMark. We found that our network was able to produce estimated watermarks with lower BERs than those of ReDMark. Therefore, our network outperformed the conventional method with respect to image quality and BER.

Comparative study of integer-order and fractional-order artificial neural networks: Application for mathematical function generation

This paper investigates the impact of fractional derivatives on the activation functions of an artificial neural network (ANN). Based on the results and analysis, a three-layer backpropagation neural network model with fractional and integer derivatives employed in the activation function and fractional gradient descent method backpropagation learning algorithm has been proposed. Specifically, three perceptrons have been proposed based on fractional and integer derivatives applied to activation functions and learning algorithms. They are fractional derivative activation function (FDAF) perceptron, integer derivative activation function (IDAF)-fractional derivative learning algorithm (FDLA) perceptron, and fractional derivative learning algorithm (FDLA) perceptron. The Riemann-Liouville (RL) fractional derivative, Grunwald-Letnikov (GL) derivative, Caputo-Fabrizio (CF) fractional derivative, Caputo (C) fractional derivative, and Atangana-Baleanu (AB) fractional derivative have been employed. The impact of these derivatives’ fractional-order (FO) is investigated in a wide range from 0.1−0.9 on testing mean square error (MSE) and time required to train the perceptron. FO-based and IO-based perceptrons are compared with the help of performance metrics such as testing MSE and the time required to train the models. The training and testing simulation results illustrate that Caputo-Fabrizio derivative-based perceptrons outperform other fractional derivatives. Also, the performance of the FO-based perceptron is better in terms of the least MSE.

An Improved Activation Function in Convolution Neural Network to Estimate the Hazardous Air Pollutant Based on Images

An Improved Activation Function in Convolution Neural Network to Estimate the Hazardous Air Pollutant Based on Images

Activation Functions for Neural Networks: Application and Performance-based Comparison

Past decade has seen explosive growth of Deep Learning (DL) algorithms based on Artificial Neural Networks (ANNs) and its applications in vast emerging domains to solve real world complex problems. The DL architecture uses Activation Functions (AFs), to perform the task of finding relationship between the input feature and the output. Essential building blocks of any ANN are AFs which bring the required non-linearity of the output in the Output layer of network. Layers of ANNs are combinations of linear and nonlinear AFs. Most extensively used AFs are Sigmoid, Hyperbolic Tangent (Tanh), Rectified Linear Unit (ReLU) etc to name a few. Choosing an AF for a particular AF depends on various factors such as Nature of Application, Design of ANN, Optimizers used in the network, Complexity of Data etc. This paper presents a survey on most widely used AFs along with the important consideration while selecting an AF on a specific problem domain. A broad guideline on selecting an AF based on the literature survey has been presented to help researchers in employing suitable AF in their problem domain.

Anti-derivatives approximator for enhancing physics-informed neural networks

This study presents a novel strategy for constructing an approximator for arbitrary univariate functions. The proposed approximation utilizes the anti-derivatives of a Fourier series expansion for the presumed piecewise function, resulting in a remarkable feature that enables the simultaneous approximation of an arbitrary function and its anti-derivatives. These anti-derivatives can be employed to discover solution curves for systems of ordinary differential equations based on an optimization scheme, even in the presence of chaotic dynamics. Additionally, the anti-derivatives approximator is extended as an adaptive activation function for physics-informed neural networks, leveraging the high-order differentiability of the anti-derivatives. Systematic experiments have demonstrated the outstanding merits of the proposed anti-derivatives-based approximator, including its ability to construct regression models for volatile data and their anti-derivatives, solve differential equations, and enhance the capabilities of physics-informed neural networks.

An Innovative Deep Learning Approach for Image Semantic and Instance Segmentation

In this study, we propose a segmentation model based on convolutional neural networks (CNNs) to address image segmentation challenges in computer vision. Prior to designing the model, the activation function and other modules of the convolutional neural network were optimized to meet specific requirements. The segmentation task was transformed into binary classification problem to simplify network calculations and improve efficiency. Additionally, the model utilized a mask map obtained from the semantic segmentation model to aid in instance segmentation. Class activation technology was introduced to extract feature mapping maps. The corresponding thermal maps were obtained to achieve target instance segmentation. To further validate the effectiveness of the segmentation model, simulation experiments were conducted on semantic segmentation and instance segmentation respectively. The results show that the accuracy of the basic semantic segmentation model reached 87.58%, while the average accuracy of the entire class of the optimized instance segmentation model reached 97.9%. Therefore, the research and design of image segmentation models demonstrate high accuracy and good robustness.

ReActHE: A homomorphic encryption friendly deep neural network for privacy-preserving biomedical prediction

The growing distribution of deep learning models to individuals’ devices on sensitive healthcare data introduces challenging privacy and security problems when computation is being operated on an untrusted server. Homomorphic encryption (HE) is one of the appropriate cryptographic techniques to provide secure machine learning computation by directly computing over encrypted data, so that allows the data owner and model owner to outsource processing of sensitive information to an untrusted server without leaking any information about the data. However, most current HE schemes only support limited arithmetic operations, which significantly hinder their applications to implement a secure deep learning algorithm, especially on the nonlinear activation function of a deep neural network. In this paper, we develop a novel HE-friendly deep neural network, named REsidue ACTivation HE (ReActHE), to implement a precise and privacy-preserving algorithm with a non-approximating HE scheme on the activation function. We consider a residue activation strategy with a scaled power activation function in a deep neural network for HE-friendly nonlinear activation. Moreover, we propose a residue activation network structure to constrain the latent space in the training process to alleviate the optimization difficulty. We comprehensively evaluate the proposed ReActHE method using various biomedical datasets and widely-used image datasets. Our results demonstrate that ReActHE outperforms other alternative solutions to secure machine learning with HE and achieves low approximation errors in classification and regression tasks.

Comparison of Activation Functions in Convolutional Neural Network for Poisson Noisy Image Classification

Deep learning, specifically the Convolutional Neural Network (CNN), has been a significant technology tool for image processing and human health. CNNs, which mimic the working principles of the human brain, can learn robust representations of images. However, CNNs are susceptible to noise interference, which can impact classification performance. Choosing the right activation function can improve CNNs performance and accuracy. This research aims to test the accuracy of CNN with ResNet50, VGG16, and GoogleNet architectures combined with several activation functions such as ReLU, Leaky ReLU, Sigmoid, and Tanh in the classification of images that experience Poisson noise. Poisson noise is applied to each test data to evaluate CNN accuracy. The data used in this study consists of three scenarios of different numbers of classes, namely 3 classes, 5 classes, and 10 classes. The results showed that combining ResNet50 with the ReLU activation function produced the best performance in class recognition in each scenario of the number of classes experiencing Poisson noise interference. The model achieved 97% accuracy for 3-class data, 95% for 5-class data, and 90% for 10-class data. These results show that using ResNet50 with the ReLU activation function can provide excellent resistance to Poisson noise in image processing. It was found that as the number of classes increases, the accuracy of image recognition tends to decrease. This shows that the more complex the image classification task is with a larger number of classes, the more difficult it is for CNNs to distinguish between different classes. Doi: 10.28991/ESJ-2024-08-02-014 Full Text: PDF

Amplifying Sine Unit: An Oscillatory Activation Function for Deep Neural Networks to Recover Nonlinear Oscillations Efficiently

Many industrial and real-life problems exhibit highly nonlinear periodic behaviors and the conventional methods may fall short of finding their analytical or closed-form solutions. Such problems demand cutting-edge computational tools with increased functionality and reduced cost. Recently, deep neural networks have gained massive research interest due to their ability to handle large data and universality to learn complex functions. In this work, we put forward a methodology based on deep neural networks with responsive layers structure to deal nonlinear oscillations in microelectromechanical systems. We incorporated some oscillatory and non-oscillatory activation functions such as growing cosine unit known as GCU, Sine, Mish and Tanh in our designed network to have a comprehensive analysis on their performance for highly nonlinear problems. Integrating oscillatory activation functions with deep neural networks definitely outperform in predicting the periodic patterns of underlying systems. To support oscillatory actuation for nonlinear systems, we have proposed a novel oscillatory activation function called Amplifying Sine Unit denoted as ASU which is more efficient than GCU for complex vibratory systems such as microelectromechanical systems. Experimental results show that the designed network with our proposed activation function ASU is more reliable and robust to handle the challenges posed by oscillations

Performance of Fourier-based activation function in physics-informed neural networks for patient-specific cardiovascular flows

Background and ObjectivesPhysics-informed neural networks (PINNs) can be used to inversely model complex physical systems by encoding the governing partial differential equations and training data into the neural network. However, neural networks are known to be biased towards learning less complex functions, called spectral bias. This has important implications in modeling cardiovascular flows, where spatial frequencies can vary substantially across anatomies and pathologies (e.g., aneurysms or stenoses). Recent evidence suggests that Fourier-based activation functions have desirable properties, and can potentially reduce spectral bias; however, the performance and adequacy of such Fourier activation functions have not yet been evaluated in patient-specific cardiovascular flow applications. MethodsThe performance of sine activation function was evaluated against tanh and swish activation functions in a 1D advection-diffusion problem, an eccentric 2D stenosis model (Re=5000), and a patient-specific 3D aortic model (Re=823) under pulsatile flow conditions. CFD simulations were performed at high spatio-temporal resolution and data points were extracted for training the neural network. The number of training data points were normalized by L/D. The performance of the PINNs framework was evaluated with increasing number of training data points and across all three activation functions. ResultsOur results demonstrate that sine activation function presents desirable characteristics, such as monotonic reduction in errors, relatively faster convergence, and accurate eigen spectra at higher modes, compared to tanh and swish activation functions. Interestingly, for all activation functions, the domain-averaged errors tended to asymptote at ≈15−20% despite substantial increase in training point density. For 2D eccentric stenosis, errors asymptoted at a sensor point density of 40L/D. For 3D patient-specific aorta, this asymptote was achieved at 180L/D for all three activation functions with an error of ≈15% although sine activation function demonstrated relatively faster convergence. ConclusionsWe have demonstrated that Fourier-based activation functions have higher performance in terms of accuracy and convergence properties for cardiovascular flow applications; however, inherent challenges of neural networks (e.g., spectral bias) can limit the accuracy to ≈15% under physiological, 3D patient-specific blood flow conditions.

ModSwish: a new activation function for neural network

modSwish: a new activation function for neural network

A Formal Characterization of Activation Functions in Deep Neural Networks.

In this article, a mathematical formulation for describing and designing activation functions in deep neural networks is provided. The methodology is based on a precise characterization of the desired activation functions that satisfy particular criteria, including circumventing vanishing or exploding gradients during training. The problem of finding desired activation functions is formulated as an infinite-dimensional optimization problem, which is later relaxed to solving a partial differential equation. Furthermore, bounds that guarantee the optimality of the designed activation function are provided. Relevant examples with some state-of-the-art activation functions are provided to illustrate the methodology.

Multiobjective Multiverse Optimizer for Multirobotic U-Shaped Disassembly Line Balancing Problems

The development of technology accelerates the upgrade of products, which results in a significant number of obsolete products. This research aims to solve the Multi-robotic Multi-product U-shaped Disassembly-line-balancing Problem (M2UDP), in which different products are disassembled on a U-shaped model in a pre-set cycle time and assigned tasks to robots in each workstation reasonably. A linear mixed-integer model is established to maximize disassembly profit and minimize carbon emissions. An improved multi-objective multi-verse optimizer (IMMO) that utilizes the sigmoid activation function in neural networks is proposed to find the optimal plan for the model. The improved algorithm is verified via a set of real-life instances and compared with three classical multi-objective optimization algorithms. The experimental results show that the proposed IMMO performs better than those peers in solving the M2UDP problems.